From phyo.arkarlwin at gmail.com  Sat Aug  1 22:35:40 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Sun, 2 Aug 2015 03:05:40 +0630
Subject: [pypy-dev] pypy-2.3.1 memory profiling strategies
In-Reply-To: <CALZWULo13nxTrq2AAkjj5CrCEn486MLtCmc_Vny92adLNj3i0A@mail.gmail.com>
References: <CALZWULo13nxTrq2AAkjj5CrCEn486MLtCmc_Vny92adLNj3i0A@mail.gmail.com>
Message-ID: <CA+HjJrhv_JuEeYYQu8ppSmw=FD=uENDDr8GO5z0vbG9L+GuoKQ@mail.gmail.com>

Memory handling is improved on 2.6.1 , theres no memory leak there. we
running for 2 months already.
You gotta upgrade.

On Fri, Jul 31, 2015 at 3:51 PM, jim horng <jimhorng at gmail.com> wrote:

> Hi all,
> Not sure if it's appropriate to post here since there's no pypy-user group
> :)
> My question is as title, we got memory leak on pypy process and process
> will down when out of memory, only on production site.
> Our simplified environment as below:
> * OS: Centos 6
> * pypy-2.3.1
>
> objgraph is seems the only profiling library we can use in this env, and
> only with its partial function of printing all current objects in memory
> instead of any further info such as references (.getrefcount not
> implemented).
> It turns out we can only see lots of "int", "str", "list" objects seems
> leaking rather than knowing who are using them or whom they are using. :(
>
> Our constraint are
> * it's hard to change production python runtime since it might affect to
> our users
> * we cannot reproduce on other environment
>
> "pmap" produced data only shows memory growing in a [anon] block.
>
> Please advise if there's other tool/methodologies to attack this problem,
> thanks a lot in advance :)
>
> Best Regards,
> Jim(???)
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150802/ea70926a/attachment.html>

From jimhorng at gmail.com  Sun Aug  2 08:57:34 2015
From: jimhorng at gmail.com (jim horng)
Date: Sun, 2 Aug 2015 14:57:34 +0800
Subject: [pypy-dev] pypy-2.3.1 memory profiling strategies
In-Reply-To: <CA+HjJrhv_JuEeYYQu8ppSmw=FD=uENDDr8GO5z0vbG9L+GuoKQ@mail.gmail.com>
References: <CALZWULo13nxTrq2AAkjj5CrCEn486MLtCmc_Vny92adLNj3i0A@mail.gmail.com>
 <CA+HjJrhv_JuEeYYQu8ppSmw=FD=uENDDr8GO5z0vbG9L+GuoKQ@mail.gmail.com>
Message-ID: <CALZWULqECHa3jwn=cBEzHF_qTp+Pge_S-ixPurYfOCjH0143AQ@mail.gmail.com>

Hi Phyo,
thanks for prompt response :)
I didn't state clear that I already tried periodic gc but memory usage
still increase over team till out of memory, so we assume it's a real leak
cause by python program but we don't have good tool/methodology to hunt it,
thanks :)

Best Regards,
Jim(???)

On Sun, Aug 2, 2015 at 4:35 AM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:

> Memory handling is improved on 2.6.1 , theres no memory leak there. we
> running for 2 months already.
> You gotta upgrade.
>
> On Fri, Jul 31, 2015 at 3:51 PM, jim horng <jimhorng at gmail.com> wrote:
>
>> Hi all,
>> Not sure if it's appropriate to post here since there's no pypy-user
>> group :)
>> My question is as title, we got memory leak on pypy process and process
>> will down when out of memory, only on production site.
>> Our simplified environment as below:
>> * OS: Centos 6
>> * pypy-2.3.1
>>
>> objgraph is seems the only profiling library we can use in this env, and
>> only with its partial function of printing all current objects in memory
>> instead of any further info such as references (.getrefcount not
>> implemented).
>> It turns out we can only see lots of "int", "str", "list" objects seems
>> leaking rather than knowing who are using them or whom they are using. :(
>>
>> Our constraint are
>> * it's hard to change production python runtime since it might affect to
>> our users
>> * we cannot reproduce on other environment
>>
>> "pmap" produced data only shows memory growing in a [anon] block.
>>
>> Please advise if there's other tool/methodologies to attack this problem,
>> thanks a lot in advance :)
>>
>> Best Regards,
>> Jim(???)
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150802/355bfaec/attachment.html>

From mount.sarah at gmail.com  Sun Aug  2 23:00:16 2015
From: mount.sarah at gmail.com (Sarah Mount)
Date: Sun, 2 Aug 2015 17:00:16 -0400
Subject: [pypy-dev] Best way to use py.test with RPython projects and
	travis-ci?
Message-ID: <CALG3WZnv94ECappVrcap2hewht1eGBE2A3YQuZtm1srSoobh3Q@mail.gmail.com>

This week someone pushed a nice commit to a Python repository I contribute
to, which sped up Travis builds by upgrading pip and using the cache to
store requirements. I thought "that's neat, I should try that trick in my
own (R)Python projects". And that, friends, is how I entered a whole new
world of pain and suffering.

It seems that if you install py.test using pip and cache pip packages in
Travis, like this:

https://travis-ci.org/futurecore/revelation/builds/73584887

I think because PyPy packages its own version of py.test and the two get
confused.

On the other hand, if you strip py and py.test from PyPy then you can't
translate your interpreter:

https://travis-ci.org/futurecore/revelation/builds/73694483

What is the recommended way to manage this stuff?

Thanks,

Sarah

-- 
Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
website:  http://www.snim2.org/
twitter: @snim2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150802/cae2985f/attachment.html>

From arigo at tunes.org  Mon Aug  3 09:47:56 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Aug 2015 09:47:56 +0200
Subject: [pypy-dev] Best way to use py.test with RPython projects and
	travis-ci?
In-Reply-To: <CALG3WZnv94ECappVrcap2hewht1eGBE2A3YQuZtm1srSoobh3Q@mail.gmail.com>
References: <CALG3WZnv94ECappVrcap2hewht1eGBE2A3YQuZtm1srSoobh3Q@mail.gmail.com>
Message-ID: <CAMSv6X05vXM8B35QXsK42BhOw74bezwMMoE72rTRX81+WQYezQ@mail.gmail.com>

Hi Sarah,

On 2 August 2015 at 23:00, Sarah Mount <mount.sarah at gmail.com> wrote:
> What is the recommended way to manage this stuff?

There is no recommended way as far as I know.  Maybe you should try
stripping "_pytest" from the pypy repository but not "py"?


Armin

From arigo at tunes.org  Mon Aug  3 09:52:17 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 3 Aug 2015 09:52:17 +0200
Subject: [pypy-dev] pypy-2.3.1 memory profiling strategies
In-Reply-To: <CALZWULqECHa3jwn=cBEzHF_qTp+Pge_S-ixPurYfOCjH0143AQ@mail.gmail.com>
References: <CALZWULo13nxTrq2AAkjj5CrCEn486MLtCmc_Vny92adLNj3i0A@mail.gmail.com>
 <CA+HjJrhv_JuEeYYQu8ppSmw=FD=uENDDr8GO5z0vbG9L+GuoKQ@mail.gmail.com>
 <CALZWULqECHa3jwn=cBEzHF_qTp+Pge_S-ixPurYfOCjH0143AQ@mail.gmail.com>
Message-ID: <CAMSv6X0b6U-gbsyPoKx8n+1y3psgomiZv_WzNh9m09TcM89jeA@mail.gmail.com>

Hi Jim,

On 2 August 2015 at 08:57, jim horng <jimhorng at gmail.com> wrote:
> I didn't state clear that I already tried periodic gc but memory usage still
> increase over team till out of memory, so we assume it's a real leak cause
> by python program but we don't have good tool/methodology to hunt it, thanks
> :)

I believe you are failing to understand Phyo's answer: PyPy 2.3.1 had
some internal memory leaks that were fixed later.  If you're running
into these, there is nothing you can do but upgrade to a more recent
PyPy.  There are no Python-level tricks that can free this leaked
memory.


A bient?t,

Armin.

From phyo.arkarlwin at gmail.com  Mon Aug  3 21:13:12 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Tue, 4 Aug 2015 01:43:12 +0630
Subject: [pypy-dev] For Donations, List How Much is Left (or Spent)?
In-Reply-To: <CAK5idxSfSwTWRyae24qh93tY0TJ9RphPqGq2FsSCwMpArhpVRQ@mail.gmail.com>
References: <etPan.556cb9c8.4f34c5bc.18516@Draupnir.home>
 <CAMSv6X0Z9Zq7=L7CnZGR3t_yWmC6+qsEf0_k-QX4OB0cWaLDhg@mail.gmail.com>
 <CA+HjJrjuHt6Kce++e4vXr=tHNhJE-7xhaNyFYwy94WKhViA6OA@mail.gmail.com>
 <CAGmFidYv1ud6WR-HmWMEh4Riut_kfwk5iwPXVG246j2av7PSGA@mail.gmail.com>
 <CAMSv6X0fp=LXQeXzUkapqto17DNM0ERL04=7uSAr5sAenoX2bg@mail.gmail.com>
 <CAGmFidabudtBG9+QHB=LhBkQLV9n+Tnk+zT_RuRPVQT-HHcrBw@mail.gmail.com>
 <CAK5idxSfSwTWRyae24qh93tY0TJ9RphPqGq2FsSCwMpArhpVRQ@mail.gmail.com>
Message-ID: <CA+HjJri1SyXXVfEPmQS6++N8QnsqQ-rU_g+jdtx_psjWs4zJbw@mail.gmail.com>

Anything new on this Pypy 3.x move?

On Fri, Jun 26, 2015 at 12:14 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> I can help you to iron out issue tomorrow I guess?
>
> On Thu, Jun 25, 2015 at 9:50 AM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
> > 2015-06-25 9:11 GMT+02:00 Armin Rigo <arigo at tunes.org>:
> >>
> >> Hi Amaury,
> >>
> >> On 25 June 2015 at 01:10, Amaury Forgeot d'Arc <amauryfa at gmail.com>
> wrote:
> >> > Currently I'm kind of blocked on finishing the "more-rposix" branch
> >> > (needed for the recent expansion of the posix module:
> >> > os.supports_dir_fd...)
> >> > It has still some failures: win32 does not translate, for example.
> >> > Since it's mostly changes in rpython/ I wanted to make it land in the
> >> > "default" branch first,
> >> > but I'll probably move along and merge it into the py3.3 branch.
> >>
> >> If it's only adding new functions, then I think it's fine to merge it
> >> to default even if some of these functions cause translation errors on
> >> Windows.  The default pypy won't call them anyway.
> >
> >
> > No, this branch is actually a full rewrite of the ll_os.py file,
> > all the code has been moved to rposix.py.
> >
> > The code is much simpler (no lazy registering callback function),
> > and we have a consistent way to add new wrapper to C functions,
> > even if they don't map to any 2.7 feature.
> >
> >
> > --
> > Amaury Forgeot d'Arc
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150804/e4b18207/attachment.html>

From fijall at gmail.com  Wed Aug  5 09:16:36 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 5 Aug 2015 09:16:36 +0200
Subject: [pypy-dev] Cobra down
Message-ID: <CAK5idxSj2JQpxrMBBHuxpH1e5hj5YFxm7uDOuFt5wWL80_XZOQ@mail.gmail.com>

According to David, cobra/buildbot will be down for a day or two

From fijall at gmail.com  Wed Aug  5 09:52:23 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 5 Aug 2015 09:52:23 +0200
Subject: [pypy-dev] RinohType and PyPy2
In-Reply-To: <1431737446.616651.1393713257918.JavaMail.sas1@172.29.252.247>
References: <1431737446.616651.1393713257918.JavaMail.sas1@172.29.252.247>
Message-ID: <CAK5idxQzBHD=SLtED4S+EFUf-t0M0uJopMGY8mDh-DAyrh01+w@mail.gmail.com>

Hi Brecht

We tried the strbuf with rinohtype and (after warmup) it's the speed
of cpython, which is bad.

investigating some more....

On Sun, Mar 2, 2014 at 12:34 AM, Brecht Machiels <brecht at mos6581.org> wrote:
> Hello,
>
> I've managed to backport RinohType to Python 2 (took me only a couple of hours thankfully).
>
> Results on my Celeron T3000 (Arch Linux x86_64):
> CPython 3.3.4        14 s
> PyPy3 2.1.0-beta1  61 s
> CPython 2.7.6        15 s
> PyPy  2.2.1            35 s
>
> If you want to give it a try (no external dependencies):
>
>     git clone --branch pypy2 https://github.com/brechtm/rinohtype.git
>     cd rinohtype/examples/rfic2009
>     rm -rf template.ptc; PYTHONPATH=../.. pypy template.py
>
>
> While PyPy2 performs better than PyPy3, it's still much slower than CPython. Is RinohType hitting a weak spot in PyPy? Any hints on what I can do to improve performance?
>
> Best regards,
> Brecht
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From vaibhav_sood at persistent.com  Wed Aug  5 10:22:41 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Wed, 5 Aug 2015 08:22:41 +0000
Subject: [pypy-dev] Running pypy unit tests
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD7A6@HJ-MBX1.persistent.co.in>

Hi All,

I work on Python through which I came across pypy and I am interested in making some open-source contributions to the project. As first steps I tried building the source code (successfully) and running the unit tests as per the "Getting Started Developing with PyPy" page but I run into an issue

The way I run the tests is "./pytest.py <folder>" for each folder in the source code, I get the test results all folders but the command hangs for one particular folder "pypy/module"

For example,

./pytest.py -rfExs pypy/module
>> Running this never returns, doing a ps -ef I see multiple pytest.py processes running

I run the tests on a x86 VM with 4GB RAM. I am not sure if I am doing this right but is there a way around this issue? Also is there any recommended way to run all the pypy tests and is there any other recommended strategy to test new code added to pypy that I am missing?

I apologize for the slightly lengthy mail and the multiple questions

Thanks!
Vaibhav

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150805/654b7d4f/attachment.html>

From brecht__gmane at mos6581.org  Wed Aug  5 16:51:19 2015
From: brecht__gmane at mos6581.org (Brecht Machiels)
Date: Wed, 05 Aug 2015 16:51:19 +0200
Subject: [pypy-dev] RinohType and PyPy2
References: <1431737446.616651.1393713257918.JavaMail.sas1@172.29.252.247>
 <CAK5idxQzBHD=SLtED4S+EFUf-t0M0uJopMGY8mDh-DAyrh01+w@mail.gmail.com>
Message-ID: <op.x2wgvt0fcmspgy@brechts-mbp-2.fritz.box>

Hello Maciej,

Mmm, too bad. Thankfully, with PyPy 2.6.0, speed is about the same as for  
CPython. PyPy 2.2.1 was significantly slower.

In any case, thanks for trying out the strbuf optimization. I'm sure it  
will be helpful for other applications. And we can rule this out as the  
cause of the low performance with RinohType.

I've made some changes to make RinohType more suitable for benchmarking.  
You can find it here:
branch pypy_benchmark of git at github.com:brechtm/rinohtype.git

Simply run benchmark_pypy.sh (and benchmark_python.sh).

You can change the length of the document by changing the 'repeat'  
parameter in examples/rfic2009/template.xml

For repeat=500, PyPy 2.6.0 is about as fast as CPython 2.7.9
For repeat=5000, PyPy = 214s and CPython = 227s

If you think this benchmark is useful, please include it in  
http://speed.pypy.org

Let me know if there's anything else I can do.

Cheers,
Brecht

On Wed, 05 Aug 2015 09:52:23 +0200, Maciej Fijalkowski <fijall at gmail.com>  
wrote:

> Hi Brecht
>
> We tried the strbuf with rinohtype and (after warmup) it's the speed
> of cpython, which is bad.
>
> investigating some more....
>
> On Sun, Mar 2, 2014 at 12:34 AM, Brecht Machiels <brecht at mos6581.org>  
> wrote:
>> Hello,
>>
>> I've managed to backport RinohType to Python 2 (took me only a couple  
>> of hours thankfully).
>>
>> Results on my Celeron T3000 (Arch Linux x86_64):
>> CPython 3.3.4        14 s
>> PyPy3 2.1.0-beta1  61 s
>> CPython 2.7.6        15 s
>> PyPy  2.2.1            35 s
>>
>> If you want to give it a try (no external dependencies):
>>
>>     git clone --branch pypy2 https://github.com/brechtm/rinohtype.git
>>     cd rinohtype/examples/rfic2009
>>     rm -rf template.ptc; PYTHONPATH=../.. pypy template.py
>>
>>
>> While PyPy2 performs better than PyPy3, it's still much slower than  
>> CPython. Is RinohType hitting a weak spot in PyPy? Any hints on what I  
>> can do to improve performance?
>>
>> Best regards,
>> Brecht


From cyrus at daboo.name  Wed Aug  5 17:59:54 2015
From: cyrus at daboo.name (Cyrus Daboo)
Date: Wed, 05 Aug 2015 11:59:54 -0400
Subject: [pypy-dev] Building on case-insensitive file system
Message-ID: <6C74FD21FA449E4A984B7F41@caldav.corp.apple.com>

Hi,
OS X defaults to having a case-insensitive file system. I believe there is 
a problem with building pypy from source on such a system.

During the build process I see files being written to the jitcodes 
directory in the /tmp build directory. There are some "W_ArrayTypeX.*" 
files being written there, where "X" is a typecode (which I believe comes 
from pypy/module/array/interp_array.py). The allowable codes come from the 
set "c, b, B, u, h, H, i, I, l, L, f or d". Note that there are a mixture 
of lower and upper case characters that are the same.

When I build on OS X, with an option to force an error when trying to 
create a file with a name that matches another, but with different case, I 
see this:

[translation:ERROR] EEXIST: [File exists]: 
file('/tmp/usession-release-2.6.x-0/jitcodes/W_ArrayTypel._charbuf_stop', 
'w')

That is because "W_ArrayTypeL._charbuf_stop" exists - i.e., "l" and "L" are 
in conflict.

Now, I am not a PyPy dev expert, so I have no idea what the impact of this 
might be, but it seems wrong to me. Though I cannot detect any obvious 
problems in the built pypy binary.

I am not sure what an appropriate fix for this would be. Perhaps changing 
the typecode set to use unique characters (case insensitive) for all the 
different types?

-- 
Cyrus Daboo


From fijall at gmail.com  Wed Aug  5 20:15:50 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 5 Aug 2015 20:15:50 +0200
Subject: [pypy-dev] RinohType and PyPy2
In-Reply-To: <op.x2wgvt0fcmspgy@brechts-mbp-2.fritz.box>
References: <1431737446.616651.1393713257918.JavaMail.sas1@172.29.252.247>
 <CAK5idxQzBHD=SLtED4S+EFUf-t0M0uJopMGY8mDh-DAyrh01+w@mail.gmail.com>
 <op.x2wgvt0fcmspgy@brechts-mbp-2.fritz.box>
Message-ID: <CAK5idxTTXe071SMjOrjPR8WOuLnjMaUU9W6hPW1x921-7yDd_g@mail.gmail.com>

no, the strbuf does improve stuff about 20-30% it's just not what I
was hoping for :-)

On Wed, Aug 5, 2015 at 4:51 PM, Brecht Machiels
<brecht__gmane at mos6581.org> wrote:
> Hello Maciej,
>
> Mmm, too bad. Thankfully, with PyPy 2.6.0, speed is about the same as for
> CPython. PyPy 2.2.1 was significantly slower.
>
> In any case, thanks for trying out the strbuf optimization. I'm sure it will
> be helpful for other applications. And we can rule this out as the cause of
> the low performance with RinohType.
>
> I've made some changes to make RinohType more suitable for benchmarking. You
> can find it here:
> branch pypy_benchmark of git at github.com:brechtm/rinohtype.git
>
> Simply run benchmark_pypy.sh (and benchmark_python.sh).
>
> You can change the length of the document by changing the 'repeat' parameter
> in examples/rfic2009/template.xml
>
> For repeat=500, PyPy 2.6.0 is about as fast as CPython 2.7.9
> For repeat=5000, PyPy = 214s and CPython = 227s
>
> If you think this benchmark is useful, please include it in
> http://speed.pypy.org
>
> Let me know if there's anything else I can do.
>
> Cheers,
> Brecht
>
>
> On Wed, 05 Aug 2015 09:52:23 +0200, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>
>> Hi Brecht
>>
>> We tried the strbuf with rinohtype and (after warmup) it's the speed
>> of cpython, which is bad.
>>
>> investigating some more....
>>
>> On Sun, Mar 2, 2014 at 12:34 AM, Brecht Machiels <brecht at mos6581.org>
>> wrote:
>>>
>>> Hello,
>>>
>>> I've managed to backport RinohType to Python 2 (took me only a couple of
>>> hours thankfully).
>>>
>>> Results on my Celeron T3000 (Arch Linux x86_64):
>>> CPython 3.3.4        14 s
>>> PyPy3 2.1.0-beta1  61 s
>>> CPython 2.7.6        15 s
>>> PyPy  2.2.1            35 s
>>>
>>> If you want to give it a try (no external dependencies):
>>>
>>>     git clone --branch pypy2 https://github.com/brechtm/rinohtype.git
>>>     cd rinohtype/examples/rfic2009
>>>     rm -rf template.ptc; PYTHONPATH=../.. pypy template.py
>>>
>>>
>>> While PyPy2 performs better than PyPy3, it's still much slower than
>>> CPython. Is RinohType hitting a weak spot in PyPy? Any hints on what I can
>>> do to improve performance?
>>>
>>> Best regards,
>>> Brecht
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From arigo at tunes.org  Thu Aug  6 10:46:27 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Aug 2015 10:46:27 +0200
Subject: [pypy-dev] Running pypy unit tests
In-Reply-To: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD7A6@HJ-MBX1.persistent.co.in>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD7A6@HJ-MBX1.persistent.co.in>
Message-ID: <CAMSv6X3XFk4OHyD5x11rTN9WDOWDSP6q3h32aXD=w+Ms6fATmw@mail.gmail.com>

Hi Vaibhav,

On 5 August 2015 at 10:22, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
> The way I run the tests is ?./pytest.py <folder>? for each folder in the
> source code, I get the test results all folders but the command hangs for
> one particular folder ?pypy/module?

By "each folder" we mean "each individual 'test' subfolder".  The
pypy/module directory contains a large number of subdirectories, each
with their own tests, so you should not run all the pypy/module tests
at once.

Yes, there is an automatic way.  However in general we don't use it
ourselves.  It is used automatically by the nightly test runs which
show up at http://buildbot.pypy.org (when the web site is not down
like now...).

Instead, when we do changes somewhere, we run directly the tests in
the "test" subdirectory of where we did the changes, and if they pass,
we commit.  In the rare case where this breaks things at some
unrelated place, we will notice the next morning after the nightly
run, and fix it then.


A bient?t,

Armin.

From fijall at gmail.com  Thu Aug  6 11:15:25 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Aug 2015 11:15:25 +0200
Subject: [pypy-dev] Running pypy unit tests
In-Reply-To: <CAMSv6X3XFk4OHyD5x11rTN9WDOWDSP6q3h32aXD=w+Ms6fATmw@mail.gmail.com>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD7A6@HJ-MBX1.persistent.co.in>
 <CAMSv6X3XFk4OHyD5x11rTN9WDOWDSP6q3h32aXD=w+Ms6fATmw@mail.gmail.com>
Message-ID: <CAK5idxSKGvOnwTexNS5CiQCvhnRPVUUKOdiX_FvFfNT8Y8m2HA@mail.gmail.com>

It's maybe worth noting additionally that running a full test suite
takes several hours, so we don't recommend it as a way to do
development.

On Thu, Aug 6, 2015 at 10:46 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Vaibhav,
>
> On 5 August 2015 at 10:22, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
>> The way I run the tests is ?./pytest.py <folder>? for each folder in the
>> source code, I get the test results all folders but the command hangs for
>> one particular folder ?pypy/module?
>
> By "each folder" we mean "each individual 'test' subfolder".  The
> pypy/module directory contains a large number of subdirectories, each
> with their own tests, so you should not run all the pypy/module tests
> at once.
>
> Yes, there is an automatic way.  However in general we don't use it
> ourselves.  It is used automatically by the nightly test runs which
> show up at http://buildbot.pypy.org (when the web site is not down
> like now...).
>
> Instead, when we do changes somewhere, we run directly the tests in
> the "test" subdirectory of where we did the changes, and if they pass,
> we commit.  In the rare case where this breaks things at some
> unrelated place, we will notice the next morning after the nightly
> run, and fix it then.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From mstuchli at redhat.com  Thu Aug  6 14:46:45 2015
From: mstuchli at redhat.com (Matej Stuchlik)
Date: Thu, 6 Aug 2015 08:46:45 -0400 (EDT)
Subject: [pypy-dev] Running pypy unit tests
In-Reply-To: <486382865.4935173.1438852711608.JavaMail.zimbra@redhat.com>
Message-ID: <799815441.5002580.1438865205454.JavaMail.zimbra@redhat.com>

Hi Armin,

----- Original Message -----
> From: "Armin Rigo" <arigo at tunes.org>
> To: "Vaibhav Sood" <vaibhav_sood at persistent.com>
> Cc: pypy-dev at python.org
> Sent: Thursday, August 6, 2015 10:46:27 AM
> Subject: Re: [pypy-dev] Running pypy unit tests
> 
> Hi Vaibhav,
> 
> On 5 August 2015 at 10:22, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
> > The way I run the tests is ?./pytest.py <folder>? for each folder in the
> > source code, I get the test results all folders but the command hangs for
> > one particular folder ?pypy/module?
> 
> By "each folder" we mean "each individual 'test' subfolder".  The
> pypy/module directory contains a large number of subdirectories, each
> with their own tests, so you should not run all the pypy/module tests
> at once.
> 
> Yes, there is an automatic way.  However in general we don't use it
> ourselves.  It is used automatically by the nightly test runs which
> show up at http://buildbot.pypy.org (when the web site is not down
> like now...).

Could you please describe the automatic way in a little more detail?
buildbot.pypy.org seems to be down right now, so I can't look it up there.
We run all the tests when building pypy for Fedora and I'm pretty sure the
way we run them is not exactly current by now. :)

Matt


> Instead, when we do changes somewhere, we run directly the tests in
> the "test" subdirectory of where we did the changes, and if they pass,
> we commit.  In the rare case where this breaks things at some
> unrelated place, we will notice the next morning after the nightly
> run, and fix it then.
> 
> 
> A bient?t,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 

From fijall at gmail.com  Thu Aug  6 15:18:48 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 6 Aug 2015 15:18:48 +0200
Subject: [pypy-dev] Running pypy unit tests
In-Reply-To: <799815441.5002580.1438865205454.JavaMail.zimbra@redhat.com>
References: <486382865.4935173.1438852711608.JavaMail.zimbra@redhat.com>
 <799815441.5002580.1438865205454.JavaMail.zimbra@redhat.com>
Message-ID: <CAK5idxSbakrDsogirxAedu4okaZmpt8r+99Onq9a9hG695TEyw@mail.gmail.com>

On Thu, Aug 6, 2015 at 2:46 PM, Matej Stuchlik <mstuchli at redhat.com> wrote:
> Hi Armin,
>
> ----- Original Message -----
>> From: "Armin Rigo" <arigo at tunes.org>
>> To: "Vaibhav Sood" <vaibhav_sood at persistent.com>
>> Cc: pypy-dev at python.org
>> Sent: Thursday, August 6, 2015 10:46:27 AM
>> Subject: Re: [pypy-dev] Running pypy unit tests
>>
>> Hi Vaibhav,
>>
>> On 5 August 2015 at 10:22, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
>> > The way I run the tests is ?./pytest.py <folder>? for each folder in the
>> > source code, I get the test results all folders but the command hangs for
>> > one particular folder ?pypy/module?
>>
>> By "each folder" we mean "each individual 'test' subfolder".  The
>> pypy/module directory contains a large number of subdirectories, each
>> with their own tests, so you should not run all the pypy/module tests
>> at once.
>>
>> Yes, there is an automatic way.  However in general we don't use it
>> ourselves.  It is used automatically by the nightly test runs which
>> show up at http://buildbot.pypy.org (when the web site is not down
>> like now...).
>
> Could you please describe the automatic way in a little more detail?
> buildbot.pypy.org seems to be down right now, so I can't look it up there.
> We run all the tests when building pypy for Fedora and I'm pretty sure the
> way we run them is not exactly current by now. :)
>
> Matt
>
>
>> Instead, when we do changes somewhere, we run directly the tests in
>> the "test" subdirectory of where we did the changes, and if they pass,
>> we commit.  In the rare case where this breaks things at some
>> unrelated place, we will notice the next morning after the nightly
>> run, and fix it then.
>>
>>
>> A bient?t,
>>
>> Armin.
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

The buildbot repository is up https://bitbucket.org/pypy/buildbot

There are instructions there, shout if you have trouble running them.

From vaibhav_sood at persistent.com  Thu Aug  6 16:23:50 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Thu, 6 Aug 2015 14:23:50 +0000
Subject: [pypy-dev] Running pypy unit tests
In-Reply-To: <CAK5idxSKGvOnwTexNS5CiQCvhnRPVUUKOdiX_FvFfNT8Y8m2HA@mail.gmail.com>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD7A6@HJ-MBX1.persistent.co.in>
 <CAMSv6X3XFk4OHyD5x11rTN9WDOWDSP6q3h32aXD=w+Ms6fATmw@mail.gmail.com>,
 <CAK5idxSKGvOnwTexNS5CiQCvhnRPVUUKOdiX_FvFfNT8Y8m2HA@mail.gmail.com>
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025AD88C@HJ-MBX1.persistent.co.in>

This helps, thanks for the replies!
________________________________________
From: Maciej Fijalkowski [fijall at gmail.com]
Sent: Thursday, August 06, 2015 2:45 PM
To: Armin Rigo
Cc: Vaibhav Sood; pypy-dev at python.org
Subject: Re: [pypy-dev] Running pypy unit tests

It's maybe worth noting additionally that running a full test suite
takes several hours, so we don't recommend it as a way to do
development.

On Thu, Aug 6, 2015 at 10:46 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Vaibhav,
>
> On 5 August 2015 at 10:22, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
>> The way I run the tests is ?./pytest.py <folder>? for each folder in the
>> source code, I get the test results all folders but the command hangs for
>> one particular folder ?pypy/module?
>
> By "each folder" we mean "each individual 'test' subfolder".  The
> pypy/module directory contains a large number of subdirectories, each
> with their own tests, so you should not run all the pypy/module tests
> at once.
>
> Yes, there is an automatic way.  However in general we don't use it
> ourselves.  It is used automatically by the nightly test runs which
> show up at http://buildbot.pypy.org (when the web site is not down
> like now...).
>
> Instead, when we do changes somewhere, we run directly the tests in
> the "test" subdirectory of where we did the changes, and if they pass,
> we commit.  In the rare case where this breaks things at some
> unrelated place, we will notice the next morning after the nightly
> run, and fix it then.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


From hengha.mao at gmail.com  Fri Aug  7 12:19:29 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Fri, 7 Aug 2015 18:19:29 +0800
Subject: [pypy-dev] serialize and deserialize objects in Pypy
Message-ID: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>

Hi,

Are there any good methods to serialize Pypy object to C++ char* and
deserialize to Pypy object from  C++ char*?

Our scenario like this:
We have at least two C++ process running, and each process embeded Pypy to
execute some functions.
And we woule like to reuse Pypy object from one process to another process.
For primitive objects, int/char/bool/double etc., we could use cffi to pass
objects.
Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object

But for complex objects, e.g. list/array/dict/tuple, are there any methods?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150807/4f6dcc96/attachment.html>

From fijall at gmail.com  Fri Aug  7 12:24:05 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 7 Aug 2015 12:24:05 +0200
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
Message-ID: <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>

you can use cffi buffers (e.g. cffi char*) that you manipulate from
python using wrappers. They would behave (sort of) like python objects
and the wrapper code is really not a performance penalty in most cases

On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
> Hi,
>
> Are there any good methods to serialize Pypy object to C++ char* and
> deserialize to Pypy object from  C++ char*?
>
> Our scenario like this:
> We have at least two C++ process running, and each process embeded Pypy to
> execute some functions.
> And we woule like to reuse Pypy object from one process to another process.
> For primitive objects, int/char/bool/double etc., we could use cffi to pass
> objects.
> Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
>
> But for complex objects, e.g. list/array/dict/tuple, are there any methods?
>
>
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From cfbolz at gmx.de  Fri Aug  7 12:27:48 2015
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 07 Aug 2015 12:27:48 +0200
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
Message-ID: <8b57c1ab-cd6e-4870-9bd2-444ba5a5966d@email.android.com>

Can't you use pickle or the json module? 

Cheers, 

Carl Friedrich

On August 7, 2015 12:24:05 PM GMT+02:00, Maciej Fijalkowski <fijall at gmail.com> wrote:
>you can use cffi buffers (e.g. cffi char*) that you manipulate from
>python using wrappers. They would behave (sort of) like python objects
>and the wrapper code is really not a performance penalty in most cases
>
>On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
>wrote:
>> Hi,
>>
>> Are there any good methods to serialize Pypy object to C++ char* and
>> deserialize to Pypy object from  C++ char*?
>>
>> Our scenario like this:
>> We have at least two C++ process running, and each process embeded
>Pypy to
>> execute some functions.
>> And we woule like to reuse Pypy object from one process to another
>process.
>> For primitive objects, int/char/bool/double etc., we could use cffi
>to pass
>> objects.
>> Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
>>
>> But for complex objects, e.g. list/array/dict/tuple, are there any
>methods?
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150807/342f9177/attachment.html>

From hengha.mao at gmail.com  Fri Aug  7 13:09:50 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Fri, 7 Aug 2015 19:09:50 +0800
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
Message-ID: <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>

Hi Maciej,

Could you provide some details on how to use cffi buffers? (a simple demo
code or documents)
I tried the below code but it did not work.

#python list
x=[1,2,3,4]
#intend to convert to c++ char*, but failed
p=ffi.from_buffer(x)
#intend to covert back to python list, but failed
y=ffi.buffer(p)

Thanks!

-Ethan

On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> you can use cffi buffers (e.g. cffi char*) that you manipulate from
> python using wrappers. They would behave (sort of) like python objects
> and the wrapper code is really not a performance penalty in most cases
>
> On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
> wrote:
> > Hi,
> >
> > Are there any good methods to serialize Pypy object to C++ char* and
> > deserialize to Pypy object from  C++ char*?
> >
> > Our scenario like this:
> > We have at least two C++ process running, and each process embeded Pypy
> to
> > execute some functions.
> > And we woule like to reuse Pypy object from one process to another
> process.
> > For primitive objects, int/char/bool/double etc., we could use cffi to
> pass
> > objects.
> > Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
> >
> > But for complex objects, e.g. list/array/dict/tuple, are there any
> methods?
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150807/f9e478da/attachment.html>

From fijall at gmail.com  Fri Aug  7 13:46:38 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 7 Aug 2015 13:46:38 +0200
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
 <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
Message-ID: <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>

you really can't convert it to the C stuff like that. You need to do:

foo = ffi.new("int[]", 13)
foo[0] = 12
foo[1] = 55

etc.

You can write nice python wrapper around those things too, if the data
is too bare bone.

On Fri, Aug 7, 2015 at 1:09 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
> Hi Maciej,
>
> Could you provide some details on how to use cffi buffers? (a simple demo
> code or documents)
> I tried the below code but it did not work.
>
> #python list
> x=[1,2,3,4]
> #intend to convert to c++ char*, but failed
> p=ffi.from_buffer(x)
> #intend to covert back to python list, but failed
> y=ffi.buffer(p)
>
> Thanks!
>
> -Ethan
>
> On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>> you can use cffi buffers (e.g. cffi char*) that you manipulate from
>> python using wrappers. They would behave (sort of) like python objects
>> and the wrapper code is really not a performance penalty in most cases
>>
>> On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Are there any good methods to serialize Pypy object to C++ char* and
>> > deserialize to Pypy object from  C++ char*?
>> >
>> > Our scenario like this:
>> > We have at least two C++ process running, and each process embeded Pypy
>> > to
>> > execute some functions.
>> > And we woule like to reuse Pypy object from one process to another
>> > process.
>> > For primitive objects, int/char/bool/double etc., we could use cffi to
>> > pass
>> > objects.
>> > Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
>> >
>> > But for complex objects, e.g. list/array/dict/tuple, are there any
>> > methods?
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >
>
>

From hengha.mao at gmail.com  Fri Aug  7 14:06:20 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Fri, 7 Aug 2015 20:06:20 +0800
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
 <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
 <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>
Message-ID: <CAOEzsGBxX_3vkk_fnSg1hnOY7Oog=0_69HgKAC=+vJP7TxQaKQ@mail.gmail.com>

The problem is we might need to support several kinds of collection
objects: list, dict, tuple.
In addition, we might not know what is exact type of objects stored in the
collections. A list might store int, string, double.
So it might involve a lot of work to write the python wrapper.
Do you have any good ideas?

On Fri, Aug 7, 2015 at 7:46 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> you really can't convert it to the C stuff like that. You need to do:
>
> foo = ffi.new("int[]", 13)
> foo[0] = 12
> foo[1] = 55
>
> etc.
>
> You can write nice python wrapper around those things too, if the data
> is too bare bone.
>
> On Fri, Aug 7, 2015 at 1:09 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
> > Hi Maciej,
> >
> > Could you provide some details on how to use cffi buffers? (a simple demo
> > code or documents)
> > I tried the below code but it did not work.
> >
> > #python list
> > x=[1,2,3,4]
> > #intend to convert to c++ char*, but failed
> > p=ffi.from_buffer(x)
> > #intend to covert back to python list, but failed
> > y=ffi.buffer(p)
> >
> > Thanks!
> >
> > -Ethan
> >
> > On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >>
> >> you can use cffi buffers (e.g. cffi char*) that you manipulate from
> >> python using wrappers. They would behave (sort of) like python objects
> >> and the wrapper code is really not a performance penalty in most cases
> >>
> >> On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > Are there any good methods to serialize Pypy object to C++ char* and
> >> > deserialize to Pypy object from  C++ char*?
> >> >
> >> > Our scenario like this:
> >> > We have at least two C++ process running, and each process embeded
> Pypy
> >> > to
> >> > execute some functions.
> >> > And we woule like to reuse Pypy object from one process to another
> >> > process.
> >> > For primitive objects, int/char/bool/double etc., we could use cffi to
> >> > pass
> >> > objects.
> >> > Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
> >> >
> >> > But for complex objects, e.g. list/array/dict/tuple, are there any
> >> > methods?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > pypy-dev mailing list
> >> > pypy-dev at python.org
> >> > https://mail.python.org/mailman/listinfo/pypy-dev
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150807/4c695427/attachment.html>

From fijall at gmail.com  Fri Aug  7 14:08:52 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 7 Aug 2015 14:08:52 +0200
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAOEzsGBxX_3vkk_fnSg1hnOY7Oog=0_69HgKAC=+vJP7TxQaKQ@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
 <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
 <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>
 <CAOEzsGBxX_3vkk_fnSg1hnOY7Oog=0_69HgKAC=+vJP7TxQaKQ@mail.gmail.com>
Message-ID: <CAK5idxTC+knD3hMDZ0toc1O4BcEUjX+kgXROdiMHWgieBKS_nw@mail.gmail.com>

yes, you need a serializer protocol, like JSON (or protobuf or
anything). You generally need to express stuff in a way C can
understand and read it in python

On Fri, Aug 7, 2015 at 2:06 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
> The problem is we might need to support several kinds of collection objects:
> list, dict, tuple.
> In addition, we might not know what is exact type of objects stored in the
> collections. A list might store int, string, double.
> So it might involve a lot of work to write the python wrapper.
> Do you have any good ideas?
>
> On Fri, Aug 7, 2015 at 7:46 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>> you really can't convert it to the C stuff like that. You need to do:
>>
>> foo = ffi.new("int[]", 13)
>> foo[0] = 12
>> foo[1] = 55
>>
>> etc.
>>
>> You can write nice python wrapper around those things too, if the data
>> is too bare bone.
>>
>> On Fri, Aug 7, 2015 at 1:09 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
>> > Hi Maciej,
>> >
>> > Could you provide some details on how to use cffi buffers? (a simple
>> > demo
>> > code or documents)
>> > I tried the below code but it did not work.
>> >
>> > #python list
>> > x=[1,2,3,4]
>> > #intend to convert to c++ char*, but failed
>> > p=ffi.from_buffer(x)
>> > #intend to covert back to python list, but failed
>> > y=ffi.buffer(p)
>> >
>> > Thanks!
>> >
>> > -Ethan
>> >
>> > On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fijall at gmail.com>
>> > wrote:
>> >>
>> >> you can use cffi buffers (e.g. cffi char*) that you manipulate from
>> >> python using wrappers. They would behave (sort of) like python objects
>> >> and the wrapper code is really not a performance penalty in most cases
>> >>
>> >> On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > Are there any good methods to serialize Pypy object to C++ char* and
>> >> > deserialize to Pypy object from  C++ char*?
>> >> >
>> >> > Our scenario like this:
>> >> > We have at least two C++ process running, and each process embeded
>> >> > Pypy
>> >> > to
>> >> > execute some functions.
>> >> > And we woule like to reuse Pypy object from one process to another
>> >> > process.
>> >> > For primitive objects, int/char/bool/double etc., we could use cffi
>> >> > to
>> >> > pass
>> >> > objects.
>> >> > Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
>> >> >
>> >> > But for complex objects, e.g. list/array/dict/tuple, are there any
>> >> > methods?
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > pypy-dev mailing list
>> >> > pypy-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >> >
>> >
>> >
>
>

From hengha.mao at gmail.com  Fri Aug  7 15:13:27 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Fri, 7 Aug 2015 21:13:27 +0800
Subject: [pypy-dev] serialize and deserialize objects in Pypy
In-Reply-To: <CAK5idxTC+knD3hMDZ0toc1O4BcEUjX+kgXROdiMHWgieBKS_nw@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
 <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
 <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>
 <CAOEzsGBxX_3vkk_fnSg1hnOY7Oog=0_69HgKAC=+vJP7TxQaKQ@mail.gmail.com>
 <CAK5idxTC+knD3hMDZ0toc1O4BcEUjX+kgXROdiMHWgieBKS_nw@mail.gmail.com>
Message-ID: <CAOEzsGCJVCvNKEiNnJTfhivm_Q1o6A4xpG94WuNconC=RuEtQw@mail.gmail.com>

In general, we don't need to parse or process Pypy object in C code, but
only transfer the object from one Pypy enviroment to another Pypy
enviroment.
I am thinking could we have the alternavtive easier way to do this, like
PyObject in Python?

Could we get the address of the Pypy Object and the length of the object,
and memcpy the content from Pypy to C++ and then from C++ to Pypy?


On Fri, Aug 7, 2015 at 8:08 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> yes, you need a serializer protocol, like JSON (or protobuf or
> anything). You generally need to express stuff in a way C can
> understand and read it in python
>
> On Fri, Aug 7, 2015 at 2:06 PM, Yicong Huang <hengha.mao at gmail.com> wrote:
> > The problem is we might need to support several kinds of collection
> objects:
> > list, dict, tuple.
> > In addition, we might not know what is exact type of objects stored in
> the
> > collections. A list might store int, string, double.
> > So it might involve a lot of work to write the python wrapper.
> > Do you have any good ideas?
> >
> > On Fri, Aug 7, 2015 at 7:46 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >>
> >> you really can't convert it to the C stuff like that. You need to do:
> >>
> >> foo = ffi.new("int[]", 13)
> >> foo[0] = 12
> >> foo[1] = 55
> >>
> >> etc.
> >>
> >> You can write nice python wrapper around those things too, if the data
> >> is too bare bone.
> >>
> >> On Fri, Aug 7, 2015 at 1:09 PM, Yicong Huang <hengha.mao at gmail.com>
> wrote:
> >> > Hi Maciej,
> >> >
> >> > Could you provide some details on how to use cffi buffers? (a simple
> >> > demo
> >> > code or documents)
> >> > I tried the below code but it did not work.
> >> >
> >> > #python list
> >> > x=[1,2,3,4]
> >> > #intend to convert to c++ char*, but failed
> >> > p=ffi.from_buffer(x)
> >> > #intend to covert back to python list, but failed
> >> > y=ffi.buffer(p)
> >> >
> >> > Thanks!
> >> >
> >> > -Ethan
> >> >
> >> > On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fijall at gmail.com>
> >> > wrote:
> >> >>
> >> >> you can use cffi buffers (e.g. cffi char*) that you manipulate from
> >> >> python using wrappers. They would behave (sort of) like python
> objects
> >> >> and the wrapper code is really not a performance penalty in most
> cases
> >> >>
> >> >> On Fri, Aug 7, 2015 at 12:19 PM, Yicong Huang <hengha.mao at gmail.com>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Are there any good methods to serialize Pypy object to C++ char*
> and
> >> >> > deserialize to Pypy object from  C++ char*?
> >> >> >
> >> >> > Our scenario like this:
> >> >> > We have at least two C++ process running, and each process embeded
> >> >> > Pypy
> >> >> > to
> >> >> > execute some functions.
> >> >> > And we woule like to reuse Pypy object from one process to another
> >> >> > process.
> >> >> > For primitive objects, int/char/bool/double etc., we could use cffi
> >> >> > to
> >> >> > pass
> >> >> > objects.
> >> >> > Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object
> >> >> >
> >> >> > But for complex objects, e.g. list/array/dict/tuple, are there any
> >> >> > methods?
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > pypy-dev mailing list
> >> >> > pypy-dev at python.org
> >> >> > https://mail.python.org/mailman/listinfo/pypy-dev
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150807/86665244/attachment-0001.html>

From arigo at tunes.org  Fri Aug  7 16:30:08 2015
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 7 Aug 2015 16:30:08 +0200
Subject: [pypy-dev] [python-cffi] Re: serialize and deserialize objects
	in Pypy
In-Reply-To: <CAOEzsGCJVCvNKEiNnJTfhivm_Q1o6A4xpG94WuNconC=RuEtQw@mail.gmail.com>
References: <CAOEzsGCyB98+ign-E5NQchYov8+TvwwSRAH4Hgnxe6BCAK76MA@mail.gmail.com>
 <CAK5idxSKx9=u2LuuSDJF=LZS1XWEtr+8QhZMJ-734+fmDHPZyg@mail.gmail.com>
 <CAOEzsGCObD18Mbfh+3UNH36Gq9FkOscbKVmutQQt=1KRZqZQjg@mail.gmail.com>
 <CAK5idxQPLJaA38bnnfJswQtfRa=mcyGf7=pX07r7YCH2fv2aUA@mail.gmail.com>
 <CAOEzsGBxX_3vkk_fnSg1hnOY7Oog=0_69HgKAC=+vJP7TxQaKQ@mail.gmail.com>
 <CAK5idxTC+knD3hMDZ0toc1O4BcEUjX+kgXROdiMHWgieBKS_nw@mail.gmail.com>
 <CAOEzsGCJVCvNKEiNnJTfhivm_Q1o6A4xpG94WuNconC=RuEtQw@mail.gmail.com>
Message-ID: <CAMSv6X1moBOEU0WL-W8q85P=pAKGPgVL_F-toviWjFri9=4+HQ@mail.gmail.com>

Hi Yicong,

On 7 August 2015 at 15:13, Yicong Huang <hengha.mao at gmail.com> wrote:
> Could we get the address of the Pypy Object and the length of the object,
> and memcpy the content from Pypy to C++ and then from C++ to Pypy?

No.  And if you try the same thing on CPython with PyObjects, it would
crash in 99% of the cases too.


A bient?t,

Armin.

From lac at openend.se  Sat Aug  8 11:32:26 2015
From: lac at openend.se (Laura Creighton)
Date: Sat, 8 Aug 2015 11:32:26 +0200
Subject: [pypy-dev] is the handling of python signals a language
	implementation detail?
Message-ID: <201508080932.t789WQjD001648@fido.openend.se>

I am reading 18.8.1.1 and 18.8.1.2 in
https://docs.python.org/3/library/signal.html which clearly is written
from the CPython point of view.

....A Python signal handler does not get executed inside the low-level
(C) signal handler. Instead, the low-level signal handler sets a flag
which tells the virtual machine to execute the corresponding Python
signal handler at a later point(for example at the next bytecode
instruction) ... Python signal handlers are always executed in the
main Python thread, even if the signal was received in another thread ...

But is this part of the language definition?  Or an implementation detail?

Laura


From n210241048576 at gmail.com  Sun Aug  9 05:51:24 2015
From: n210241048576 at gmail.com (Robert Grosse)
Date: Sat, 8 Aug 2015 20:51:24 -0700
Subject: [pypy-dev] Poor performance for Krakatau
In-Reply-To: <CAMSv6X25HM6aUV=EMbDa20HxKp_G5j7y7WKFGcpbWi9zZmr7Zw@mail.gmail.com>
References: <CALJjGitxXJE3Pd7aUOSGv3Pd3HojqyDSEVrZk9ESoPnAvLF=-A@mail.gmail.com>
 <CAK5idxSDaO0fvAmJbeYujn9X6Cp1SYTcMCkQ6hVSZp2fpDLn4A@mail.gmail.com>
 <CAMSv6X25HM6aUV=EMbDa20HxKp_G5j7y7WKFGcpbWi9zZmr7Zw@mail.gmail.com>
Message-ID: <CALJjGispLCn7ncWE5hNAdRuc-Es658VbMO_O30c7JgSDtZ-gUw@mail.gmail.com>

I decided to update the benchmark today and compare it with pypy2.6. If
anything, the performance issues with Pypy seem to have gotten worse with
2.6.

As before, you can see the benchmark at
https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark
(commit 234edc936b958596d843b91b963c4f61f56f2410)

I also decided to compare it with different number of iterations for warmup.

CPython: 12.65s
Pypy2.6
100 iters: 36.83s
500 iters: 28.68s
1000 iters: 25.56s

So performance does improve when increasing the amount of time spent
warming up, but even with 1000 iterations, which is over a full minute, it
is still half the speed of CPython.








On Mon, Dec 8, 2014 at 7:49 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Robert,
>
> On 10 November 2014 at 05:27, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > I've been looking at krakatau performance for a while, it's almost
> > exclusively warmup time. We are going to address it, I hope rather
> > sooner than later :-)
>
> We added Krakatau to the official benchmark suite.  It turns out not
> to be exclusively warmup time: after the program is fully warmed up,
> it is still almost 2 times slower than CPython.  We'll look at it at
> some point now that it's on speed.pypy.org and annoying us regularly
> :-)
>
>
> A bient?t,
>
> Armin & Carl Friedrich
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150808/b7ed5f6a/attachment.html>

From arigo at tunes.org  Sun Aug  9 10:54:54 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 9 Aug 2015 10:54:54 +0200
Subject: [pypy-dev] do not raise exceptions in __init__
Message-ID: <CAMSv6X0oHNDT7P9fwLKC2zA2SQmdBXk2u33ykarOa_fWA-29gw@mail.gmail.com>

Hi Matti,

Replying to commit "614e585": "do not raise exceptions in __init__"

The __init__ can raise exceptions, it's not the problem here.  This
commit changes (only) something else: it moves around which "*" turn
into an overflow-checking multiplication, and which ones don't.  The
"*" operation in RPython, like the other common arithmetic operations,
turn by default into C code as a "*" that assumes that overflow does
not occur (and the C compiler sometimes optimizes based on this).  On
the other hand, if the "*" operation in RPython appears immediately
inside a "try: except OverflowError:" (and only in this case), then in
C it will also do the same check.

Notably, this checkin fixes this code (by removing it):

        try:
             length = support.product(shape)
        except OverflowError as e:
             ...

which after translation to C would never catch an OverflowError
occurring implicitly inside product().


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Aug  9 17:39:36 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 09 Aug 2015 18:39:36 +0300
Subject: [pypy-dev] do not raise exceptions in __init__
In-Reply-To: <CAMSv6X0oHNDT7P9fwLKC2zA2SQmdBXk2u33ykarOa_fWA-29gw@mail.gmail.com>
References: <CAMSv6X0oHNDT7P9fwLKC2zA2SQmdBXk2u33ykarOa_fWA-29gw@mail.gmail.com>
Message-ID: <55C77438.4040704@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150809/b0823545/attachment.html>

From fijall at gmail.com  Mon Aug 10 10:58:53 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 10 Aug 2015 10:58:53 +0200
Subject: [pypy-dev] Poor performance for Krakatau
In-Reply-To: <CALJjGispLCn7ncWE5hNAdRuc-Es658VbMO_O30c7JgSDtZ-gUw@mail.gmail.com>
References: <CALJjGitxXJE3Pd7aUOSGv3Pd3HojqyDSEVrZk9ESoPnAvLF=-A@mail.gmail.com>
 <CAK5idxSDaO0fvAmJbeYujn9X6Cp1SYTcMCkQ6hVSZp2fpDLn4A@mail.gmail.com>
 <CAMSv6X25HM6aUV=EMbDa20HxKp_G5j7y7WKFGcpbWi9zZmr7Zw@mail.gmail.com>
 <CALJjGispLCn7ncWE5hNAdRuc-Es658VbMO_O30c7JgSDtZ-gUw@mail.gmail.com>
Message-ID: <CAK5idxT6xyG5LzO0yXy7xewcPCNBzc-Usn_r3Kr1JkzqizGu_Q@mail.gmail.com>

you *eventually* get somewhere. Try running this:

PYPYLOG=jit-summary:- pypy <other args>

and see with what number of iterations the warmup time stops growing.
Also we have vmprof these days so maybe it's a good time to look into
the performance again, I'll keep that in mind

On Sun, Aug 9, 2015 at 5:51 AM, Robert Grosse <n210241048576 at gmail.com> wrote:
> I decided to update the benchmark today and compare it with pypy2.6. If
> anything, the performance issues with Pypy seem to have gotten worse with
> 2.6.
>
> As before, you can see the benchmark at
> https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (commit
> 234edc936b958596d843b91b963c4f61f56f2410)
>
> I also decided to compare it with different number of iterations for warmup.
>
> CPython: 12.65s
> Pypy2.6
> 100 iters: 36.83s
> 500 iters: 28.68s
> 1000 iters: 25.56s
>
> So performance does improve when increasing the amount of time spent warming
> up, but even with 1000 iterations, which is over a full minute, it is still
> half the speed of CPython.
>
>
>
>
>
>
>
>
> On Mon, Dec 8, 2014 at 7:49 AM, Armin Rigo <arigo at tunes.org> wrote:
>>
>> Hi Robert,
>>
>> On 10 November 2014 at 05:27, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> > I've been looking at krakatau performance for a while, it's almost
>> > exclusively warmup time. We are going to address it, I hope rather
>> > sooner than later :-)
>>
>> We added Krakatau to the official benchmark suite.  It turns out not
>> to be exclusively warmup time: after the program is fully warmed up,
>> it is still almost 2 times slower than CPython.  We'll look at it at
>> some point now that it's on speed.pypy.org and annoying us regularly
>> :-)
>>
>>
>> A bient?t,
>>
>> Armin & Carl Friedrich
>
>

From fijall at gmail.com  Mon Aug 10 11:01:31 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 10 Aug 2015 11:01:31 +0200
Subject: [pypy-dev] is the handling of python signals a language
 implementation detail?
In-Reply-To: <201508080932.t789WQjD001648@fido.openend.se>
References: <201508080932.t789WQjD001648@fido.openend.se>
Message-ID: <CAK5idxRq_0A0N2mHYJVyHiCROu9fn1=FXyPpkVLz6p_1aTck4Q@mail.gmail.com>

I think this is a part of the language definition, at the very least
pypy does the same. The reason is that you can't write python code
that can be safely executed at an arbitrary point in time (it's hard
enough to write runnable-from-a-signal handler C code), so you defer
it until at least the bytecode boundary so the internal structures are
in a somewhat consistent shape

cheers,
fijal

From william.leslie.ttg at gmail.com  Mon Aug 10 11:43:40 2015
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Mon, 10 Aug 2015 19:43:40 +1000
Subject: [pypy-dev] is the handling of python signals a language
 implementation detail?
In-Reply-To: <201508080932.t789WQjD001648@fido.openend.se>
References: <201508080932.t789WQjD001648@fido.openend.se>
Message-ID: <CAHgd1hGL-QOmeoJKk4EihNVkRuzqE6bGGqMrh4tLtdwSM6N4iw@mail.gmail.com>

On 8 August 2015 at 19:32, Laura Creighton <lac at openend.se> wrote:
> I am reading 18.8.1.1 and 18.8.1.2 in
> https://docs.python.org/3/library/signal.html which clearly is written
> from the CPython point of view.
>
> ....A Python signal handler does not get executed inside the low-level
> (C) signal handler. Instead, the low-level signal handler sets a flag
> which tells the virtual machine to execute the corresponding Python
> signal handler at a later point(for example at the next bytecode
> instruction) ... Python signal handlers are always executed in the
> main Python thread, even if the signal was received in another thread ...
>
> But is this part of the language definition?  Or an implementation detail?

Since Python does not define a concept of signal-safe, I guess in
practice the handler must be observed to run at a later point.  The
thread that a handler runs in can be semantically significant; I think
you should be able to rely on the behaviour of the signal module in
this regard.  Different APIs and semantic-preserving optimisations
aside.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

From hengha.mao at gmail.com  Tue Aug 11 14:38:24 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Tue, 11 Aug 2015 20:38:24 +0800
Subject: [pypy-dev] pypy callback function pointer become invalid
Message-ID: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>

Hi,

We embeded Pypy in C++ code, and used the callback function pointer to
execute Python function.
And we noticed that, the callback function pointers might become invalid as
the memory usage grows.
Here is the piece of test code:

    for(int i = 0; i < LOOP; i++){
       //read python code from file
       char* pyBuffer = readPyFile("test_Lower.py");
       pypy_execute_source_ptr(pyBuffer, &ptr);
       callers[i] = ptr.pyPtrs[0];
    }
    //check 1st function pointer and call python function
    caller = callers[0];
    ret = (*((char *(*)(char *))caller))("A");
    printf("%s\n", ret);

In the experiment, if the LOOP >150, caller becomes invalid, the callback
to python function could not succed, and "Segmentation fault" error was
reported.
For more complex Python function, the valid LOOP might be smaller.

I am wondering whether the error was caused by GC in Pypy that recycled the
function pointers.
In python fill_api() code, I have used a gloabl list to append function
pointer, but it did not help.

#global list
no_gc = []
def fill_api(ptr):
   api = ffi.cast("struct pypyAPI*", ptr)
   api.pyPtrs[0] = test_Lower
   no_gc.append(api)
   no_gc.append(test_Lower)
   return api

And are there any methods to prevent such errors?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150811/a5fd7135/attachment.html>

From vaibhav_sood at persistent.com  Wed Aug 12 14:15:07 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Wed, 12 Aug 2015 12:15:07 +0000
Subject: [pypy-dev] pypy platform specific tests
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025AE106@HJ-MBX1.persistent.co.in>

Hi,

I am just starting off with pypy, I tried building pypy and running some tests from the test suite earlier and had a query regarding the tests. I was wondering if there are any tests in the test suite which help to test the built binary? I ask this because I believe there is platform specific code in the binary and if so are there are any platform specific tests (I am assuming they will be .c files and not .py files) in the code?

Thanks!
Vaibhav

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150812/fd024db4/attachment.html>

From ndbecker2 at gmail.com  Thu Aug 13 14:17:14 2015
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 13 Aug 2015 08:17:14 -0400
Subject: [pypy-dev] [newb] failed build 2.6.0 on fedora 22
Message-ID: <mqi1sb$p0b$1@ger.gmane.org>

I thought I'd try grabbing the fedora 2.4.0 srpm and see if it would build 
2.6.0, but didn't get far:

PYPY_USESSION_DIR=/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal 
PYPY_USESSION_BASENAME=pypy python ../../rpython/bin/rpython --output=pypy 
--gcrootfinder=shadowstack -Ojit targetpypystandalone
[translation:info] 2.7.10 (default, Jul  5 2015, 14:15:43)
[translation:info] [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)]
[platform:msg] Set platform with 'host' cc=None, using cc='gcc', 
version='Unknown'
[translation:info] Translating target as defined by targetpypystandalone
[platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/gcctest.o
[platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/gcctest
[platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/gcctest.o
[platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/gcctest
[platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c 
/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/platcheck_0.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/platcheck_0.o
[platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c 
/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/rpython/translator/c/src/thread.c -o 
/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/rpython/translator/c/src/thread.o
[platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/usession-pypy-1/platcheck_0.o 
/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/rpython/translator/c/src/thread.o -pthread -Wl,--export-dynamic -lrt 
-o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
pypy-1/platcheck_0
[translation:info] Error:
[translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/rpython/translator/goal/translate.py", line 284, in main
[translation:info]     default_goal='compile')
[translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/rpython/translator/driver.py", line 562, in from_targetspec
[translation:info]     spec = target(driver, args)
[translation:info]    File "targetpypystandalone.py", line 244, in target
[translation:info]     raise Exception("Cannot use the --output option with 
PyPy "
[translation:ERROR] Exception: Cannot use the --output option with PyPy when 
--shared is on (it is by default). See issue #1971.
[translation] start debugger...
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
src/pypy/goal/targetpypystandalone.py(244)target()
-> raise Exception("Cannot use the --output option with PyPy "


From fijall at gmail.com  Thu Aug 13 15:42:19 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 13 Aug 2015 15:42:19 +0200
Subject: [pypy-dev] [newb] failed build 2.6.0 on fedora 22
In-Reply-To: <mqi1sb$p0b$1@ger.gmane.org>
References: <mqi1sb$p0b$1@ger.gmane.org>
Message-ID: <CAK5idxTPkUW7dP2tHNmHOkSsRP84YwrFTVSMHDZXkc3RO-k57A@mail.gmail.com>

Please don't use --output :-) The reason is that because --shared is
on you might get a new pypy-xxyyz file, but the libpypy-c.so (which is
not easy to rename) will overwrite the existing one anyway.

On Thu, Aug 13, 2015 at 2:17 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I thought I'd try grabbing the fedora 2.4.0 srpm and see if it would build
> 2.6.0, but didn't get far:
>
> PYPY_USESSION_DIR=/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal
> PYPY_USESSION_BASENAME=pypy python ../../rpython/bin/rpython --output=pypy
> --gcrootfinder=shadowstack -Ojit targetpypystandalone
> [translation:info] 2.7.10 (default, Jul  5 2015, 14:15:43)
> [translation:info] [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)]
> [platform:msg] Set platform with 'host' cc=None, using cc='gcc',
> version='Unknown'
> [translation:info] Translating target as defined by targetpypystandalone
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
> o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
> o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/platcheck_0.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/platcheck_0.o
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/c/src/thread.c -o
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/rpython/translator/c/src/thread.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/platcheck_0.o
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/rpython/translator/c/src/thread.o -pthread -Wl,--export-dynamic -lrt
> -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/platcheck_0
> [translation:info] Error:
> [translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/goal/translate.py", line 284, in main
> [translation:info]     default_goal='compile')
> [translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/driver.py", line 562, in from_targetspec
> [translation:info]     spec = target(driver, args)
> [translation:info]    File "targetpypystandalone.py", line 244, in target
> [translation:info]     raise Exception("Cannot use the --output option with
> PyPy "
> [translation:ERROR] Exception: Cannot use the --output option with PyPy when
> --shared is on (it is by default). See issue #1971.
> [translation] start debugger...
>> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/targetpypystandalone.py(244)target()
> -> raise Exception("Cannot use the --output option with PyPy "
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From matti.picus at gmail.com  Thu Aug 13 18:13:41 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 13 Aug 2015 19:13:41 +0300
Subject: [pypy-dev] [newb] failed build 2.6.0 on fedora 22
In-Reply-To: <CAK5idxTPkUW7dP2tHNmHOkSsRP84YwrFTVSMHDZXkc3RO-k57A@mail.gmail.com>
References: <mqi1sb$p0b$1@ger.gmane.org>
 <CAK5idxTPkUW7dP2tHNmHOkSsRP84YwrFTVSMHDZXkc3RO-k57A@mail.gmail.com>
Message-ID: <55CCC235.4090308@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150813/b0e5fe53/attachment.html>

From mstuchli at redhat.com  Fri Aug 14 13:37:33 2015
From: mstuchli at redhat.com (Matej Stuchlik)
Date: Fri, 14 Aug 2015 07:37:33 -0400 (EDT)
Subject: [pypy-dev] [newb] failed build 2.6.0 on fedora 22
In-Reply-To: <mqi1sb$p0b$1@ger.gmane.org>
References: <mqi1sb$p0b$1@ger.gmane.org>
Message-ID: <1582372910.9708504.1439552253278.JavaMail.zimbra@redhat.com>

FYI fixed .spec for pypy 2.6.0 is being worked on, I'm hoping it will
be done by next week, take a look at [0] for some more info. :)

Matt

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1239815

----- Original Message -----
> From: "Neal Becker" <ndbecker2 at gmail.com>
> To: pypy-dev at python.org
> Sent: Thursday, August 13, 2015 2:17:14 PM
> Subject: [pypy-dev] [newb] failed build 2.6.0 on fedora 22
> 
> I thought I'd try grabbing the fedora 2.4.0 srpm and see if it would build
> 2.6.0, but didn't get far:
> 
> PYPY_USESSION_DIR=/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal
> PYPY_USESSION_BASENAME=pypy python ../../rpython/bin/rpython --output=pypy
> --gcrootfinder=shadowstack -Ojit targetpypystandalone
> [translation:info] 2.7.10 (default, Jul  5 2015, 14:15:43)
> [translation:info] [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)]
> [platform:msg] Set platform with 'host' cc=None, using cc='gcc',
> version='Unknown'
> [translation:info] Translating target as defined by targetpypystandalone
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
> o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/gcctest.o -pthread -Wl,--export-dynamic -lrt -
> o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/gcctest
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/platcheck_0.c -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/platcheck_0.o
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-
> unused -I/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/rpython/translator/c
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/c/src/thread.c -o
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/rpython/translator/c/src/thread.o
> [platform:execute] gcc /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/usession-pypy-1/platcheck_0.o
> /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/rpython/translator/c/src/thread.o -pthread -Wl,--export-dynamic -lrt
> -o /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-src/pypy/goal/usession-
> pypy-1/platcheck_0
> [translation:info] Error:
> [translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/goal/translate.py", line 284, in main
> [translation:info]     default_goal='compile')
> [translation:info]    File "/home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/rpython/translator/driver.py", line 562, in from_targetspec
> [translation:info]     spec = target(driver, args)
> [translation:info]    File "targetpypystandalone.py", line 244, in target
> [translation:info]     raise Exception("Cannot use the --output option with
> PyPy "
> [translation:ERROR] Exception: Cannot use the --output option with PyPy when
> --shared is on (it is by default). See issue #1971.
> [translation] start debugger...
> > /home/nbecker/rpmbuild/BUILD/pypy-2.6.0-
> src/pypy/goal/targetpypystandalone.py(244)target()
> -> raise Exception("Cannot use the --output option with PyPy "
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 

From lac at openend.se  Sat Aug 15 12:40:33 2015
From: lac at openend.se (Laura Creighton)
Date: Sat, 15 Aug 2015 12:40:33 +0200
Subject: [pypy-dev] memory control in Python (fwd)
Message-ID: <201508151040.t7FAeXhM027631@fido.openend.se>

This showed up on python-list.
https://mail.python.org/pipermail/python-list/2015-August/695441.html
(if you want to read his attatchments, but, grrr you need libreoffice
to read them).

Cursory googling for CPLEX indicates it's written in C++.

Do we have any joy for this man?

Laura

------- Forwarded Message

Return-Path: <python-list-bounces+lac=openend.se at python.org>
Received: from mail.python.org (mail.python.org [82.94.164.166])
Subject: memory control in Python
From: Ping Liu <yanzhipingliu at gmail.com>
To: python-list at python.org

Dear All,

I am working on an optimization problem, where we are trying to minimize
some indicators like energy usage, energy cost, CO2 emission. In this
problem, we have a bunch of energy conversion technologies for electricity
and thermal purpose, such as heat pump, boiler, chiller, etc.. We are
trying to model it with a one year time period. the current time step is
one hour.

We have preselected the  candidate technologies to exclude those we don't
want to study so that the problem size could be reduced with a limited
candidate technologies. In the current case study, we only analyze the
electric chiller and heat pump to supply the cooling load, while power grid
will supply the electricity for all electric loads. There are binary
variables regarding installation decisions of technologies and continuous
variables related to installation capacity and hourly operational
decisions.

For small cases, Python works well. But if we consider longer time period.
then it would fail due to the memory usage issues. We have tested several
case studies to check the memory use for different time period, including
1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
are feasible while the last case gives out the memory error.

When we are testing the forth case, the memory error comes out while
creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.

The solver is CPLEX (academic). It turns out that the solver is taking a
lot of memory as you can see in the memory test report. for the first three
cases, different memory usage is observed, and it grows up dramatically
with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
19.5 MB; 3) solver memory usage: 830.0742 MB.

Based on my observations, I have some of the following questions regarding
1) In order to create the optimization problem (Aeq, Aineq), how can we
reduce the memory usage in python programming?  2) how can I reduce the
memory usage of different solvers in Python? Why would the memory decrease
for the CPLEX solver within the 24-hours-in-one-day case compared with the
case 1?

Thanks in advance,
Ping


From yury at shurup.com  Sat Aug 15 13:01:26 2015
From: yury at shurup.com (Yury V. Zaytsev)
Date: Sat, 15 Aug 2015 13:01:26 +0200
Subject: [pypy-dev] memory control in Python (fwd)
In-Reply-To: <201508151040.t7FAeXhM027631@fido.openend.se>
References: <201508151040.t7FAeXhM027631@fido.openend.se>
Message-ID: <1439636486.21639.187.camel@newpride>

On Sat, 2015-08-15 at 12:40 +0200, Laura Creighton wrote:
> 
> Cursory googling for CPLEX indicates it's written in C++. Do we have
> any joy for this man?

CPLEX is a commercial linear & quadratic optimization package currently
developed by IBM; the Python bindings are just an interface to the C API
that is provided by the optimization kernel.

It has to be analyzed where the memory usage comes from: if it's CPLEX
kernel itself, then there is no joy (well, at least if it's being used
correctly), and if it's Python (which is doubtful, but possible) there
might, or might not be any.

Either way, I guess it's his job to find this out :-)

-- 
Sincerely yours,
Yury V. Zaytsev



From arigo at tunes.org  Sat Aug 15 16:09:16 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 15 Aug 2015 16:09:16 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
Message-ID: <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>

Hi Yicong,

On 11 August 2015 at 14:38, Yicong Huang <hengha.mao at gmail.com> wrote:
>        pypy_execute_source_ptr(pyBuffer, &ptr);

The documentation recommends to call pypy_execute_source_ptr() only
once.  I've fixed anyway the particular problem you are encountering
when calling it multiple times, in 20f74886a28e.


A bient?t,

Armin.

From fijall at gmail.com  Sat Aug 15 18:24:40 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 Aug 2015 18:24:40 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
Message-ID: <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>

On Sat, Aug 15, 2015 at 4:09 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Yicong,
>
> On 11 August 2015 at 14:38, Yicong Huang <hengha.mao at gmail.com> wrote:
>>        pypy_execute_source_ptr(pyBuffer, &ptr);
>
> The documentation recommends to call pypy_execute_source_ptr() only
> once.  I've fixed anyway the particular problem you are encountering
> when calling it multiple times, in 20f74886a28e.
>
>
> A bient?t,
>
> Armin.

Note that if you execute it many many times, you will leak memory.

From hengha.mao at gmail.com  Sun Aug 16 06:12:50 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Sun, 16 Aug 2015 12:12:50 +0800
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
Message-ID: <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>

Thanks for reminding!
I looked at the code, and observed the gloal new dict might leak memory.

Calling pypy_execute_source_ptr() multiple times might be a common usage:
you might have several python files to execute, or you might get python
code segment from input one by one ...
I am thinking could we have better ways to do such calls, e.g.:

pypy_initialize();
pypy_execute_source_ptr()
#call many times
...
#clean up
pypy_execute_end();
...
#to start a clean new session
pypy_initialize();
pypy_execute_source_ptr();
...
pypy_execute_end();



On Sun, Aug 16, 2015 at 12:24 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> On Sat, Aug 15, 2015 at 4:09 PM, Armin Rigo <arigo at tunes.org> wrote:
> > Hi Yicong,
> >
> > On 11 August 2015 at 14:38, Yicong Huang <hengha.mao at gmail.com> wrote:
> >>        pypy_execute_source_ptr(pyBuffer, &ptr);
> >
> > The documentation recommends to call pypy_execute_source_ptr() only
> > once.  I've fixed anyway the particular problem you are encountering
> > when calling it multiple times, in 20f74886a28e.
> >
> >
> > A bient?t,
> >
> > Armin.
>
> Note that if you execute it many many times, you will leak memory.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150816/ecf7f758/attachment.html>

From fijall at gmail.com  Sun Aug 16 09:42:02 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 16 Aug 2015 09:42:02 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
Message-ID: <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>

On Sun, Aug 16, 2015 at 6:12 AM, Yicong Huang <hengha.mao at gmail.com> wrote:
> Thanks for reminding!
> I looked at the code, and observed the gloal new dict might leak memory.
>
> Calling pypy_execute_source_ptr() multiple times might be a common usage:
> you might have several python files to execute, or you might get python code
> segment from input one by one ...
> I am thinking could we have better ways to do such calls, e.g.:

That use case is much better handled from python (e.g. using exec() or
import or any other way to execute more python code)

From admalledd at gmail.com  Sun Aug 16 10:01:08 2015
From: admalledd at gmail.com (Eric Driggers)
Date: Sun, 16 Aug 2015 01:01:08 -0700
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
Message-ID: <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>

Agreed, have been doing basically that for some internal code, as the
docs kind of hint:

http://pypy.readthedocs.org/en/latest/embedding.html#more-complete-example

Basically call the pypy_execute_source_ptr() once, passing in a API
struct to fill out for low-level call-backs. (And passing in at the
same time functions for python to call to C).

On Sun, Aug 16, 2015 at 12:42 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Sun, Aug 16, 2015 at 6:12 AM, Yicong Huang <hengha.mao at gmail.com> wrote:
>> Thanks for reminding!
>> I looked at the code, and observed the gloal new dict might leak memory.
>>
>> Calling pypy_execute_source_ptr() multiple times might be a common usage:
>> you might have several python files to execute, or you might get python code
>> segment from input one by one ...
>> I am thinking could we have better ways to do such calls, e.g.:
>
> That use case is much better handled from python (e.g. using exec() or
> import or any other way to execute more python code)
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From fijall at gmail.com  Sun Aug 16 11:00:10 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 16 Aug 2015 11:00:10 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
Message-ID: <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>

Think about the API that way: there is only one "entry" python program
that you run and you can still execute arbitrary code.

On Sun, Aug 16, 2015 at 10:01 AM, Eric Driggers <admalledd at gmail.com> wrote:
> Agreed, have been doing basically that for some internal code, as the
> docs kind of hint:
>
> http://pypy.readthedocs.org/en/latest/embedding.html#more-complete-example
>
> Basically call the pypy_execute_source_ptr() once, passing in a API
> struct to fill out for low-level call-backs. (And passing in at the
> same time functions for python to call to C).
>
> On Sun, Aug 16, 2015 at 12:42 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Sun, Aug 16, 2015 at 6:12 AM, Yicong Huang <hengha.mao at gmail.com> wrote:
>>> Thanks for reminding!
>>> I looked at the code, and observed the gloal new dict might leak memory.
>>>
>>> Calling pypy_execute_source_ptr() multiple times might be a common usage:
>>> you might have several python files to execute, or you might get python code
>>> segment from input one by one ...
>>> I am thinking could we have better ways to do such calls, e.g.:
>>
>> That use case is much better handled from python (e.g. using exec() or
>> import or any other way to execute more python code)
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

From fijall at gmail.com  Sun Aug 16 11:20:43 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 16 Aug 2015 11:20:43 +0200
Subject: [pypy-dev] weird armhf-v7 errors
Message-ID: <CAK5idxRJqaEuA8YGyXEAYD9vw+BJ47sLgmUgPNhKmD-rasJoZw@mail.gmail.com>

Hi

I wonder why this:
http://buildbot.pypy.org/builders/pypy-c-jit-linux-armhf-v7/builds/745/steps/shell_11/logs/stdio

has errors that don't show up in the summary anywhere?

does anyone know?

Cheers,
fijal

From arigo at tunes.org  Sun Aug 16 11:38:13 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 16 Aug 2015 11:38:13 +0200
Subject: [pypy-dev] weird armhf-v7 errors
In-Reply-To: <CAK5idxRJqaEuA8YGyXEAYD9vw+BJ47sLgmUgPNhKmD-rasJoZw@mail.gmail.com>
References: <CAK5idxRJqaEuA8YGyXEAYD9vw+BJ47sLgmUgPNhKmD-rasJoZw@mail.gmail.com>
Message-ID: <CAMSv6X2UoC9k3gUbgRy1z4yhiteB2smefJN8wzTJb8GgzHdv0Q@mail.gmail.com>

Hi Fijal,

On 16 August 2015 at 11:20, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I wonder why this:
> http://buildbot.pypy.org/builders/pypy-c-jit-linux-armhf-v7/builds/745/steps/shell_11/logs/stdio
>
> has errors that don't show up in the summary anywhere?

Because it's not done.  As far as I know, the summary only shows up
when the whole build is finished.


A bient?t,

Armin.

From fijall at gmail.com  Sun Aug 16 12:13:39 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 16 Aug 2015 12:13:39 +0200
Subject: [pypy-dev] weird armhf-v7 errors
In-Reply-To: <CAMSv6X2UoC9k3gUbgRy1z4yhiteB2smefJN8wzTJb8GgzHdv0Q@mail.gmail.com>
References: <CAK5idxRJqaEuA8YGyXEAYD9vw+BJ47sLgmUgPNhKmD-rasJoZw@mail.gmail.com>
 <CAMSv6X2UoC9k3gUbgRy1z4yhiteB2smefJN8wzTJb8GgzHdv0Q@mail.gmail.com>
Message-ID: <CAK5idxQzyY-dpCp0jTanORo4PinSu5+H49ck4reZSCwJWmDvyQ@mail.gmail.com>

I can't find a green version of that buildbot on the scroll list
though, are you sure it's the only problem?

On Sun, Aug 16, 2015 at 11:38 AM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Fijal,
>
> On 16 August 2015 at 11:20, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I wonder why this:
>> http://buildbot.pypy.org/builders/pypy-c-jit-linux-armhf-v7/builds/745/steps/shell_11/logs/stdio
>>
>> has errors that don't show up in the summary anywhere?
>
> Because it's not done.  As far as I know, the summary only shows up
> when the whole build is finished.
>
>
> A bient?t,
>
> Armin.

From arigo at tunes.org  Sun Aug 16 13:23:16 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 16 Aug 2015 13:23:16 +0200
Subject: [pypy-dev] pypy platform specific tests
In-Reply-To: <4176FFD2EDA9C24CA9AB011273CFDDC8025AE106@HJ-MBX1.persistent.co.in>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AE106@HJ-MBX1.persistent.co.in>
Message-ID: <CAMSv6X2CufYu=-=Lf+=4quXqRyNFyyqoeLSUZx-bcjOXWjX=fA@mail.gmail.com>

Hi Vaibhav,

On 12 August 2015 at 14:15, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
> wondering if there are any tests in the test suite which help to test the
> built binary? I ask this because I believe there is platform specific code
> in the binary and if so are there are any platform specific tests (I am
> assuming they will be .c files and not .py files) in the code?

No, our tests are almost always .py files.  If you have some more
precise query, like "where is code xyz tested?", I can answer more
precisely.

We run some tests with the pypy binaries, like CPython's own tests and
more.  Obviously it requires first getting a pypy in binary form.
Then see e.g. http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/2722
.  The last four sections (numbers 10 to 13) are running some tests;
for example section number 11 is the CPython tests.  You can click on
"stdio" to see the exact command used to run the tests (in the first
line).


A bient?t,

Armin.

From arigo at tunes.org  Sun Aug 16 13:17:09 2015
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 16 Aug 2015 13:17:09 +0200
Subject: [pypy-dev] do not raise exceptions in __init__
In-Reply-To: <55C77438.4040704@gmail.com>
References: <CAMSv6X0oHNDT7P9fwLKC2zA2SQmdBXk2u33ykarOa_fWA-29gw@mail.gmail.com>
 <55C77438.4040704@gmail.com>
Message-ID: <CAMSv6X3ocP20w06uAhGYcW1aVxxkcQhQynVwU-NLn0c+RT=UFw@mail.gmail.com>

Hi Matti,

On 9 August 2015 at 17:39, Matti Picus <matti.picus at gmail.com> wrote:
> Are you saying that the ovfcheck will only raise an exception if it is
> immediately surrounded with a try: except OverflowError clause, and not
> raise an exception if the clause wraps the product(shape) function call

Sorry, after more digging, I'm wrong: ovfcheck() will always raise
OverflowError, which can be caught some levels up the stack.


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Aug 16 22:36:24 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 16 Aug 2015 23:36:24 +0300
Subject: [pypy-dev] arm testing buildbot builders
Message-ID: <55D0F448.9040405@gmail.com>

The TranslatedTesting target was not copying the cffi import libraries 
in nightly runs.
I committed an (untested, sorry) change, 3275430ea6da, to the buildbot 
repo, could someone merge that to the build master?

BTW, it turns out that the hg pypy repo for 
pypy-c-app-level-linux-armhf-v7 and pypy-c-jit-linux-armhf-v7 somehow 
got the version numbers out of sync with upstream, which caused the test 
results to be ignored on the summary page. I hope I have fixed that too.

Matti

From hengha.mao at gmail.com  Mon Aug 17 08:40:02 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Mon, 17 Aug 2015 14:40:02 +0800
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
 <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
Message-ID: <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>

The problem is there is *only one* entry python program.
>From the document example (
http://pypy.readthedocs.org/en/latest/embedding.html#more-complete-example),
yes, we could use a structure to wrap severl python callback function
pointers in one execution.
However, the issue is that we might not be able to get all python functions
that would be executed at the beginning.

Thinking of typical scenario of interactive usage:
1. Case 1: c++ call to python function -> python execute and return results
-> c++ process python results and choose the next python function to execute
If there is *only one* entry, we need to wrap all possible python functions
that might be called at one time.

2. Case 2: A client-server service that also serve to run python code
submited by users
C/S server infrastuture were written in C++, and the python code is
submited by user. The client submits python code to server, and server runs
the code and return result to the client. In this case, we've no ideas what
python code might execute in future.
Alternatively, we could create and destory a C++ process every time to
embed pypy, but it is inefficient.

On Sun, Aug 16, 2015 at 5:00 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Think about the API that way: there is only one "entry" python program
> that you run and you can still execute arbitrary code.
>
> On Sun, Aug 16, 2015 at 10:01 AM, Eric Driggers <admalledd at gmail.com>
> wrote:
> > Agreed, have been doing basically that for some internal code, as the
> > docs kind of hint:
> >
> >
> http://pypy.readthedocs.org/en/latest/embedding.html#more-complete-example
> >
> > Basically call the pypy_execute_source_ptr() once, passing in a API
> > struct to fill out for low-level call-backs. (And passing in at the
> > same time functions for python to call to C).
> >
> > On Sun, Aug 16, 2015 at 12:42 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >> On Sun, Aug 16, 2015 at 6:12 AM, Yicong Huang <hengha.mao at gmail.com>
> wrote:
> >>> Thanks for reminding!
> >>> I looked at the code, and observed the gloal new dict might leak
> memory.
> >>>
> >>> Calling pypy_execute_source_ptr() multiple times might be a common
> usage:
> >>> you might have several python files to execute, or you might get
> python code
> >>> segment from input one by one ...
> >>> I am thinking could we have better ways to do such calls, e.g.:
> >>
> >> That use case is much better handled from python (e.g. using exec() or
> >> import or any other way to execute more python code)
> >> _______________________________________________
> >> pypy-dev mailing list
> >> pypy-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150817/d5689bbf/attachment.html>

From arigo at tunes.org  Mon Aug 17 10:08:48 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Aug 2015 10:08:48 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
 <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
 <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>
Message-ID: <CAMSv6X205KnVWjBjL2Z5pPf2c1zxyNHD_f0ffRCLEwFXZRFq4w@mail.gmail.com>

Hi Yicong,

On 17 August 2015 at 08:40, Yicong Huang <hengha.mao at gmail.com> wrote:
> yes, we could use a structure to wrap severl python callback function
> pointers in one execution.
> However, the issue is that we might not be able to get all python functions
> that would be executed at the beginning.

The basic idea is to define whatever API you need with just one call
to pypy_execute_source_ptr().  If you want to be able to compile and
execute arbitrary Python functions, just make that feature available
in your API.  This is the same idea as PyRun_SimpleString(), which is
*one* API function but lets you execute arbitrary Python code.
Example:


struct API {
    long (*run_function)(char *);
};
struct API api;   /* global var */

int initialize_api(void) {    /* run this only once */
    static char source[] =
        "import sys; sys.path.insert(0, '.'); "
        "import interface; interface.fill_api(c_argument)";
    pypy_execute_source_ptr(source, &api);
}

/* then execute 'api.run_function(some_source_code)' any number of times */


# file "interface.py"

import cffi

ffi = cffi.FFI()
ffi.cdef('''
struct API {
    long (*run_function)(char *);
};
''')

@ffi.callback("long(char *)")
def run_function(p_src):
    src = ffi.string(p_src)
    # here 'src' is a string, we can do whatever we want with it---like eval()
    return eval(src, {})

def fill_api(ptr):
    global api
    api = ffi.cast("struct API*", ptr)
    api.run_function = run_function


---
Armin

From arigo at tunes.org  Mon Aug 17 10:13:58 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Aug 2015 10:13:58 +0200
Subject: [pypy-dev] arm testing buildbot builders
In-Reply-To: <55D0F448.9040405@gmail.com>
References: <55D0F448.9040405@gmail.com>
Message-ID: <CAMSv6X3QzkqphkwzcBONRYAK2pws4tmq2fubGjQp0A5=NMdLLg@mail.gmail.com>

Hi Matti,

On 16 August 2015 at 22:36, Matti Picus <matti.picus at gmail.com> wrote:
> I committed an (untested, sorry) change, 3275430ea6da, to the buildbot repo,
> could someone merge that to the build master?

Done.  The build master will restart when all slaves are finished.


A bient?t,

Armin.

From vaibhav_sood at persistent.com  Mon Aug 17 12:52:54 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Mon, 17 Aug 2015 10:52:54 +0000
Subject: [pypy-dev] pypy platform specific tests
In-Reply-To: <CAMSv6X2CufYu=-=Lf+=4quXqRyNFyyqoeLSUZx-bcjOXWjX=fA@mail.gmail.com>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AE106@HJ-MBX1.persistent.co.in>
 <CAMSv6X2CufYu=-=Lf+=4quXqRyNFyyqoeLSUZx-bcjOXWjX=fA@mail.gmail.com>
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025BA00D@HJ-MBX1.persistent.co.in>

Thanks Armin! I was looking at the stacklet code under "pypy/rpython/translator/c/src/stacklet" as an example , I did find a tests.c file in the same folder but couldn?t find it being invoked by buildbot or any .py file in the test suite (in the Makefile there is a 'run-all-tests' target but doing a 'make run-all-tests' gives an error) so was wondering how this code gets tested?

Thanks again!

Regards,
Vaibhav

-----Original Message-----
From: armin.rigo at gmail.com [mailto:armin.rigo at gmail.com] On Behalf Of Armin Rigo
Sent: Sunday, August 16, 2015 4:53 PM
To: Vaibhav Sood
Cc: pypy-dev at python.org
Subject: Re: [pypy-dev] pypy platform specific tests

Hi Vaibhav,

On 12 August 2015 at 14:15, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
> wondering if there are any tests in the test suite which help to test 
> the built binary? I ask this because I believe there is platform 
> specific code in the binary and if so are there are any platform 
> specific tests (I am assuming they will be .c files and not .py files) in the code?

No, our tests are almost always .py files.  If you have some more precise query, like "where is code xyz tested?", I can answer more precisely.

We run some tests with the pypy binaries, like CPython's own tests and more.  Obviously it requires first getting a pypy in binary form.
Then see e.g. http://buildbot.pypy.org/builders/pypy-c-jit-linux-x86-64/builds/2722
.  The last four sections (numbers 10 to 13) are running some tests; for example section number 11 is the CPython tests.  You can click on "stdio" to see the exact command used to run the tests (in the first line).


A bient?t,

Armin.

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


From hengha.mao at gmail.com  Mon Aug 17 12:59:52 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Mon, 17 Aug 2015 18:59:52 +0800
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAMSv6X205KnVWjBjL2Z5pPf2c1zxyNHD_f0ffRCLEwFXZRFq4w@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
 <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
 <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>
 <CAMSv6X205KnVWjBjL2Z5pPf2c1zxyNHD_f0ffRCLEwFXZRFq4w@mail.gmail.com>
Message-ID: <CAOEzsGDfzs0Dr5DLQKcvoh3C-0NQf4KKdf9Udv3dKySj6Wyf4A@mail.gmail.com>

Hi Armin,

Great thanks for the demo code!
I think the approach should work, but there might be a lot of work to do in
"run_function" to perform a simple python function call.
Assuming we would like to do the simple work as original
pypy_execute_source_ptr()
does, with such approch, we need to pass python code as char* to callback
function "def run_function(p_src)".
In theory we could do everything, even much more complex than eval().
However, to perform the python function call, we need to parse char*(python
code), a python code lexer, praser, JIT and etc, in other words do all the
work that  pypy_execute_source() did. The only difference is we don't need
pypy_execute_source_ptr(), but pypy_execute_source().

I think pypy_execute_source() plays the same role as eval() in your example
code.
So why don't PyPy expose and API to do such job ? ( pypy_execute_source(),
either python or C)

On Mon, Aug 17, 2015 at 4:08 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Yicong,
>
> On 17 August 2015 at 08:40, Yicong Huang <hengha.mao at gmail.com> wrote:
> > yes, we could use a structure to wrap severl python callback function
> > pointers in one execution.
> > However, the issue is that we might not be able to get all python
> functions
> > that would be executed at the beginning.
>
> The basic idea is to define whatever API you need with just one call
> to pypy_execute_source_ptr().  If you want to be able to compile and
> execute arbitrary Python functions, just make that feature available
> in your API.  This is the same idea as PyRun_SimpleString(), which is
> *one* API function but lets you execute arbitrary Python code.
> Example:
>
>
> struct API {
>     long (*run_function)(char *);
> };
> struct API api;   /* global var */
>
> int initialize_api(void) {    /* run this only once */
>     static char source[] =
>         "import sys; sys.path.insert(0, '.'); "
>         "import interface; interface.fill_api(c_argument)";
>     pypy_execute_source_ptr(source, &api);
> }
>
> /* then execute 'api.run_function(some_source_code)' any number of times */
>
>
> # file "interface.py"
>
> import cffi
>
> ffi = cffi.FFI()
> ffi.cdef('''
> struct API {
>     long (*run_function)(char *);
> };
> ''')
>
> @ffi.callback("long(char *)")
> def run_function(p_src):
>     src = ffi.string(p_src)
>     # here 'src' is a string, we can do whatever we want with it---like
> eval()
>     return eval(src, {})
>
> def fill_api(ptr):
>     global api
>     api = ffi.cast("struct API*", ptr)
>     api.run_function = run_function
>
>
> ---
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150817/d73e04e1/attachment.html>

From arigo at tunes.org  Mon Aug 17 15:54:34 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Aug 2015 15:54:34 +0200
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAOEzsGDfzs0Dr5DLQKcvoh3C-0NQf4KKdf9Udv3dKySj6Wyf4A@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
 <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
 <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>
 <CAMSv6X205KnVWjBjL2Z5pPf2c1zxyNHD_f0ffRCLEwFXZRFq4w@mail.gmail.com>
 <CAOEzsGDfzs0Dr5DLQKcvoh3C-0NQf4KKdf9Udv3dKySj6Wyf4A@mail.gmail.com>
Message-ID: <CAMSv6X2nkDqcpAQt1nDWp97cDzt214GPj-j_k09vyL_cezahCA@mail.gmail.com>

Hi,

On 17 August 2015 at 12:59, Yicong Huang <hengha.mao at gmail.com> wrote:
> However, to perform the python function call, we need to parse char*(python
> code), a python code lexer, praser, JIT and etc, in other words do all the
> work that  pypy_execute_source() did.

This is done in a couple of lines in Python.  The following would do all this:

@ffi.function("void(char *)")
def run_function(p_src):
    src = ffi.string(p_src)
    glob = {}
    exec src in glob

In my previous example I used "eval" instead of "exec" only to return a value.

If you want to be more JIT-friendly, and if run_function() is going to
be called with the exact same p_src a lot of times, you could do
instead:

cache = {}
@ffi.function("void(char *)")
def run_function(p_src):
    src = ffi.string(p_src)
    code = cache.get(src)
    if code is None:
        code = compile(src, 'exec', 'exec')
        cache[src] = code
    glob = {}
    exec code in glob

Or maybe:

cache = []
@ffi.function("long(char *)")
def compile_and_cache(p_src):
    code = compile(ffi.string(p_src), 'exec', 'exec')
    index = len(cache)
    cache.append(code)
    return index

@ffi.function("void(long)")
def run_compiled_function(index):
    glob = {}
    exec cache[index] in glob


> So why don't PyPy expose and API to do such job ?

Because PyPy only exposes an *API builder*, not a particular API.  The
pypy_execute_source_ptr() is the bare minimal entry point, which
you're supposed to call once and build your own API as you see fit.
The functions above are examples of such an API.  And as shown above,
even such a simple API has various tweaks that are better in various
cases, like whether caching the code object is a good idea or not; you
can implement any version you like in a few lines of Python.


A bient?t,

Armin.

From arigo at tunes.org  Mon Aug 17 16:10:39 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Aug 2015 16:10:39 +0200
Subject: [pypy-dev] pypy platform specific tests
In-Reply-To: <4176FFD2EDA9C24CA9AB011273CFDDC8025BA00D@HJ-MBX1.persistent.co.in>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025AE106@HJ-MBX1.persistent.co.in>
 <CAMSv6X2CufYu=-=Lf+=4quXqRyNFyyqoeLSUZx-bcjOXWjX=fA@mail.gmail.com>
 <4176FFD2EDA9C24CA9AB011273CFDDC8025BA00D@HJ-MBX1.persistent.co.in>
Message-ID: <CAMSv6X2rgG3=_weqcQTHuRBU8id=9+5E7vSSRaYCbQrU04PggA@mail.gmail.com>

Hi,

On 17 August 2015 at 12:52, Vaibhav Sood <vaibhav_sood at persistent.com> wrote:
> Thanks Armin! I was looking at the stacklet code under "pypy/rpython/translator/c/src/stacklet" as an example , I did find a tests.c file in the same folder but couldn?t find it being invoked by buildbot or any .py file in the test suite (in the Makefile there is a 'run-all-tests' target but doing a 'make run-all-tests' gives an error) so was wondering how this code gets tested?

This code is tested via its RPython interface in
rpython/rlib/rstacklet.py.  The test file is
rpython/rlib/test/test_rstacklet.py.

In the C directory itself, there is a Makefile with a 'run-all-tests'
target in it, indeed, but that was used during development only.  You
can still use it now if you want to test manually the C files after
making changes to them, though I guess it doesn't work on platforms
other than Linux (and doesn't really need to, as long as
test_rstacklet.py passes).


A bient?t,

Armin.

From tbaldridge at gmail.com  Mon Aug 17 21:29:32 2015
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Mon, 17 Aug 2015 13:29:32 -0600
Subject: [pypy-dev] Don't release GIL in jit_libffi?
Message-ID: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>

My calls into C are pretty fast in my custom RPython VM, but jit_libffi is
trying to release the GIL on every single call to a FFI function. Is there
a way to disable this if I know that a function call is going to be super
fast?

Timothy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150817/ce3deeec/attachment.html>

From fijall at gmail.com  Mon Aug 17 22:09:12 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 17 Aug 2015 22:09:12 +0200
Subject: [pypy-dev] Don't release GIL in jit_libffi?
In-Reply-To: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>
References: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>
Message-ID: <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>

not easily :-( would be cool if such an option exists

On Mon, Aug 17, 2015 at 9:29 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> My calls into C are pretty fast in my custom RPython VM, but jit_libffi is
> trying to release the GIL on every single call to a FFI function. Is there a
> way to disable this if I know that a function call is going to be super
> fast?
>
> Timothy
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From alex.gaynor at gmail.com  Mon Aug 17 22:13:30 2015
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 17 Aug 2015 16:13:30 -0400
Subject: [pypy-dev] Don't release GIL in jit_libffi?
In-Reply-To: <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>
References: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>
 <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>
Message-ID: <CAFRnB2XYcSfpsk8we+i3H7oG7GZxA1nAH_Vo9F_agBa=ogo8fg@mail.gmail.com>

In theory it's just a matter of plumbing everything all the way through the
layers of JIT?

Alex

On Mon, Aug 17, 2015 at 4:09 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> not easily :-( would be cool if such an option exists
>
> On Mon, Aug 17, 2015 at 9:29 PM, Timothy Baldridge <tbaldridge at gmail.com>
> wrote:
> > My calls into C are pretty fast in my custom RPython VM, but jit_libffi
> is
> > trying to release the GIL on every single call to a FFI function. Is
> there a
> > way to disable this if I know that a function call is going to be super
> > fast?
> >
> > Timothy
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150817/2274858e/attachment.html>

From hengha.mao at gmail.com  Tue Aug 18 14:30:49 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Tue, 18 Aug 2015 20:30:49 +0800
Subject: [pypy-dev] pypy callback function pointer become invalid
In-Reply-To: <CAMSv6X2nkDqcpAQt1nDWp97cDzt214GPj-j_k09vyL_cezahCA@mail.gmail.com>
References: <CAOEzsGD5BFO0H+qD9srpVOesVkQHooe+PLVs0nKwaK36xLuDjg@mail.gmail.com>
 <CAMSv6X03FNiEY7i1WJq07zXzPc5iE_0LpvfSGbapjcddUSfXFg@mail.gmail.com>
 <CAK5idxQaD7HKDTLFWL3XCc2CBi7p9_r3-dS8ZBpZ77fdzUD7VQ@mail.gmail.com>
 <CAOEzsGD7UMPnMn=JyiB2k6eE+wMctkj2E1=dJcFHpO30rek24Q@mail.gmail.com>
 <CAK5idxTbzU1D+93q9MVd=k6611c3cpq641SerPcWkg4s-QxPHQ@mail.gmail.com>
 <CAK-ecOiSKbbXhHWFdGbfSxO0dL-BaNs0N9PaigZosdoUKxfFOQ@mail.gmail.com>
 <CAK5idxSXXuJjbv52ZGT5yVJzbM5v8B2WDu+=wMufAtbpVF4Hrw@mail.gmail.com>
 <CAOEzsGB9jurM6wWj_pHqSOByVWNm86_12EHtvnY8FriMb9961w@mail.gmail.com>
 <CAMSv6X205KnVWjBjL2Z5pPf2c1zxyNHD_f0ffRCLEwFXZRFq4w@mail.gmail.com>
 <CAOEzsGDfzs0Dr5DLQKcvoh3C-0NQf4KKdf9Udv3dKySj6Wyf4A@mail.gmail.com>
 <CAMSv6X2nkDqcpAQt1nDWp97cDzt214GPj-j_k09vyL_cezahCA@mail.gmail.com>
Message-ID: <CAOEzsGB8m_gMOcfGPvGypOPE_A-udOLg2LEgwqH+8ThPRGc8nA@mail.gmail.com>

Great thanks Armin! The method should work.
And I thought "exec", "compiled" and "cached" together do the quite similar
work as pypy_execute_source() does.
Yes, in theory we could do all these work by ourselves, and also could be
more flexible to chose what need to do and what need not to do.
But I think if we could wrap these work and have similar API as Python, it
would be more easier and friendly to use PyPy, especially for those users
transferring from Python to Pypy.

On Mon, Aug 17, 2015 at 9:54 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi,
>
> On 17 August 2015 at 12:59, Yicong Huang <hengha.mao at gmail.com> wrote:
> > However, to perform the python function call, we need to parse
> char*(python
> > code), a python code lexer, praser, JIT and etc, in other words do all
> the
> > work that  pypy_execute_source() did.
>
> This is done in a couple of lines in Python.  The following would do all
> this:
>
> @ffi.function("void(char *)")
> def run_function(p_src):
>     src = ffi.string(p_src)
>     glob = {}
>     exec src in glob
>
> In my previous example I used "eval" instead of "exec" only to return a
> value.
>
> If you want to be more JIT-friendly, and if run_function() is going to
> be called with the exact same p_src a lot of times, you could do
> instead:
>
> cache = {}
> @ffi.function("void(char *)")
> def run_function(p_src):
>     src = ffi.string(p_src)
>     code = cache.get(src)
>     if code is None:
>         code = compile(src, 'exec', 'exec')
>         cache[src] = code
>     glob = {}
>     exec code in glob
>
> Or maybe:
>
> cache = []
> @ffi.function("long(char *)")
> def compile_and_cache(p_src):
>     code = compile(ffi.string(p_src), 'exec', 'exec')
>     index = len(cache)
>     cache.append(code)
>     return index
>
> @ffi.function("void(long)")
> def run_compiled_function(index):
>     glob = {}
>     exec cache[index] in glob
>
>
> > So why don't PyPy expose and API to do such job ?
>
> Because PyPy only exposes an *API builder*, not a particular API.  The
> pypy_execute_source_ptr() is the bare minimal entry point, which
> you're supposed to call once and build your own API as you see fit.
> The functions above are examples of such an API.  And as shown above,
> even such a simple API has various tweaks that are better in various
> cases, like whether caching the code object is a good idea or not; you
> can implement any version you like in a few lines of Python.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150818/cc2e37de/attachment.html>

From arigo at tunes.org  Tue Aug 18 21:38:55 2015
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 18 Aug 2015 21:38:55 +0200
Subject: [pypy-dev] Don't release GIL in jit_libffi?
In-Reply-To: <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>
References: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>
 <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>
Message-ID: <CAMSv6X2AKj71WMyHsf+en4m7VuwnvpLujJUO6eWTr+C7=e1R3g@mail.gmail.com>

Hi Tim,

On 17 August 2015 at 22:09, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Aug 17, 2015 at 9:29 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
>> My calls into C are pretty fast in my custom RPython VM, but jit_libffi is
>> trying to release the GIL on every single call to a FFI function. Is there a
>> way to disable this if I know that a function call is going to be super
>> fast?
>
> not easily :-( would be cool if such an option exists

Additionally, look at the kind of machine code emitted.  It is not as
fast as theoretically possible if we didn't release the GIL at all,
but it's still much faster than calling a lock release and reacquire
function.


A bient?t,

Armin.

From fijall at gmail.com  Tue Aug 18 21:45:43 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 18 Aug 2015 21:45:43 +0200
Subject: [pypy-dev] Don't release GIL in jit_libffi?
In-Reply-To: <CAMSv6X2AKj71WMyHsf+en4m7VuwnvpLujJUO6eWTr+C7=e1R3g@mail.gmail.com>
References: <CAL36E+tWJ6NvR9Gk2trz+O71b464cJ9YOs-FUvchghdeGrdNSw@mail.gmail.com>
 <CAK5idxS6kBgpwg2_ghPGphPZq2CPwTbu-_9h9jZEKx5qKC+3Ow@mail.gmail.com>
 <CAMSv6X2AKj71WMyHsf+en4m7VuwnvpLujJUO6eWTr+C7=e1R3g@mail.gmail.com>
Message-ID: <CAK5idxRu8Yimt8MnLWHWGH_LO_XcgLhPy=Dw1wjmBR5fjaCr-Q@mail.gmail.com>

On Tue, Aug 18, 2015 at 9:38 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Tim,
>
> On 17 August 2015 at 22:09, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Mon, Aug 17, 2015 at 9:29 PM, Timothy Baldridge <tbaldridge at gmail.com> wrote:
>>> My calls into C are pretty fast in my custom RPython VM, but jit_libffi is
>>> trying to release the GIL on every single call to a FFI function. Is there a
>>> way to disable this if I know that a function call is going to be super
>>> fast?
>>
>> not easily :-( would be cool if such an option exists
>
> Additionally, look at the kind of machine code emitted.  It is not as
> fast as theoretically possible if we didn't release the GIL at all,
> but it's still much faster than calling a lock release and reacquire
> function.
>
>
> A bient?t,
>
> Armin.

Yes and no. The code *itself* is not that bad, but it also trashes all
the caches in the JIT optimizer, because it has to assume that
everything can be potentially invalidated. There are ways to
addressing that separately, but the JIT CALL_RELEASE_GIL is not as
fast as it could be

From matti.picus at gmail.com  Tue Aug 18 23:10:41 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 19 Aug 2015 00:10:41 +0300
Subject: [pypy-dev] buildbot summary by hg hash w/out revision number
Message-ID: <55D39F51.9030604@gmail.com>

The build summaries, like the one here
http://buildbot.pypy.org/summary?category=linux-armhf
gather results together by revision so that all different types of tests 
for a single revision line up in a single column of results.
ARM is built on one machine (cross-translated since the ARM machines do 
not have 4GB memory) and tested on another.
Since the Mercurial build step is given an revision it does
hg pull -r <rev hash>
hg up -r <rev hash>
which seems to cause the revision number in the testing repo to get out 
of sync with the revision number of the building repo,
which in turn prevents gathering results by rev_int:rev:hash from the 
tests together with the builds.

In short, I committed a change to the buildbot summary creation script 
(with tests) on the coalesce-by-hash branch that should fix it by only 
coalescing test runs by hg hash.
The proof would be if the link above displayed results from all 7 armhf 
build slaves[0][1] together in one column, without breaking navigation 
around the site.

Could a buildbot admin please upgrade to the branch, restart the 
buildbot, and click around the links on the page to see if I achieved 
hashing Nirvana? If things go wrong, just revert to default and try to 
let me know what I broke.

Matti

[0] Do we really need all these builds every night?
[1] Here are the builders, did I forget any?
build-pypy-c-jit-linux-armhf-raring
build-pypy-c-jit-linux-armhf-raspbian
build-pypy-c-linux-armhf-raspbian
jitbackendonly-own-linux-armhf
jitbackendonly-own-linux-armhf-v7
pypy-c-app-level-linux-armhf-v7
pypy-c-jit-linux-armhf-v7

From matti.picus at gmail.com  Wed Aug 19 20:36:37 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 19 Aug 2015 21:36:37 +0300
Subject: [pypy-dev] 2.6.1 release cycle has begun
Message-ID: <55D4CCB5.7080708@gmail.com>

I have started the release cycle for 2.6.1, are there outstanding issues 
or soon-to-be-committed features I should wait for?
Please help with the release notice, what have we done since 2.6.0 or at 
all?
Any suggestions for names?

https://bitbucket.org/pypy/pypy/src/release-2.6.x/pypy/doc/release-2.6.1.rst

Matti

From fijall at gmail.com  Wed Aug 19 21:02:40 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 19 Aug 2015 21:02:40 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <55D4CCB5.7080708@gmail.com>
References: <55D4CCB5.7080708@gmail.com>
Message-ID: <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>

I want to fix OS X issues on vmprof. maybe will do it now now

On Wed, Aug 19, 2015 at 8:36 PM, Matti Picus <matti.picus at gmail.com> wrote:
> I have started the release cycle for 2.6.1, are there outstanding issues or
> soon-to-be-committed features I should wait for?
> Please help with the release notice, what have we done since 2.6.0 or at
> all?
> Any suggestions for names?
>
> https://bitbucket.org/pypy/pypy/src/release-2.6.x/pypy/doc/release-2.6.1.rst
>
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From fijall at gmail.com  Wed Aug 19 21:54:56 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 19 Aug 2015 21:54:56 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
Message-ID: <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>

I fixed a few bugs, I will write a parser for vmmap tomorrow

On Wed, Aug 19, 2015 at 9:02 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I want to fix OS X issues on vmprof. maybe will do it now now
>
> On Wed, Aug 19, 2015 at 8:36 PM, Matti Picus <matti.picus at gmail.com> wrote:
>> I have started the release cycle for 2.6.1, are there outstanding issues or
>> soon-to-be-committed features I should wait for?
>> Please help with the release notice, what have we done since 2.6.0 or at
>> all?
>> Any suggestions for names?
>>
>> https://bitbucket.org/pypy/pypy/src/release-2.6.x/pypy/doc/release-2.6.1.rst
>>
>> Matti
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

From matti.picus at gmail.com  Thu Aug 20 00:01:48 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 20 Aug 2015 01:01:48 +0300
Subject: [pypy-dev] buildbot summary by hg hash w/out revision number
In-Reply-To: <55D39F51.9030604@gmail.com>
References: <55D39F51.9030604@gmail.com>
Message-ID: <55D4FCCC.8070604@gmail.com>

I give up.
Could someone please revert the build master to the default branch when 
it is safe to do so.

The summary pages do not properly sort by latest build, the nightly 
build pages will need alot of work to use only the hash not the int:hash.
Maybe someone will pick this up and run with it, it seems to require 
more work than I am willing to put in now.

Matti

P.S. - an alternative would be to override the hg update mechanism so 
the repos do not get out of sync, perhaps that will be easier.
Matti

On 19/08/15 00:10, Matti Picus wrote:
> The build summaries, like the one here
> http://buildbot.pypy.org/summary?category=linux-armhf
> gather results together by revision so that all different types of 
> tests for a single revision line up in a single column of results.
> ARM is built on one machine (cross-translated since the ARM machines 
> do not have 4GB memory) and tested on another.
> Since the Mercurial build step is given an revision it does
> hg pull -r <rev hash>
> hg up -r <rev hash>
> which seems to cause the revision number in the testing repo to get 
> out of sync with the revision number of the building repo,
> which in turn prevents gathering results by rev_int:rev:hash from the 
> tests together with the builds.
>
> In short, I committed a change to the buildbot summary creation script 
> (with tests) on the coalesce-by-hash branch that should fix it by only 
> coalescing test runs by hg hash.
> The proof would be if the link above displayed results from all 7 
> armhf build slaves[0][1] together in one column, without breaking 
> navigation around the site.
>
> Could a buildbot admin please upgrade to the branch, restart the 
> buildbot, and click around the links on the page to see if I achieved 
> hashing Nirvana? If things go wrong, just revert to default and try to 
> let me know what I broke.
>
> Matti
>
> [0] Do we really need all these builds every night?
> [1] Here are the builders, did I forget any?
> build-pypy-c-jit-linux-armhf-raring
> build-pypy-c-jit-linux-armhf-raspbian
> build-pypy-c-linux-armhf-raspbian
> jitbackendonly-own-linux-armhf
> jitbackendonly-own-linux-armhf-v7
> pypy-c-app-level-linux-armhf-v7
> pypy-c-jit-linux-armhf-v7


From arigo at tunes.org  Thu Aug 20 12:07:47 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 20 Aug 2015 12:07:47 +0200
Subject: [pypy-dev] buildbot summary by hg hash w/out revision number
In-Reply-To: <55D4FCCC.8070604@gmail.com>
References: <55D39F51.9030604@gmail.com> <55D4FCCC.8070604@gmail.com>
Message-ID: <CAMSv6X2ngAVH0-YkvDjvxtTJy8qhM_DzwkzbYzs+KaoCQiqXsQ@mail.gmail.com>

Hi Matti,

On 20 August 2015 at 00:01, Matti Picus <matti.picus at gmail.com> wrote:
> P.S. - an alternative would be to override the hg update mechanism so the
> repos do not get out of sync, perhaps that will be easier.

Alternatively, we should check out the out-of-sync repo.  It's likely
because it has custom commits in it that it is out-of-sync.  If we
freshly clone from bitbucket, we always get the same revisions in the
same order, as far as I know.


A bient?t,

Armin.

From arigo at tunes.org  Thu Aug 20 12:15:37 2015
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 20 Aug 2015 12:15:37 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <55D4CCB5.7080708@gmail.com>
References: <55D4CCB5.7080708@gmail.com>
Message-ID: <CAMSv6X2wgUHdK10nGHKrh2K9zVYTdELN+vdQqLi-HQTMT8tSKg@mail.gmail.com>

Hi Matti,

On 19 August 2015 at 20:36, Matti Picus <matti.picus at gmail.com> wrote:
> Please help with the release notice, what have we done since 2.6.0 or at
> all?

The stdlib is now 2.7.10 and cffi is 1.2.

I think the most interesting change is int-float-list-strategy, at
least in terms of explaining it:

"""Use a compact strategy for Python lists that mix integers and floats,
at least if the integers fit inside 32 bits.  These lists are now
stored as an array of floats, like lists that contain only floats; the
difference is that integers are stored as tagged NaNs.  (This should
have no visible effect!  After ``lst = [42, 42.5]``, the value of
``lst[0]`` is still *not* the float ``42.0`` but the integer ``42``.)"""

Maybe mention that numpy support is getting more and more complete,
and vmprof support is improving.


A bient?t,

Armin.

From matti.picus at gmail.com  Thu Aug 20 21:31:02 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 20 Aug 2015 22:31:02 +0300
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
Message-ID: <55D62AF6.3090005@gmail.com>

On 19/08/15 22:54, Maciej Fijalkowski wrote:
> I fixed a few bugs, I will write a parser for vmmap tomorrow
> Maciej Fijalkowski

Will the vmmap parser be part of pypy, and if so are you requesting I 
wait with 2.6.1 until it is ready?
Matti

From naylor.b.david at gmail.com  Thu Aug 20 21:52:17 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Thu, 20 Aug 2015 21:52:17 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <55D4CCB5.7080708@gmail.com>
References: <55D4CCB5.7080708@gmail.com>
Message-ID: <38923020.zYJJ8QdoQJ@dragon.local>

On Wednesday, 19 August 2015 21:36:37 Matti Picus wrote:
> I have started the release cycle for 2.6.1, are there outstanding issues
> or soon-to-be-committed features I should wait for?

Hi Matti,

It would be great if the patch from issue #2107 could be committed.  The patch 
has been compile tested an FreeBSD.  

After this patch has been committed FreeBSD will not need any patches to 
compile PyPy (from the Ports Collection) :-D.  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150820/ae79ccc4/attachment.sig>

From fijall at gmail.com  Thu Aug 20 22:38:49 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 20 Aug 2015 22:38:49 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <55D62AF6.3090005@gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
 <55D62AF6.3090005@gmail.com>
Message-ID: <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>

On Thu, Aug 20, 2015 at 9:31 PM, Matti Picus <matti.picus at gmail.com> wrote:
> On 19/08/15 22:54, Maciej Fijalkowski wrote:
>>
>> I fixed a few bugs, I will write a parser for vmmap tomorrow
>> Maciej Fijalkowski
>
>
> Will the vmmap parser be part of pypy, and if so are you requesting I wait
> with 2.6.1 until it is ready?
> Matti

I hope not, but it needs to be tested. When do you want to release it?
I can look into OS X issues tomorrow

From phyo.arkarlwin at gmail.com  Thu Aug 20 22:50:00 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Fri, 21 Aug 2015 03:20:00 +0630
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
 <55D62AF6.3090005@gmail.com>
 <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>
Message-ID: <CA+HjJrhBcGw2thKuy=+Y0+fwq-Ted2P4-WUVnoL3h0TcZyYW+g@mail.gmail.com>

How is python 3.4 release coming?.

On Fri, Aug 21, 2015 at 3:08 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> On Thu, Aug 20, 2015 at 9:31 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
> > On 19/08/15 22:54, Maciej Fijalkowski wrote:
> >>
> >> I fixed a few bugs, I will write a parser for vmmap tomorrow
> >> Maciej Fijalkowski
> >
> >
> > Will the vmmap parser be part of pypy, and if so are you requesting I
> wait
> > with 2.6.1 until it is ready?
> > Matti
>
> I hope not, but it needs to be tested. When do you want to release it?
> I can look into OS X issues tomorrow
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150821/ea3d2527/attachment.html>

From matti.picus at gmail.com  Fri Aug 21 13:23:08 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 21 Aug 2015 14:23:08 +0300
Subject: [pypy-dev] buildbot summary by hg hash w/out revision number
In-Reply-To: <CAMSv6X2ngAVH0-YkvDjvxtTJy8qhM_DzwkzbYzs+KaoCQiqXsQ@mail.gmail.com>
References: <55D39F51.9030604@gmail.com> <55D4FCCC.8070604@gmail.com>
 <CAMSv6X2ngAVH0-YkvDjvxtTJy8qhM_DzwkzbYzs+KaoCQiqXsQ@mail.gmail.com>
Message-ID: <55D70A1C.3090704@gmail.com>



On 20/08/15 13:07, Armin Rigo wrote:
> Hi Matti,
>
> It's likely
> because it has custom commits in it that it is out-of-sync.  If we
> freshly clone from bitbucket, we always get the same revisions in the
> same order, as far as I know.
>
>
> A bient?t,
>
> Armin.
For some reason the "hg pull" command is doing so with a revision hash, 
as can be seen here
http://buildbot.pypy.org/builders/pypy-c-app-level-linux-armhf-v7/builds/765/steps/hg/logs/stdio
(search for "hg --verbose pull")
In my sources for buildbot, I could not a reason why "pull" is followed 
by --rev <revision>.
While buildbot claims it is running version 0.8.8 in the page footers, 
maybe someone modified the Mercurial class used by cobra's buildbot 
sources to add the --rev option to the build step?

In the meantime, I have a crontab set up to "hg pull" on the 
cubieboard-bob buildbot, it seems to have improved the situation for armhf
http://buildbot.pypy.org/summary?category=linux-armhf

Now if we could only _fix_ the failures before 2.6.1 ...
Matti


From arigo at tunes.org  Fri Aug 21 13:59:12 2015
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Aug 2015 13:59:12 +0200
Subject: [pypy-dev] buildbot summary by hg hash w/out revision number
In-Reply-To: <55D70A1C.3090704@gmail.com>
References: <55D39F51.9030604@gmail.com> <55D4FCCC.8070604@gmail.com>
 <CAMSv6X2ngAVH0-YkvDjvxtTJy8qhM_DzwkzbYzs+KaoCQiqXsQ@mail.gmail.com>
 <55D70A1C.3090704@gmail.com>
Message-ID: <CAMSv6X11KMiVzsuODb=aiTuNYwFed8vJRHrK2WVZpeaFTW2cPg@mail.gmail.com>

Hi Matti,

On 21 August 2015 at 13:23, Matti Picus <matti.picus at gmail.com> wrote:
> For some reason the "hg pull" command is doing so with a revision hash, as
> can be seen here

Ah, you mean that doing "hg pull --rev xxx" breaks the order of the
revisions.  Indeed, it does...


Armin

From vaibhav_sood at persistent.com  Fri Aug 21 14:14:52 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Fri, 21 Aug 2015 12:14:52 +0000
Subject: [pypy-dev] [newb] fixing a nightly test case failure
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025BCB8E@HJ-MBX1.persistent.co.in>

Hi,

I am a newbie to pypy and looking to start making some contributions to the project. I was looking at the buildbot test case failures (after referring to an earlier thread on this forum: https://mail.python.org/pipermail/pypy-dev/2015-July/013751.html ) for the py3.3 branch and to start off picked a test failure to work on: http://buildbot.pypy.org/summary/longrepr?testname=AppTestFunction.%28%29.test_call_error_message&builder=own-linux-x86-64&build=3996&mod=interpreter.test.test_function
I was looking for some guidance as to the steps to make a code change contribution. I checked the bug tracking list but didn't find this test failure listed, would the typical steps be to create a issue in bug tracking system first, work on the change and then submit the patch for commit? My concern is I want to verify I can work on this failure and that I am not duplicating or doing unnecessary work

Thanks,
Vaibhav

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150821/da765e9d/attachment.html>

From phyo.arkarlwin at gmail.com  Fri Aug 21 16:07:58 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Fri, 21 Aug 2015 20:37:58 +0630
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CA+HjJrhBcGw2thKuy=+Y0+fwq-Ted2P4-WUVnoL3h0TcZyYW+g@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
 <55D62AF6.3090005@gmail.com>
 <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>
 <CA+HjJrhBcGw2thKuy=+Y0+fwq-Ted2P4-WUVnoL3h0TcZyYW+g@mail.gmail.com>
Message-ID: <CA+HjJrjz471172wX+QBw9fZRpnoCK4CrXD5OO9JcK=5pQHyAPg@mail.gmail.com>

Will there be any 3.x release included in this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150821/87786640/attachment.html>

From ronan.lamy at gmail.com  Fri Aug 21 16:50:44 2015
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Fri, 21 Aug 2015 15:50:44 +0100
Subject: [pypy-dev] [newb] fixing a nightly test case failure
In-Reply-To: <4176FFD2EDA9C24CA9AB011273CFDDC8025BCB8E@HJ-MBX1.persistent.co.in>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025BCB8E@HJ-MBX1.persistent.co.in>
Message-ID: <55D73AC4.7090402@gmail.com>

Le 21/08/15 13:14, Vaibhav Sood a ?crit :
> Hi,
>
> I am a newbie to pypy and looking to start making some contributions to
> the project. I was looking at the buildbot test case failures (after
> referring to an earlier thread on this forum:
> https://mail.python.org/pipermail/pypy-dev/2015-July/013751.html ) for
> the py3.3 branch and to start off picked a test failure to work on:
> http://buildbot.pypy.org/summary/longrepr?testname=AppTestFunction.%28%29.test_call_error_message&builder=own-linux-x86-64&build=3996&mod=interpreter.test.test_function
>
> I was looking for some guidance as to the steps to make a code change
> contribution. I checked the bug tracking list but didn?t find this test
> failure listed, would the typical steps be to create a issue in bug
> tracking system first, work on the change and then submit the patch for
> commit? My concern is I want to verify I can work on this failure and
> that I am not duplicating or doing unnecessary work

We don't typically open issues for the py3.3 branch since we know that 
there is still a large of number of test failures to address. The 
typical workflow would rather be to ask on IRC (#pypy on 
irc.freenode.net) if anybody is working on this, do the work, and then 
send a pull request against the py3.3 branch. Feel free to open an issue 
beforehand though, if you think it's useful.

Cheers,
Ronan


From fijall at gmail.com  Fri Aug 21 18:38:13 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Aug 2015 18:38:13 +0200
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CA+HjJrjz471172wX+QBw9fZRpnoCK4CrXD5OO9JcK=5pQHyAPg@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
 <55D62AF6.3090005@gmail.com>
 <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>
 <CA+HjJrhBcGw2thKuy=+Y0+fwq-Ted2P4-WUVnoL3h0TcZyYW+g@mail.gmail.com>
 <CA+HjJrjz471172wX+QBw9fZRpnoCK4CrXD5OO9JcK=5pQHyAPg@mail.gmail.com>
Message-ID: <CAK5idxQhjoiqQXoNGD56aR754y795FDVEqM97+jK34L45VdBVQ@mail.gmail.com>

I don't think so, our 3.x releases are usually done at different times

On Fri, Aug 21, 2015 at 4:07 PM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> Will there be any 3.x release included in this?

From phyo.arkarlwin at gmail.com  Fri Aug 21 18:42:28 2015
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Fri, 21 Aug 2015 23:12:28 +0630
Subject: [pypy-dev] 2.6.1 release cycle has begun
In-Reply-To: <CAK5idxQhjoiqQXoNGD56aR754y795FDVEqM97+jK34L45VdBVQ@mail.gmail.com>
References: <55D4CCB5.7080708@gmail.com>
 <CAK5idxT13oCY_aU8CnJjEuyn2jCS3Tk9fcsiXYakJo0=ALQFtw@mail.gmail.com>
 <CAK5idxQiiAiRHhDdY_ZpZCy5yYRuzAX0JEJMPWSfQapQTmN1sA@mail.gmail.com>
 <55D62AF6.3090005@gmail.com>
 <CAK5idxR79bq3-waBV_MCRSK8iayvYyHzuYxyM+viNLTdUVvpxA@mail.gmail.com>
 <CA+HjJrhBcGw2thKuy=+Y0+fwq-Ted2P4-WUVnoL3h0TcZyYW+g@mail.gmail.com>
 <CA+HjJrjz471172wX+QBw9fZRpnoCK4CrXD5OO9JcK=5pQHyAPg@mail.gmail.com>
 <CAK5idxQhjoiqQXoNGD56aR754y795FDVEqM97+jK34L45VdBVQ@mail.gmail.com>
Message-ID: <CA+HjJri4jXFN-mh7VP-8qO86EDvhzjFQ=DLZSFadTUJhkqObUQ@mail.gmail.com>

ah i c

On Fri, Aug 21, 2015 at 11:08 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> I don't think so, our 3.x releases are usually done at different times
>
> On Fri, Aug 21, 2015 at 4:07 PM, Phyo Arkar <phyo.arkarlwin at gmail.com>
> wrote:
> > Will there be any 3.x release included in this?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150821/154f3fa3/attachment.html>

From matti.picus at gmail.com  Sat Aug 22 20:25:44 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 22 Aug 2015 21:25:44 +0300
Subject: [pypy-dev] 2.6.1 and freebsd-9
Message-ID: <55D8BEA8.2070907@gmail.com>

I would like to add the freebsd-9-x86-64 binary tgz to our released 
downloads. We still have a number of failing tests showing up on
http://buildbot.pypy.org/summary?category=freebsd64
among them many
DLOpenError: "opening 'libm.so' with ctypes.CDLL() works, but not with 
c_dlopen()??"
and the more worrying SIGSEGV in
http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-c-jit-freebsd-9-x86-64&build=655&mod=lib-python.2.7.test.test_io

Could someone proficient in freebsd suggest what is going on?
Matti

From naylor.b.david at gmail.com  Mon Aug 24 23:07:20 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Mon, 24 Aug 2015 23:07:20 +0200
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <55D8BEA8.2070907@gmail.com>
References: <55D8BEA8.2070907@gmail.com>
Message-ID: <9800207.igzAqY9dSP@dragon.local>

On Saturday, 22 August 2015 21:25:44 Matti Picus wrote:
> I would like to add the freebsd-9-x86-64 binary tgz to our released
> downloads. We still have a number of failing tests showing up on
> http://buildbot.pypy.org/summary?category=freebsd64
> among them many
> DLOpenError: "opening 'libm.so' with ctypes.CDLL() works, but not with
> c_dlopen()??"
> and the more worrying SIGSEGV in
> http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-c
> -jit-freebsd-9-x86-64&build=655&mod=lib-python.2.7.test.test_io
> 
> Could someone proficient in freebsd suggest what is going on?

Hi Matti,

I am able to reproduce this on FreeBSD 10.2:
# cat > test.py << __EOF
from ctypes import *

libc = CDLL("libc.so.7")
dlopen = libc["dlopen"]

# see <dlfnc.h>: void    *dlopen(const char *, int);
dlopen.argtypes = [c_char_p, c_int]
dlopen.restype = c_void_p

print dlopen(c_char_p("libm.so"), c_int(0))
__EOF
# python test.py
None

However, contract this with the C equivalent:
# cat > test.c << __EOF
#include <dlfcn.h>
#include <stdio.h>

int main(int argc, char** argv)
{
    printf("%p\n", dlopen("libm.so", RTLD_LOCAL));
}
__EOF
# cc -o test test.c
# ./test
0x800620800

I will investigate further.  

As for the SIGSEGV (it does not occur on python):
# gdb `which pypy` pypy.core
gdb) bt
#0  0x00000008006548a7 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x00000008020b9cd3 in pypy_thread_attach () from 
/usr/local/pypy-2.6/bin//libpypy-c.so
#3  0x00000008020c2ea0 in pypy_thread_attach () from 
/usr/local/pypy-2.6/bin//libpypy-c.so
#4  0x07f4e08f801b663a in ?? ()
#5  0x0000000000000000 in ?? ()

I will need to translate an unstripped pypy to debug further.  

Regards,

David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150824/6acf4dc0/attachment.sig>

From ronan.lamy at gmail.com  Tue Aug 25 00:13:48 2015
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Mon, 24 Aug 2015 23:13:48 +0100
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <55D8BEA8.2070907@gmail.com>
References: <55D8BEA8.2070907@gmail.com>
Message-ID: <55DB971C.7020700@gmail.com>

Le 22/08/15 19:25, Matti Picus a ?crit :
> I would like to add the freebsd-9-x86-64 binary tgz to our released
> downloads. We still have a number of failing tests showing up on
> http://buildbot.pypy.org/summary?category=freebsd64
> among them many
> DLOpenError: "opening 'libm.so' with ctypes.CDLL() works, but not with
> c_dlopen()??"

This is a manifestation of https://bitbucket.org/pypy/pypy/issues/1641
You can pretty much assume that every such failure should actually be a 
skipped test.

> and the more worrying SIGSEGV in
> http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy-c-jit-freebsd-9-x86-64&build=655&mod=lib-python.2.7.test.test_io
>
>
> Could someone proficient in freebsd suggest what is going on?
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev


From naylor.b.david at gmail.com  Tue Aug 25 20:26:07 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Tue, 25 Aug 2015 20:26:07 +0200
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <9800207.igzAqY9dSP@dragon.local>
References: <55D8BEA8.2070907@gmail.com> <9800207.igzAqY9dSP@dragon.local>
Message-ID: <6519460.EHz3AaWGWU@dragon.local>

On Monday, 24 August 2015 23:07:20 David Naylor wrote:
> On Saturday, 22 August 2015 21:25:44 Matti Picus wrote:
> > I would like to add the freebsd-9-x86-64 binary tgz to our released
> > downloads. We still have a number of failing tests showing up on
> > http://buildbot.pypy.org/summary?category=freebsd64
> > among them many
> > DLOpenError: "opening 'libm.so' with ctypes.CDLL() works, but not with
> > c_dlopen()??"
> > and the more worrying SIGSEGV in
> > http://buildbot.pypy.org/summary/longrepr?testname=unmodified&builder=pypy
> > -c -jit-freebsd-9-x86-64&build=655&mod=lib-python.2.7.test.test_io
> > 
> > Could someone proficient in freebsd suggest what is going on?
> 
> Hi Matti,
> 
> I am able to reproduce this on FreeBSD 10.2:
> # cat > test.py << __EOF
> from ctypes import *
> 
> libc = CDLL("libc.so.7")
> dlopen = libc["dlopen"]
> 
> # see <dlfnc.h>: void    *dlopen(const char *, int);
> dlopen.argtypes = [c_char_p, c_int]
> dlopen.restype = c_void_p
> 
> print dlopen(c_char_p("libm.so"), c_int(0))
> __EOF
> # python test.py
> None

I have found the root cause of this problem (from the FreeBSD-Python team):
"""
Because dlopen symbol is magic.  It is provided by the shared libc.so
to satisfy the static linker, but real definition comes from the dynamic
linker.  The libc.so dlopen() is a null stub.

You are asking for the dlopen symbol from libc, which is returned to
you in faith, and which cannot load a library for real. While in the C
example, you use normal symbol resolution and get the dlopen from the
loader.

To get a handle to real dlopen with dlopen/dlsym, you should do in C:
        dlopenX = dlsym(RTLD_DEFAULT, "dlopen");
""" - Konstantin Belousov

Please find attached for a patch that fixes this issue.  Would you like me to 
submit a bug report for this patch?  

I have not translation tested this patch as yet.  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch-rpython_rtyper_lltypesystem_ll2ctypes.py
Type: text/x-python
Size: 2361 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150825/7c91b631/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150825/7c91b631/attachment.sig>

From matti.picus at gmail.com  Tue Aug 25 23:09:14 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 26 Aug 2015 00:09:14 +0300
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <6519460.EHz3AaWGWU@dragon.local>
References: <55D8BEA8.2070907@gmail.com> <9800207.igzAqY9dSP@dragon.local>
 <6519460.EHz3AaWGWU@dragon.local>
Message-ID: <55DCD97A.6040201@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150826/0c14a767/attachment.html>

From naylor.b.david at gmail.com  Tue Aug 25 23:27:35 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Tue, 25 Aug 2015 23:27:35 +0200
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <55DCD97A.6040201@gmail.com>
References: <55D8BEA8.2070907@gmail.com> <6519460.EHz3AaWGWU@dragon.local>
 <55DCD97A.6040201@gmail.com>
Message-ID: <1886759.xRKPg38jIu@dragon.local>

On Wednesday, 26 August 2015 00:09:14 Matti Picus wrote:
> On 25/08/15 21:26, David Naylor wrote:
> 
> Please find attached for a patch that fixes this issue.  Would you like me
> to submit a bug report for this patch?
> 
> I have not translation tested this patch as yet.
> 
> Regards
> 
> I committed your patches as c2f97b8c2415, the last chunk looks strange (why
> standard_c_lib.__error.restype ? ) , but let's give it a shot and see what
> happens. Matti

The last chunk was unrelated to this change - it was just cleaning up 
references to FreeBSD.  

Although the tests now pass when run with python they still do not pass when 
run from pypy.  

It appears pypy does not properly handle ctypes.CDLL(name, handle=X) where X 
is an integer as returned by dlopen(3).  I have an idea on how to implement 
the fix but that will need to wait for tomorrow.  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150825/8258a363/attachment.sig>

From naylor.b.david at gmail.com  Wed Aug 26 22:32:59 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Wed, 26 Aug 2015 22:32:59 +0200
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <1886759.xRKPg38jIu@dragon.local>
References: <55D8BEA8.2070907@gmail.com> <55DCD97A.6040201@gmail.com>
 <1886759.xRKPg38jIu@dragon.local>
Message-ID: <1540361.3XTmRKBNIP@dragon.local>

On Tuesday, 25 August 2015 23:27:35 David Naylor wrote:
> On Wednesday, 26 August 2015 00:09:14 Matti Picus wrote:
> > On 25/08/15 21:26, David Naylor wrote:
> > 
> > Please find attached for a patch that fixes this issue.  Would you like me
> > to submit a bug report for this patch?
> > 
> > I have not translation tested this patch as yet.
> > 
> > Regards
> > 
> > I committed your patches as c2f97b8c2415, the last chunk looks strange
> > (why
> > standard_c_lib.__error.restype ? ) , but let's give it a shot and see what
> > happens. Matti
> 
> The last chunk was unrelated to this change - it was just cleaning up
> references to FreeBSD.
> 
> Although the tests now pass when run with python they still do not pass when
> run from pypy.
> 
> It appears pypy does not properly handle ctypes.CDLL(name, handle=X) where X
> is an integer as returned by dlopen(3).  I have an idea on how to implement
> the fix but that will need to wait for tomorrow.

Please find attached for a patch that adds support for:
	ctypes.CDLL(name, handle=X)

The patch includes tests for the pypy and rpython changes.  I am not sure 
about the policy regarding changes to the python libraries and thus did not 
add a unit test there (the standard tests are lacking one for handle support 
anyway).  

I have translation tested the change and the previous fix I submitted now 
works on pypy :-D.  

I also filed an issue #2126 to clean up the handling of 'freebsd' for 
sys.platform.  

Lastly, regarding the segfault: it appears to be threading related.  I'll file 
an issue detailing all the stack traces and anything else that I can find that 
would be helpful.  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-CDLL-handle.diff
Type: text/x-patch
Size: 4886 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150826/d6e246b1/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150826/d6e246b1/attachment.sig>

From matti.picus at gmail.com  Wed Aug 26 23:53:05 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 27 Aug 2015 00:53:05 +0300
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <1540361.3XTmRKBNIP@dragon.local>
References: <55D8BEA8.2070907@gmail.com> <55DCD97A.6040201@gmail.com>
 <1886759.xRKPg38jIu@dragon.local> <1540361.3XTmRKBNIP@dragon.local>
Message-ID: <55DE3541.7060604@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150827/933586e9/attachment.html>

From naylor.b.david at gmail.com  Thu Aug 27 05:41:51 2015
From: naylor.b.david at gmail.com (David Naylor)
Date: Thu, 27 Aug 2015 05:41:51 +0200
Subject: [pypy-dev] 2.6.1 and freebsd-9
In-Reply-To: <55DE3541.7060604@gmail.com>
References: <55D8BEA8.2070907@gmail.com> <1540361.3XTmRKBNIP@dragon.local>
 <55DE3541.7060604@gmail.com>
Message-ID: <8475636.xbMoj7DVSb@dragon.local>

On Thursday, 27 August 2015 00:53:05 Matti Picus wrote:
> On 26/08/15 23:32, David Naylor wrote:
> Please find attached for a patch that adds support for:
> 	ctypes.CDLL(name, handle=X)
> 
> The patch includes tests for the pypy and rpython changes.  I am not sure
> about the policy regarding changes to the python libraries and thus did not
> add a unit test there (the standard tests are lacking one for handle support
> anyway).
> 
> I have translation tested the change and the previous fix I submitted now
> works on pypy :-D.
> 
> Regards
> Hi David. Thanks for the contributions. It would be more convenient if you
> would submit pull requests, or even better would be to do this work on the
> pypy/pypy repo on a branch. In the meantime, I have commited this set of
> changes on https://bitbucket.org/mattip/pypy and given you a write bit
> there, so if you do not want to fork on your own you may feel free to use
> that repo.

Thank you Matti :-D

> Unfortunately the patch causes a segfault in testing on ubuntu 14.04, in the
> test_handle test you added, I'm pretty sure handle=-2 will not work on
> linux.

That can be fixed, I had to hardcode -2 elsewhere (to prevent a cyclical 
reference on importing) but that was FreeBSD specific anyway. This can be done 
properly.  

I am away today and tomorrow on business but will attend to this over the 
weekend.   

> Also, it would be good to find a work-around so we do not modify upstream
> stdlib in lib-python/2.7, perhaps the handle-as-int could be dealt with
> later on in _ctypes, or however cpython deals with the issue?

The problem is that on cPython CDLL.handle is an integer (corresponding to 
void*) but on PyPy it is a _rawffi.CDLL class instance.  The 
ctypes/__init__.py file was already modified to this effect and the patch 
added support to converting on int into a _rawffi.CDLL class instance.  

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150827/3baa1433/attachment-0001.sig>

From vaibhav_sood at persistent.com  Thu Aug 27 05:46:35 2015
From: vaibhav_sood at persistent.com (Vaibhav Sood)
Date: Thu, 27 Aug 2015 03:46:35 +0000
Subject: [pypy-dev] [newb] Regarding pull requests
Message-ID: <4176FFD2EDA9C24CA9AB011273CFDDC8025BD356@HJ-MBX1.persistent.co.in>

Hi,

I had submitted a pull request earlier (https://bitbucket.org/pypy/pypy/pull-requests/331/fix-test-case-for-py33-branch/diff) for a nightly test case fix on the py3.3 branch. I haven't received any reviews, just wanted to check regarding the same, I apologize if this is not the right way to prod but thought I'll ask

I also had a query regarding submitting pull requests (this is my first time submitting one), do I need to specify somebody as a reviewer when creating the PR? Or do I just submit and wait for a review?

Thanks you!

Regards,
Vaibhav


DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150827/29ade87f/attachment.html>

From ronan.lamy at gmail.com  Thu Aug 27 06:02:00 2015
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Thu, 27 Aug 2015 05:02:00 +0100
Subject: [pypy-dev] [newb] Regarding pull requests
In-Reply-To: <4176FFD2EDA9C24CA9AB011273CFDDC8025BD356@HJ-MBX1.persistent.co.in>
References: <4176FFD2EDA9C24CA9AB011273CFDDC8025BD356@HJ-MBX1.persistent.co.in>
Message-ID: <55DE8BB8.4090409@gmail.com>

Le 27/08/15 04:46, Vaibhav Sood a ?crit :
> Hi,
>
> I had submitted a pull request earlier
> (https://bitbucket.org/pypy/pypy/pull-requests/331/fix-test-case-for-py33-branch/diff)
> for a nightly test case fix on the py3.3 branch. I haven?t received any
> reviews, just wanted to check regarding the same, I apologize if this is
> not the right way to prod but thought I?ll ask
>
> I also had a query regarding submitting pull requests (this is my first
> time submitting one), do I need to specify somebody as a reviewer when
> creating the PR? Or do I just submit and wait for a review?

I'm sorry that your PR wasn't reviewed sooner - I've done so now. You 
don't need to set a reviewer, it'll be reviewed by whoever feels like 
doing it.

Cheers,
Ronan


From andrey.ayupov at gmail.com  Thu Aug 27 20:23:00 2015
From: andrey.ayupov at gmail.com (Andrey Ayupov)
Date: Thu, 27 Aug 2015 11:23:00 -0700
Subject: [pypy-dev] type inference error
Message-ID: <CA+amRpX0xiPVzKy43x9JxVMC7Xx10th5+UU01yEHBhQnRVn+-Q@mail.gmail.com>

Hello,

Hopefully a very simple question. I am trying out RPython toolchain. pypy
2.6, python 2.7. I am using the translator shell.

Here's my simple example:

python bin/translatorshell.py

class mtype():
    def __init__(self):
        self.x = 0;

def mmm(x):
    z = x.x + 1
    return z

t = Translation(mmm, [mtype])

t.annotate()

and it gives me the following. if you put mtype class and mmm function
definitions in a separate module, then you can see it complains about this
line:     z = x.x + 1. Why can't it infer type for all arguments and the
result here? I know i am missing something basic.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/interactive.py",
line 89, in annotate
    return self.driver.annotate()
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/driver.py",
line 106, in proc
    return self.proceed(backend_goal)
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/driver.py",
line 549, in proceed
    return self._execute(goals, task_skip = self._maybe_skip())
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/tool/taskengine.py",
line 114, in _execute
    res = self._do(goal, taskcallable, *args, **kwds)
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/driver.py",
line 277, in _do
    res = func()
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/translator/driver.py",
line 314, in task_annotate
    s = annotator.build_types(self.entry_point, self.inputtypes)
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/annotator/annrpython.py",
line 89, in build_types
    return self.build_graph_types(flowgraph, inputcells,
complete_now=complete_now)
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/annotator/annrpython.py",
line 135, in build_graph_types
    self.complete()
  File
"/nfs/site/disks/scl.work.58/ppt/aayupov/hld/pypy-2.6.0-src/rpython/annotator/annrpython.py",
line 211, in complete
    raise annmodel.AnnotatorError(text)
rpython.annotator.model.AnnotatorError:

Blocked block -- operation cannot succeed

    v0 = getattr(x_0, ('x'))

In <FunctionGraph of (__main__:1)mmm at 0x7ffff2b69ed0>:
no source!
Known variable annotations:
 x_0 = SomeInstance(can_be_None=False, classdef=__main__.mtype)


Thanks!
Andrey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150827/0ba80a67/attachment.html>

From william.leslie.ttg at gmail.com  Fri Aug 28 05:34:15 2015
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Fri, 28 Aug 2015 13:34:15 +1000
Subject: [pypy-dev] type inference error
In-Reply-To: <CA+amRpX0xiPVzKy43x9JxVMC7Xx10th5+UU01yEHBhQnRVn+-Q@mail.gmail.com>
References: <CA+amRpX0xiPVzKy43x9JxVMC7Xx10th5+UU01yEHBhQnRVn+-Q@mail.gmail.com>
Message-ID: <CAHgd1hG6sQsWbxq3++2PhK9aD50jdMLbjdgXkBb+3woGUdUhpA@mail.gmail.com>

On 28 August 2015 at 04:23, Andrey Ayupov <andrey.ayupov at gmail.com> wrote:
> Hello,
>
> Hopefully a very simple question. I am trying out RPython toolchain. pypy
> 2.6, python 2.7. I am using the translator shell.
>
> Here's my simple example:
>
> python bin/translatorshell.py
>
> class mtype():
>     def __init__(self):
>         self.x = 0;
>
> def mmm(x):
>     z = x.x + 1
>     return z
>
> t = Translation(mmm, [mtype])
>
> t.annotate()
>
> and it gives me the following. if you put mtype class and mmm function
> definitions in a separate module, then you can see it complains about this
> line:     z = x.x + 1. Why can't it infer type for all arguments and the
> result here? I know i am missing something basic.

It never saw the creation of an mytpe, so it doesn't know that mtypes
have an x attribute, let alone know what type it has.  Try a bigger
entry point:

def entry(x, y):
    m = mtype()
    return mmm(m)

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

From william.leslie.ttg at gmail.com  Fri Aug 28 05:35:47 2015
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Fri, 28 Aug 2015 13:35:47 +1000
Subject: [pypy-dev] type inference error
In-Reply-To: <CAHgd1hG6sQsWbxq3++2PhK9aD50jdMLbjdgXkBb+3woGUdUhpA@mail.gmail.com>
References: <CA+amRpX0xiPVzKy43x9JxVMC7Xx10th5+UU01yEHBhQnRVn+-Q@mail.gmail.com>
 <CAHgd1hG6sQsWbxq3++2PhK9aD50jdMLbjdgXkBb+3woGUdUhpA@mail.gmail.com>
Message-ID: <CAHgd1hFhr4iCa7nGyJNWVTrBFKxV1WgO0gi_bYQuwSvir31ymA@mail.gmail.com>

On 28 August 2015 at 13:34, William ML Leslie
<william.leslie.ttg at gmail.com> wrote:
> def entry(x, y):
>     m = mtype()
>     return mmm(m)
>

er I meant:

def entry():
    m = mtype()
    return mmm(m)

t = Translation(entry, [])

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

From matti.picus at gmail.com  Sun Aug 30 00:00:25 2015
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 30 Aug 2015 01:00:25 +0300
Subject: [pypy-dev] PyPy 2.6.1 release candidates available on bitbucket,
 please test and report
Message-ID: <55E22B79.3050009@gmail.com>

I uploaded 2.6.1 release candidates to 
https://bitbucket.org/pypy/pypy/downloads , please make sure it is 
packaged properly for your platform and/or that the source bundles work 
for your distro

Comments, criticisms, and critiques of the release notes would be nice too,
http://doc.pypy.org/en/release-2.6.x/release-2.6.1.html

Matti

From hengha.mao at gmail.com  Mon Aug 31 11:00:04 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Mon, 31 Aug 2015 17:00:04 +0800
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
Message-ID: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>

Hi,

We observed the program was cored dump. And examing the core file, we found
the call stack:

#0  0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from
/usr/local/pypy/libpypy-c.so
#1  0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from
/usr/local/pypy/libpypy-c.so
#2  0x00007f26487a1ad4 in
pypy_g_appexec___src__glob___________________import_sys () from
/usr/local/pypy/libpypy-c.so
#3  0x00007f26486f980b in pypy_g.pypy_execute_source () from
/usr/local/pypy/libpypy-c.so
#4  0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from
/usr/local/pypy/libpypy-c.so
#5  0x00007f26484e6f42 in pypy_execute_source_ptr () from
/usr/local/pypy/libpypy-c.so
#6  0x00007f26484e6d82 in pypy_execute_source () from
/usr/local/pypy/libpypy-c.so
...

And debugging the core file, the char* buffer passed to
pypy_execute_source() was:
(gdb) print pyExBuffer
$1 = "import
sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...

The code was to add a directory to sys.path().
The error was not reproduceable by extracting the piece of code from the
whole programme, and I've no ideas why the segmentation fault happened.

Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And are
there any methods to debug and found out the cause?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/d88d205c/attachment.html>

From fijall at gmail.com  Mon Aug 31 11:15:22 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 31 Aug 2015 11:15:22 +0200
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
In-Reply-To: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
References: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
Message-ID: <CAK5idxQTHeEvjQe7j9RUj-nhUv_HjZkV0Py-SgWRgm9xxa7XuQ@mail.gmail.com>

Can you please file a bug on either pypy or vmprof-python?

On Mon, Aug 31, 2015 at 11:00 AM, Yicong Huang <hengha.mao at gmail.com> wrote:
> Hi,
>
> We observed the program was cored dump. And examing the core file, we found
> the call stack:
>
> #0  0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from
> /usr/local/pypy/libpypy-c.so
> #1  0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from
> /usr/local/pypy/libpypy-c.so
> #2  0x00007f26487a1ad4 in
> pypy_g_appexec___src__glob___________________import_sys () from
> /usr/local/pypy/libpypy-c.so
> #3  0x00007f26486f980b in pypy_g.pypy_execute_source () from
> /usr/local/pypy/libpypy-c.so
> #4  0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from
> /usr/local/pypy/libpypy-c.so
> #5  0x00007f26484e6f42 in pypy_execute_source_ptr () from
> /usr/local/pypy/libpypy-c.so
> #6  0x00007f26484e6d82 in pypy_execute_source () from
> /usr/local/pypy/libpypy-c.so
> ...
>
> And debugging the core file, the char* buffer passed to
> pypy_execute_source() was:
> (gdb) print pyExBuffer
> $1 = "import
> sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...
>
> The code was to add a directory to sys.path().
> The error was not reproduceable by extracting the piece of code from the
> whole programme, and I've no ideas why the segmentation fault happened.
>
> Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And are
> there any methods to debug and found out the cause?
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Mon Aug 31 11:19:27 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 31 Aug 2015 11:19:27 +0200
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
In-Reply-To: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
References: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
Message-ID: <CAMSv6X2vy6AcsTrYHi68+M6FHc41aGN+u=B4Aj=QU16og+XHMg@mail.gmail.com>

Hi Yicong,

On 31 August 2015 at 11:00, Yicong Huang <hengha.mao at gmail.com> wrote:
> pypy_execute_source()

Did you call rpython_startup_code() and pypy_setup_home() before?

Armin

From hengha.mao at gmail.com  Mon Aug 31 11:29:31 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Mon, 31 Aug 2015 17:29:31 +0800
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
In-Reply-To: <CAMSv6X2vy6AcsTrYHi68+M6FHc41aGN+u=B4Aj=QU16og+XHMg@mail.gmail.com>
References: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
 <CAMSv6X2vy6AcsTrYHi68+M6FHc41aGN+u=B4Aj=QU16og+XHMg@mail.gmail.com>
Message-ID: <CAOEzsGAJVwMwzY0eTwzb5Dsu6Eu4b7B8G6u-=TEeo_r4vHt9Zw@mail.gmail.com>

Yes.
And if rpython_startup_code() and pypy_setup_home()  was not called before,
the core dump function would be at:
pypy_g_IncrementalMiniMarkGC_collect_and_reserve ()

On Mon, Aug 31, 2015 at 5:19 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Yicong,
>
> On 31 August 2015 at 11:00, Yicong Huang <hengha.mao at gmail.com> wrote:
> > pypy_execute_source()
>
> Did you call rpython_startup_code() and pypy_setup_home() before?
>
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/7c64310d/attachment.html>

From arigo at tunes.org  Mon Aug 31 11:35:59 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 31 Aug 2015 11:35:59 +0200
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
In-Reply-To: <CAOEzsGAJVwMwzY0eTwzb5Dsu6Eu4b7B8G6u-=TEeo_r4vHt9Zw@mail.gmail.com>
References: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
 <CAMSv6X2vy6AcsTrYHi68+M6FHc41aGN+u=B4Aj=QU16og+XHMg@mail.gmail.com>
 <CAOEzsGAJVwMwzY0eTwzb5Dsu6Eu4b7B8G6u-=TEeo_r4vHt9Zw@mail.gmail.com>
Message-ID: <CAMSv6X3yj02AtmBowtsxaJf1b+W4qrk7y1wS3M+_TW2KH2ubbA@mail.gmail.com>

Hi Yicong,

On 31 August 2015 at 11:29, Yicong Huang <hengha.mao at gmail.com> wrote:
> Yes.

Ok.  Fwiw, it's unlikely we can figure out what is wrong without being
able to reproduce the crash ourselves...


Armin

From hengha.mao at gmail.com  Mon Aug 31 11:41:10 2015
From: hengha.mao at gmail.com (Yicong Huang)
Date: Mon, 31 Aug 2015 17:41:10 +0800
Subject: [pypy-dev] Segmentation fault at rpyvmprof_f_pypy_rrr ()
In-Reply-To: <CAK5idxQTHeEvjQe7j9RUj-nhUv_HjZkV0Py-SgWRgm9xxa7XuQ@mail.gmail.com>
References: <CAOEzsGB2b9KU6Kkv5Rb+_8O=RNoChmmAFnjqyOL02G97eyTbjw@mail.gmail.com>
 <CAK5idxQTHeEvjQe7j9RUj-nhUv_HjZkV0Py-SgWRgm9xxa7XuQ@mail.gmail.com>
Message-ID: <CAOEzsGD--OiDC4DmGdHOxkb-cX5u5MdCqX=LSUw+t1TKv--wzA@mail.gmail.com>

I am not sure whether it is a bug of pypy.
I tried extracting the code from the whole programme, but it worked
correctly.

rpython_startup_code();
int res = pypy_setup_home("/usr/local/pypy/libpypy-c.so", 1);
char *b = "import
sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\00010260";
pypy_execute_source(b);

I will do some further experiments. And if there are any methods to debug,
I could follow.

Thanks!


On Mon, Aug 31, 2015 at 5:15 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Can you please file a bug on either pypy or vmprof-python?
>
> On Mon, Aug 31, 2015 at 11:00 AM, Yicong Huang <hengha.mao at gmail.com>
> wrote:
> > Hi,
> >
> > We observed the program was cored dump. And examing the core file, we
> found
> > the call stack:
> >
> > #0  0x00007f26487e6b82 in rpyvmprof_f_pypy_rrr () from
> > /usr/local/pypy/libpypy-c.so
> > #1  0x00007f26494a797a in rpyvmprof_t_pypy_rrr () from
> > /usr/local/pypy/libpypy-c.so
> > #2  0x00007f26487a1ad4 in
> > pypy_g_appexec___src__glob___________________import_sys () from
> > /usr/local/pypy/libpypy-c.so
> > #3  0x00007f26486f980b in pypy_g.pypy_execute_source () from
> > /usr/local/pypy/libpypy-c.so
> > #4  0x00007f26486f9994 in pypy_g_pypy_execute_source_ptr () from
> > /usr/local/pypy/libpypy-c.so
> > #5  0x00007f26484e6f42 in pypy_execute_source_ptr () from
> > /usr/local/pypy/libpypy-c.so
> > #6  0x00007f26484e6d82 in pypy_execute_source () from
> > /usr/local/pypy/libpypy-c.so
> > ...
> >
> > And debugging the core file, the char* buffer passed to
> > pypy_execute_source() was:
> > (gdb) print pyExBuffer
> > $1 = "import
> >
> sys\nsys.path.insert(0,'job_runtime/lib/python2.7/site-packages/udf')\000/1440989742122"...
> >
> > The code was to add a directory to sys.path().
> > The error was not reproduceable by extracting the piece of code from the
> > whole programme, and I've no ideas why the segmentation fault happened.
> >
> > Does anyone knows what the function "rpyvmprof_f_pypy_rrr ()" did? And
> are
> > there any methods to debug and found out the cause?
> >
> >
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/49e7e3e0/attachment-0001.html>

From hamamatsu at gmx.de  Mon Aug 31 11:54:03 2015
From: hamamatsu at gmx.de (Arno Waschk)
Date: Mon, 31 Aug 2015 11:54:03 +0200
Subject: [pypy-dev] avoid pypy's csv module
Message-ID: <55E4243B.6080908@gmx.de>

dear list,

i have a problem with pypy's csv module.

Apart from it being somewhat slower than cPython's (i could live with 
that) i am getting rarely but annoyingly corrupted files written, which 
have some single corrupted lines.
I cannot provide enough information (yet) for debugging, it looks like a 
writing race condition, although i am 120% sure that my program does 
never more than one writing event at any moment.

Everything worked fine in cpython (for years), so my question is, can i 
simply tell pypy to run my program, but use CPython's implementation of 
the csv module?

thanks, Arno






-- 
Arno Waschk arno at arnowaschk.de +491723149605 +436508950949 (nur w?hrend 
Projekten in ?sterreich) current and upcoming projects: Wien Burgtheater 
Goethe: *Faust I* (Repertoire) Matthias Hartmann Berlin Staatsoper im 
Schillertheater Satie: Le Pi?ge de M?duse --- *Satiesfactionen: Wissen 
Sie wie man T?ne reinigt?* (Repertoire) J?rgen Flimm 
Neuhardenberg/Leipzig u. a. *Lesungen mit Klaus Maria Brandauer* *Via 
intolleranza II* Christoph Schlingensief Wien Burgtheater *Mea Culpa* 
(Repertoire) Christoph Schlingensief Berlin Staatsoper im 
Schillertheater Maxwell Davies & Sciarrino: *Miss Donnithorne's Maggot 
// Infinito Nero (Repertoire)* Staatskapelle Berlin Michael von zur 
M?hlen *franz. Klaviertrios* Bonn Trio Cartier Freiburg *Fanny und 
Alexander* Regie Julia H?lscher *CRUSOE*Staatstheater Oldenburg Regie: 
Julia H?lscher *Die weisse Rose*Theater Koblenz Regie: Anja Nicklich 
*Wie ich Welt wurde (AT)* Oper Z?rich Regie: Hans Neuenfels
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/f5d0d120/attachment.html>

From fijall at gmail.com  Mon Aug 31 12:01:52 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 31 Aug 2015 12:01:52 +0200
Subject: [pypy-dev] avoid pypy's csv module
In-Reply-To: <55E4243B.6080908@gmx.de>
References: <55E4243B.6080908@gmx.de>
Message-ID: <CAK5idxSiA8GQkrL5TzCoQUOBi-8+ahAYO6-uqa=wAo+Hc=SZZg@mail.gmail.com>

Hi.

Are you using multiple threads and relying on atomicity of some write
instructions?

On Mon, Aug 31, 2015 at 11:54 AM, Arno Waschk <hamamatsu at gmx.de> wrote:
> dear list,
>
> i have a problem with pypy's csv module.
>
> Apart from it being somewhat slower than cPython's (i could live with that)
> i am getting rarely but annoyingly corrupted files written, which have some
> single corrupted lines.
> I cannot provide enough information (yet) for debugging, it looks like a
> writing race condition, although i am 120% sure that my program does never
> more than one writing event at any moment.
>
> Everything worked fine in cpython (for years), so my question is, can i
> simply tell pypy to run my program, but use CPython's implementation of the
> csv module?
>
> thanks, Arno
>
>
>
>
>
>
> --
> Arno Waschk arno at arnowaschk.de +491723149605 +436508950949 (nur w?hrend
> Projekten in ?sterreich) current and upcoming projects: Wien Burgtheater
> Goethe: Faust I (Repertoire) Matthias Hartmann Berlin Staatsoper im
> Schillertheater Satie: Le Pi?ge de M?duse --- Satiesfactionen: Wissen Sie
> wie man T?ne reinigt? (Repertoire) J?rgen Flimm Neuhardenberg/Leipzig u. a.
> Lesungen mit Klaus Maria Brandauer Via intolleranza II Christoph
> Schlingensief Wien Burgtheater Mea Culpa (Repertoire) Christoph
> Schlingensief Berlin Staatsoper im Schillertheater Maxwell Davies &
> Sciarrino: Miss Donnithorne's Maggot // Infinito Nero (Repertoire)
> Staatskapelle Berlin Michael von zur M?hlen franz. Klaviertrios Bonn Trio
> Cartier Freiburg Fanny und Alexander Regie Julia H?lscher
> CRUSOEStaatstheater Oldenburg Regie: Julia H?lscher Die weisse RoseTheater
> Koblenz Regie: Anja Nicklich Wie ich Welt wurde (AT) Oper Z?rich Regie: Hans
> Neuenfels
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Mon Aug 31 19:36:04 2015
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 31 Aug 2015 19:36:04 +0200
Subject: [pypy-dev] avoid pypy's csv module
In-Reply-To: <55E4243B.6080908@gmx.de>
References: <55E4243B.6080908@gmx.de>
Message-ID: <CAMSv6X1U7oR8NaYUp_yiTqQBj49_pB5X2N88Wiz1QV5Cf2W=Xg@mail.gmail.com>

Hi Arno,

On Mon, Aug 31, 2015 at 11:54 AM, Arno Waschk <hamamatsu at gmx.de> wrote:
> i have a problem with pypy's csv module.
>
> Apart from it being somewhat slower than cPython's (i could live with that)
> i am getting rarely but annoyingly corrupted files written, which have some
> single corrupted lines.

We'll be waiting for a reproducible example to debug that.  In the
meantime, well, by trying hard enough you can indeed use CPython's
implementation of the csv module.  It is going to be very, very,
*extremely* slow (because a huge number of objects have to cross the
barrier to CPython land).  But if you want, you can try to copy from
CPython's source code the file Module/_csv.c, write a small setup.py,
and compile it with "pypy setup.py build".


A bient?t,

Armin.

From skip.montanaro at gmail.com  Mon Aug 31 22:09:48 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 31 Aug 2015 15:09:48 -0500
Subject: [pypy-dev] Dependencies in noon-standard locations
Message-ID: <CANc-5Uy34FKvn8cC+nfwf75Udhy4d8grsPpvdSCMj0N+pCgtsg@mail.gmail.com>

I've encountered a problem building the 2.6.1 distribution from source:

[translation:ERROR] /tmp/usession-release-2.6.1-0/platcheck_54.c:79:19:
error: expat.h: No such file or directory

I don't remember encountering this before, but most of my earlier builds
were probably on Mac OS X.

My guess is the error results from a non-standard expat location:

% find /opt/TWWfsw/ -name 'expat.h'
/opt/TWWfsw/libexpat20/include/expat.h

I saw nothing about how to deal with this in the instructions. Searching
the source, I saw no instances of things like LDFLAGS or CPPFLAGS which I'm
used to using in other build environments. How do I specify these oddball
locations to PyPy?

I'm using OpenSuSE 12.2, which I doubt will work with any of the PyPy
binaries. Would love to be proved incorrect though.

Thx,

Skip Montanaro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/2e6616e6/attachment.html>

From fijall at gmail.com  Mon Aug 31 22:16:00 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 31 Aug 2015 22:16:00 +0200
Subject: [pypy-dev] Dependencies in noon-standard locations
In-Reply-To: <CANc-5Uy34FKvn8cC+nfwf75Udhy4d8grsPpvdSCMj0N+pCgtsg@mail.gmail.com>
References: <CANc-5Uy34FKvn8cC+nfwf75Udhy4d8grsPpvdSCMj0N+pCgtsg@mail.gmail.com>
Message-ID: <CAK5idxS5Lpts_TqxxF_rr9oCEAp36DYf9CeK1M3j2dLFVWQjdQ@mail.gmail.com>

pypy should respect LDFLAGS and CFLAGS (but indeed not CPPFLAGS)

On Mon, Aug 31, 2015 at 10:09 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> I've encountered a problem building the 2.6.1 distribution from source:
>
> [translation:ERROR] /tmp/usession-release-2.6.1-0/platcheck_54.c:79:19:
> error: expat.h: No such file or directory
>
> I don't remember encountering this before, but most of my earlier builds
> were probably on Mac OS X.
>
> My guess is the error results from a non-standard expat location:
>
> % find /opt/TWWfsw/ -name 'expat.h'
> /opt/TWWfsw/libexpat20/include/expat.h
>
> I saw nothing about how to deal with this in the instructions. Searching the
> source, I saw no instances of things like LDFLAGS or CPPFLAGS which I'm used
> to using in other build environments. How do I specify these oddball
> locations to PyPy?
>
> I'm using OpenSuSE 12.2, which I doubt will work with any of the PyPy
> binaries. Would love to be proved incorrect though.
>
> Thx,
>
> Skip Montanaro
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From skip.montanaro at gmail.com  Mon Aug 31 22:47:14 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 31 Aug 2015 15:47:14 -0500
Subject: [pypy-dev] Dependencies in noon-standard locations
In-Reply-To: <CAK5idxS5Lpts_TqxxF_rr9oCEAp36DYf9CeK1M3j2dLFVWQjdQ@mail.gmail.com>
References: <CANc-5Uy34FKvn8cC+nfwf75Udhy4d8grsPpvdSCMj0N+pCgtsg@mail.gmail.com>
 <CAK5idxS5Lpts_TqxxF_rr9oCEAp36DYf9CeK1M3j2dLFVWQjdQ@mail.gmail.com>
Message-ID: <CANc-5Uydf28zg+SmXbLw-1vEp5uMfkrO1sYXHwd6aur52ijyyw@mail.gmail.com>

On Mon, Aug 31, 2015 at 3:16 PM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> pypy should respect LDFLAGS and CFLAGS (but indeed not CPPFLAGS)


Thanks. I gave that a try, but got the same error:

blade% pwd
/home/skipm/3rdParty/pypy-2.6.1-src/pypy/goal
blade% LDFLAGS='-L /opt/TWWfsw/libexpat20/lib' CFLAGS='-I
/opt/TWWfsw/libexpat20/Include' python ../../rpython/bin/rpython -Ojit
targetpypystandalone
... lots of output elided ...
[platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall
-Wno-unused -I /opt/TWWfsw/libexpat20/Include
/tmp/usession-release-2.6.1-2/platcheck_54.c -o
/tmp/usession-release-2.6.1-2/platcheck_54.o
...
[translation:ERROR] CompilationError: CompilationError(err="""
[translation:ERROR] /tmp/usession-release-2.6.1-2/platcheck_54.c:79:19:
error: expat.h: No such file or directory
[translation:ERROR] """)


Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150831/2f3617dd/attachment.html>

From fijall at gmail.com  Mon Aug 31 23:04:59 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 31 Aug 2015 23:04:59 +0200
Subject: [pypy-dev] Dependencies in noon-standard locations
In-Reply-To: <CANc-5Uydf28zg+SmXbLw-1vEp5uMfkrO1sYXHwd6aur52ijyyw@mail.gmail.com>
References: <CANc-5Uy34FKvn8cC+nfwf75Udhy4d8grsPpvdSCMj0N+pCgtsg@mail.gmail.com>
 <CAK5idxS5Lpts_TqxxF_rr9oCEAp36DYf9CeK1M3j2dLFVWQjdQ@mail.gmail.com>
 <CANc-5Uydf28zg+SmXbLw-1vEp5uMfkrO1sYXHwd6aur52ijyyw@mail.gmail.com>
Message-ID: <CAK5idxTbiRJOjC09wq+Gi0S4dnPuuGVecb61HM6UeHANnXF+TA@mail.gmail.com>

welll...... as you can see the C flag got passed through, so either
you have a typo or something is off. You can see what sort of gcc
invocation was there, so execute it and figure out why it's not
working

On Mon, Aug 31, 2015 at 10:47 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
>
> On Mon, Aug 31, 2015 at 3:16 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>>
>> pypy should respect LDFLAGS and CFLAGS (but indeed not CPPFLAGS)
>
>
> Thanks. I gave that a try, but got the same error:
>
> blade% pwd
> /home/skipm/3rdParty/pypy-2.6.1-src/pypy/goal
> blade% LDFLAGS='-L /opt/TWWfsw/libexpat20/lib' CFLAGS='-I
> /opt/TWWfsw/libexpat20/Include' python ../../rpython/bin/rpython -Ojit
> targetpypystandalone
> ... lots of output elided ...
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall
> -Wno-unused -I /opt/TWWfsw/libexpat20/Include
> /tmp/usession-release-2.6.1-2/platcheck_54.c -o
> /tmp/usession-release-2.6.1-2/platcheck_54.o
> ...
> [translation:ERROR] CompilationError: CompilationError(err="""
> [translation:ERROR] /tmp/usession-release-2.6.1-2/platcheck_54.c:79:19:
> error: expat.h: No such file or directory
> [translation:ERROR] """)
>
>
> Skip
>
>