From stefan_ml at behnel.de  Mon Aug  1 08:48:37 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Aug 2011 08:48:37 +0200
Subject: [Cython] OpenMP problem
In-Reply-To: <CANg26EVZv6ngFqh7ouP8SfmCQMPYbfZ0nL6wf+2hECL-t2LqxA@mail.gmail.com>
References: <CAKGHGPSYrhxmrMMT===65zFswuPog0J+a_QV4M0f6==eKRF++A@mail.gmail.com>	<CANg26EU8w+ytQ84cu28-ERiMZvPVy5OCzBb13=-BOxU-sY19dA@mail.gmail.com>
	<CANg26EVZv6ngFqh7ouP8SfmCQMPYbfZ0nL6wf+2hECL-t2LqxA@mail.gmail.com>
Message-ID: <4E364C45.7020400@behnel.de>

mark florisson, 31.07.2011 22:08:
> I pushed it to release, should I rebase master on release now?

Well, so far, the release branch hasn't been used and rc1 was created from 
master. All current changes in master are either considered safe (by me :) 
or just fixes that should go into the release. So I think it's best to just 
rebase your single commit in the current release branch back on the master, 
delete the release branch and recreate it from the tip of the master branch.

Stefan

From markflorisson88 at gmail.com  Mon Aug  1 10:00:37 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 1 Aug 2011 10:00:37 +0200
Subject: [Cython] OpenMP problem
In-Reply-To: <4E364C45.7020400@behnel.de>
References: <CAKGHGPSYrhxmrMMT===65zFswuPog0J+a_QV4M0f6==eKRF++A@mail.gmail.com>
	<CANg26EU8w+ytQ84cu28-ERiMZvPVy5OCzBb13=-BOxU-sY19dA@mail.gmail.com>
	<CANg26EVZv6ngFqh7ouP8SfmCQMPYbfZ0nL6wf+2hECL-t2LqxA@mail.gmail.com>
	<4E364C45.7020400@behnel.de>
Message-ID: <CANg26EWMUtVj8Fi2b12r6KW6TnNLStMk5dsHRDQAKGibgM9PyA@mail.gmail.com>

On 1 August 2011 08:48, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 31.07.2011 22:08:
>>
>> I pushed it to release, should I rebase master on release now?
>
> Well, so far, the release branch hasn't been used and rc1 was created from
> master. All current changes in master are either considered safe (by me :)
> or just fixes that should go into the release. So I think it's best to just
> rebase your single commit in the current release branch back on the master,
> delete the release branch and recreate it from the tip of the master branch.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Ok, thanks Dag and Stefan, I suspected the danger, but if Stefan's
changes are to go in I'll just do as Stefan suggested.

From arfrever.fta at gmail.com  Tue Aug  2 00:04:56 2011
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Tue, 2 Aug 2011 00:04:56 +0200
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>
	<CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>
Message-ID: <201108020004.58249.Arfrever.FTA@gmail.com>

2011-07-31 09:45:15 Robert Bradshaw napisa?(a):
> On Thu, Jul 21, 2011 at 4:14 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
> > Cython has seen an enormous amount of development since 0.14.1. If you
> > are not already using the latest version from the development
> > repository, we encourage you to try out the release candidate:
> > http://cython.org/release/Cython-0.15rc0.tar.gz
> 
> Thanks for all the feedback. Here's another release candidate:
> http://cython.org/release/Cython-0.15rc0.tar.gz Unless something major
> is found, this is likely to be the release.

Cython-0.15rc1.tar.gz contains tests/imports/cimport_on_import.srctree, which was
absent in Cython-0.15rc0.tar.gz and is absent on https://github.com/cython/cython.
The test from this file fails:

End-to-end cimport_on_import ... /usr/bin/python2.6 setup.py build_ext --inplace
Compiling cy_module.pyx because it changed.
Compiling main.pyx because it changed.


Error compiling Cython file:
------------------------------------------------------------
...

from cy_module import A, B
from py_module import py_object

assert typeof(A) == "type object", typeof(A)
cdef A a = A()
    ^
------------------------------------------------------------

main.pyx:10:5: 'A' is not a type identifier
Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    ext_modules = cythonize("*.pyx"),
  File "/var/tmp/portage/dev-python/cython-0.15_rc1/work/Cython-0.15rc1/Cython/Build/Dependencies.py", line 508, in cythonize
    cythonize_one(pyx_file, c_file, options)
  File "/var/tmp/portage/dev-python/cython-0.15_rc1/work/Cython-0.15rc1/Cython/Build/Dependencies.py", line 529, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: main.pyx
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named main

FAIL

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110802/79ce86d9/attachment.pgp>

From robertwb at math.washington.edu  Tue Aug  2 00:58:56 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Aug 2011 15:58:56 -0700
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <201108020004.58249.Arfrever.FTA@gmail.com>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>
	<CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>
	<201108020004.58249.Arfrever.FTA@gmail.com>
Message-ID: <CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>

On Mon, Aug 1, 2011 at 3:04 PM, Arfrever Frehtes Taifersar Arahesis
<arfrever.fta at gmail.com> wrote:
> 2011-07-31 09:45:15 Robert Bradshaw napisa?(a):
>> On Thu, Jul 21, 2011 at 4:14 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>> > Cython has seen an enormous amount of development since 0.14.1. If you
>> > are not already using the latest version from the development
>> > repository, we encourage you to try out the release candidate:
>> > http://cython.org/release/Cython-0.15rc0.tar.gz
>>
>> Thanks for all the feedback. Here's another release candidate:
>> http://cython.org/release/Cython-0.15rc0.tar.gz Unless something major
>> is found, this is likely to be the release.
>
> Cython-0.15rc1.tar.gz contains tests/imports/cimport_on_import.srctree, which was
> absent in Cython-0.15rc0.tar.gz and is absent on https://github.com/cython/cython.
> The test from this file fails:

And this is why I should always make sdists in a clean client...
Thanks for catching this. A clean tarball is up at
http://cython.org/release/Cython-0.15rc2.tar.gz . (I also took a quick
look at what's been pushed since rc1, and they all look like safe
changes worth getting in.)

- Robert

From robertwb at math.washington.edu  Tue Aug  2 02:42:25 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Aug 2011 17:42:25 -0700
Subject: [Cython] OpenMP problem
In-Reply-To: <4E364C45.7020400@behnel.de>
References: <CAKGHGPSYrhxmrMMT===65zFswuPog0J+a_QV4M0f6==eKRF++A@mail.gmail.com>
	<CANg26EU8w+ytQ84cu28-ERiMZvPVy5OCzBb13=-BOxU-sY19dA@mail.gmail.com>
	<CANg26EVZv6ngFqh7ouP8SfmCQMPYbfZ0nL6wf+2hECL-t2LqxA@mail.gmail.com>
	<4E364C45.7020400@behnel.de>
Message-ID: <CADiQ+QDw1Qqx9c1qQRvFGw_pwrp0Nxa__BCdOF=QN4npS2ybLQ@mail.gmail.com>

On Sun, Jul 31, 2011 at 11:48 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 31.07.2011 22:08:
>>
>> I pushed it to release, should I rebase master on release now?
>
> Well, so far, the release branch hasn't been used and rc1 was created from
> master. All current changes in master are either considered safe (by me :)
> or just fixes that should go into the release.

I looked at them and I'd say they're both safe and worthy of going in.
In general, if we have a distinct release branch (e.g if anything
really interesting had gone in since the last rc), I'd say we should
push release fixes to that and then merge into master.

> So I think it's best to just
> rebase your single commit in the current release branch back on the master,
> delete the release branch and recreate it from the tip of the master branch.

Yep, I'm seeing it. Thanks.

- Robert

From vitja.makarov at gmail.com  Tue Aug  2 07:44:17 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Aug 2011 09:44:17 +0400
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>
	<CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>
	<201108020004.58249.Arfrever.FTA@gmail.com>
	<CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>
Message-ID: <CAKGHGPRXYU-4vfcWpz5q8TitLQu-yd76Z6weKcRo_n+TWDjfhQ@mail.gmail.com>

2011/8/2 Robert Bradshaw <robertwb at math.washington.edu>:
> On Mon, Aug 1, 2011 at 3:04 PM, Arfrever Frehtes Taifersar Arahesis
> <arfrever.fta at gmail.com> wrote:
>> 2011-07-31 09:45:15 Robert Bradshaw napisa?(a):
>>> On Thu, Jul 21, 2011 at 4:14 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>> > Cython has seen an enormous amount of development since 0.14.1. If you
>>> > are not already using the latest version from the development
>>> > repository, we encourage you to try out the release candidate:
>>> > http://cython.org/release/Cython-0.15rc0.tar.gz
>>>
>>> Thanks for all the feedback. Here's another release candidate:
>>> http://cython.org/release/Cython-0.15rc0.tar.gz Unless something major
>>> is found, this is likely to be the release.
>>
>> Cython-0.15rc1.tar.gz contains tests/imports/cimport_on_import.srctree, which was
>> absent in Cython-0.15rc0.tar.gz and is absent on https://github.com/cython/cython.
>> The test from this file fails:
>
> And this is why I should always make sdists in a clean client...
> Thanks for catching this. A clean tarball is up at
> http://cython.org/release/Cython-0.15rc2.tar.gz . (I also took a quick
> look at what's been pushed since rc1, and they all look like safe
> changes worth getting in.)
>

Maybe it's better to use "git archive"?

-- 
vitja.

From stefan_ml at behnel.de  Tue Aug  2 08:18:42 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Aug 2011 08:18:42 +0200
Subject: [Cython] OpenMP problem
In-Reply-To: <CADiQ+QDw1Qqx9c1qQRvFGw_pwrp0Nxa__BCdOF=QN4npS2ybLQ@mail.gmail.com>
References: <CAKGHGPSYrhxmrMMT===65zFswuPog0J+a_QV4M0f6==eKRF++A@mail.gmail.com>	<CANg26EU8w+ytQ84cu28-ERiMZvPVy5OCzBb13=-BOxU-sY19dA@mail.gmail.com>	<CANg26EVZv6ngFqh7ouP8SfmCQMPYbfZ0nL6wf+2hECL-t2LqxA@mail.gmail.com>	<4E364C45.7020400@behnel.de>
	<CADiQ+QDw1Qqx9c1qQRvFGw_pwrp0Nxa__BCdOF=QN4npS2ybLQ@mail.gmail.com>
Message-ID: <4E3796C2.9070002@behnel.de>

Robert Bradshaw, 02.08.2011 02:42:
> In general, if we have a distinct release branch (e.g if anything
> really interesting had gone in since the last rc), I'd say we should
> push release fixes to that and then merge into master.

+1

Stefan

From d.s.seljebotn at astro.uio.no  Tue Aug  2 08:28:25 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 02 Aug 2011 08:28:25 +0200
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <CAKGHGPRXYU-4vfcWpz5q8TitLQu-yd76Z6weKcRo_n+TWDjfhQ@mail.gmail.com>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>	<CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>	<201108020004.58249.Arfrever.FTA@gmail.com>	<CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>
	<CAKGHGPRXYU-4vfcWpz5q8TitLQu-yd76Z6weKcRo_n+TWDjfhQ@mail.gmail.com>
Message-ID: <4E379909.6020408@astro.uio.no>

On 08/02/2011 07:44 AM, Vitja Makarov wrote:
> 2011/8/2 Robert Bradshaw<robertwb at math.washington.edu>:
>> On Mon, Aug 1, 2011 at 3:04 PM, Arfrever Frehtes Taifersar Arahesis
>> <arfrever.fta at gmail.com>  wrote:
>>> 2011-07-31 09:45:15 Robert Bradshaw napisa?(a):
>>>> On Thu, Jul 21, 2011 at 4:14 PM, Robert Bradshaw
>>>> <robertwb at math.washington.edu>  wrote:
>>>>> Cython has seen an enormous amount of development since 0.14.1. If you
>>>>> are not already using the latest version from the development
>>>>> repository, we encourage you to try out the release candidate:
>>>>> http://cython.org/release/Cython-0.15rc0.tar.gz
>>>>
>>>> Thanks for all the feedback. Here's another release candidate:
>>>> http://cython.org/release/Cython-0.15rc0.tar.gz Unless something major
>>>> is found, this is likely to be the release.
>>>
>>> Cython-0.15rc1.tar.gz contains tests/imports/cimport_on_import.srctree, which was
>>> absent in Cython-0.15rc0.tar.gz and is absent on https://github.com/cython/cython.
>>> The test from this file fails:
>>
>> And this is why I should always make sdists in a clean client...
>> Thanks for catching this. A clean tarball is up at
>> http://cython.org/release/Cython-0.15rc2.tar.gz . (I also took a quick
>> look at what's been pushed since rc1, and they all look like safe
>> changes worth getting in.)
>>
>
> Maybe it's better to use "git archive"?
>

If it's only about creating a tarball, then 'git tag' (or, I assume, a 
commit hash) will suffice -- GitHub automatically tarballs for you:

https://github.com/cython/cython/tarball/0.15rc2

However I believe sdist does some more stuff (generates some files 
before tarballing or similar); although as I don't know the specifics I 
can't tell you whether we need to do it for Cython. E.g., for libraries 
written in Cython one could hook sdist to Cythonize to C files; of 
course, with Cython we bootstrap instead so that's not an issue.

Dag Sverre

From stefan_ml at behnel.de  Tue Aug  2 08:56:20 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Aug 2011 08:56:20 +0200
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <4E379909.6020408@astro.uio.no>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>	<CADiQ+QAOODwBQmr9LZtcOf1_k=QOtFftjTxDY_geHhwozFH1Yg@mail.gmail.com>	<201108020004.58249.Arfrever.FTA@gmail.com>	<CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>	<CAKGHGPRXYU-4vfcWpz5q8TitLQu-yd76Z6weKcRo_n+TWDjfhQ@mail.gmail.com>
	<4E379909.6020408@astro.uio.no>
Message-ID: <4E379F94.10604@behnel.de>

Dag Sverre Seljebotn, 02.08.2011 08:28:
> On 08/02/2011 07:44 AM, Vitja Makarov wrote:
>> Maybe it's better to use "git archive"?
>
> If it's only about creating a tarball, then 'git tag' (or, I assume, a
> commit hash) will suffice -- GitHub automatically tarballs for you:
>
> https://github.com/cython/cython/tarball/0.15rc2
>
> However I believe sdist does some more stuff (generates some files before
> tarballing or similar); although as I don't know the specifics I can't tell
> you whether we need to do it for Cython. E.g., for libraries written in
> Cython one could hook sdist to Cythonize to C files; of course, with Cython
> we bootstrap instead so that's not an issue.

It wouldn't be the first time we miss files in the MANIFEST.in of a 
(pre-)release, and yes, it prepares a couple of things for us, so 
especially for release candidates, it's a very good idea to use distutils' 
sdist - obviously from a clean working copy.

BTW, Jenkins builds sdists for us all the time, so as long as it's taken 
from a branch that it builds anyway, you can just grab it from there, e.g.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/lastSuccessfulBuild/

We also still have the old "cython-release" build jobs that are now picking 
up the new release branch, it seems:

https://sage.math.washington.edu:8091/hudson/view/release/

I'll shortly reconfigure them to build a clean sdist as well (using "git 
clean"), then we can use that build job as a source for archives. Note that 
Jenkins has a button to "keep this build", so we can keep all the build 
logs alive for a release.

Stefan

From arfrever.fta at gmail.com  Tue Aug  2 16:39:58 2011
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Tue, 2 Aug 2011 16:39:58 +0200
Subject: [Cython] Cython 0.15 release candidate
In-Reply-To: <CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>
References: <CADiQ+QAN=aC3p=oV-YPGvkOaK4oSU-ypNWyZQgJnZAukJDx0gg@mail.gmail.com>
	<201108020004.58249.Arfrever.FTA@gmail.com>
	<CADiQ+QBt5kECg3uT3yUh4TcAu3qWELsKi-JWfs91nyF87LWBYg@mail.gmail.com>
Message-ID: <201108021640.00156.Arfrever.FTA@gmail.com>

2011-08-02 00:58:56 Robert Bradshaw napisa?(a):
> A clean tarball is up at
> http://cython.org/release/Cython-0.15rc2.tar.gz .

All tests now pass.

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110802/635a3f4a/attachment.pgp>

From vitja.makarov at gmail.com  Tue Aug  2 20:41:17 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Aug 2011 22:41:17 +0400
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <4E32794F.5040007@behnel.de>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>
	<4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org>
	<4DF5C85D.3010306@behnel.de>
	<878vt54w22.fsf@inspiron.ap.columbia.edu>
	<4E323FCC.8010002@behnel.de>
	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>
	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>
	<4E325853.5070207@behnel.de>
	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>
	<4E326DFF.2090307@behnel.de>
	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>
	<4E32745B.7040501@behnel.de>
	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>
	<4E32794F.5040007@behnel.de>
Message-ID: <CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>

2011/7/29 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 29.07.2011 10:55:
>>
>> 2011/7/29 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 29.07.2011 10:44:
>>>>
>>>> 2011/7/29 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 29.07.2011 10:08:
>>>>>>
>>>>>> this issue isn't critical and even isn't a bug at all.
>>>>>
>>>>> Agreed. It's nothing that needs to be done for 0.15. I just thought you
>>>>> might be interested. :D
>>>>>
>>>>
>>>> Yeah, I tried to do this once but I've found some problems with buffer
>>>> variables.
>>>>
>>>> What to do about local variables:
>>>>
>>>> def foo():
>>>> ? ? a = 1
>>>>
>>>> 'a' is unused here
>>>
>>> That's up to the user to fix. However, there may be restrictions
>>> regarding
>>> the signature (inheritance etc.) that the users cannot control, so unused
>>> *parameters* must not produce warnings.
>>>
>>
>> Sure. Because of that there is separate warn.unused_args option :)
>
> With the caveat that gcc 4.6 produces a warning with -Wall for them, because
> it cannot know that they originally were parameters in the Cython code.
>


I tried to implement this for c[p]def functions:

For required args I've added CYTHON_UNUSED qualifier and removed
unused optionals.

-- 
vitja.

From vitja.makarov at gmail.com  Tue Aug  2 21:38:28 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Aug 2011 23:38:28 +0400
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>
	<4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org>
	<4DF5C85D.3010306@behnel.de>
	<878vt54w22.fsf@inspiron.ap.columbia.edu>
	<4E323FCC.8010002@behnel.de>
	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>
	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>
	<4E325853.5070207@behnel.de>
	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>
	<4E326DFF.2090307@behnel.de>
	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>
	<4E32745B.7040501@behnel.de>
	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>
	<4E32794F.5040007@behnel.de>
	<CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>
Message-ID: <CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>

2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/7/29 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 29.07.2011 10:55:
>>>
>>> 2011/7/29 Stefan Behnel<stefan_ml at behnel.de>:
>>>>
>>>> Vitja Makarov, 29.07.2011 10:44:
>>>>>
>>>>> 2011/7/29 Stefan Behnel:
>>>>>>
>>>>>> Vitja Makarov, 29.07.2011 10:08:
>>>>>>>
>>>>>>> this issue isn't critical and even isn't a bug at all.
>>>>>>
>>>>>> Agreed. It's nothing that needs to be done for 0.15. I just thought you
>>>>>> might be interested. :D
>>>>>>
>>>>>
>>>>> Yeah, I tried to do this once but I've found some problems with buffer
>>>>> variables.
>>>>>
>>>>> What to do about local variables:
>>>>>
>>>>> def foo():
>>>>> ? ? a = 1
>>>>>
>>>>> 'a' is unused here
>>>>
>>>> That's up to the user to fix. However, there may be restrictions
>>>> regarding
>>>> the signature (inheritance etc.) that the users cannot control, so unused
>>>> *parameters* must not produce warnings.
>>>>
>>>
>>> Sure. Because of that there is separate warn.unused_args option :)
>>
>> With the caveat that gcc 4.6 produces a warning with -Wall for them, because
>> it cannot know that they originally were parameters in the Cython code.
>>
>
>
> I tried to implement this for c[p]def functions:
>
> For required args I've added CYTHON_UNUSED qualifier and removed
> unused optionals.
>


Stefan, do you know why skip_dispatch argument is used for
module-level cpdef function?

There is warning about that too.

-- 
vitja.

From vitja.makarov at gmail.com  Tue Aug  2 21:50:54 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Aug 2011 23:50:54 +0400
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>
	<4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org>
	<4DF5C85D.3010306@behnel.de>
	<878vt54w22.fsf@inspiron.ap.columbia.edu>
	<4E323FCC.8010002@behnel.de>
	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>
	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>
	<4E325853.5070207@behnel.de>
	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>
	<4E326DFF.2090307@behnel.de>
	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>
	<4E32745B.7040501@behnel.de>
	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>
	<4E32794F.5040007@behnel.de>
	<CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>
	<CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>
Message-ID: <CAKGHGPRdk+su2xh=xXXwC8tRXy8CM85-ZRLU59cq5QFV3LSZeg@mail.gmail.com>

2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/7/29 Stefan Behnel <stefan_ml at behnel.de>:
>>> Vitja Makarov, 29.07.2011 10:55:
>>>>
>>>> 2011/7/29 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>
>>>>> Vitja Makarov, 29.07.2011 10:44:
>>>>>>
>>>>>> 2011/7/29 Stefan Behnel:
>>>>>>>
>>>>>>> Vitja Makarov, 29.07.2011 10:08:
>>>>>>>>
>>>>>>>> this issue isn't critical and even isn't a bug at all.
>>>>>>>
>>>>>>> Agreed. It's nothing that needs to be done for 0.15. I just thought you
>>>>>>> might be interested. :D
>>>>>>>
>>>>>>
>>>>>> Yeah, I tried to do this once but I've found some problems with buffer
>>>>>> variables.
>>>>>>
>>>>>> What to do about local variables:
>>>>>>
>>>>>> def foo():
>>>>>> ? ? a = 1
>>>>>>
>>>>>> 'a' is unused here
>>>>>
>>>>> That's up to the user to fix. However, there may be restrictions
>>>>> regarding
>>>>> the signature (inheritance etc.) that the users cannot control, so unused
>>>>> *parameters* must not produce warnings.
>>>>>
>>>>
>>>> Sure. Because of that there is separate warn.unused_args option :)
>>>
>>> With the caveat that gcc 4.6 produces a warning with -Wall for them, because
>>> it cannot know that they originally were parameters in the Cython code.
>>>
>>
>>
>> I tried to implement this for c[p]def functions:
>>
>> For required args I've added CYTHON_UNUSED qualifier and removed
>> unused optionals.
>>
>
>
> Stefan, do you know why skip_dispatch argument is used for
> module-level cpdef function?
>
> There is warning about that too.
>

I tried to fix the issue here:
https://github.com/vitek/cython/commit/dc87be3f546ffe609fdce9514d939c147c99cf35

And here is my branch for unused_arg:

https://github.com/vitek/cython/commits/_unused_args

-- 
vitja.

From faltet at pytables.org  Wed Aug  3 14:18:16 2011
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 3 Aug 2011 14:18:16 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
Message-ID: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>

Hi,

I'm trying to take advantage of the exciting new parallelizing
capabilities recently introduced in forthcoming 0.15 version, but I'm
having a small difficulty.  When I try to compile a small demo routing
(attached), I'm getting this error:

$ cython -a mandel.pyx

Error compiling Cython file:
------------------------------------------------------------
...
    for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
        x = pix % width
        y = pix // width
        cr = begin_r + (x * span_r / (width + 1.0))
        ci = begin_i + (y * span_i / (height + 1.0))
        n = MandelbrotCalculate(cr, ci, maxiter)
                              ^
------------------------------------------------------------

mandel.pyx:50:31: Calling gil-requiring function not allowed without gil

While trying to figure out why MandelbrotCalculate does require GIL, I
replaced the "for  prange", by a trivial "for range", and the
annotated HTML source reveals that the only line in (light) yellow in
this function is

    return n

which is translated as:

  /* "mandel.pyx":22
 *         ti = 2*zr*zi + ci
 *         zr, zi = tr, ti
 *     return n             # <<<<<<<<<<<<<<
 *
 * @cython.boundscheck(False)
 */
  __pyx_r = __pyx_v_n;
  goto __pyx_L0;

  __pyx_r = 0;
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

My guess is that this __Pyx_RefNannyFinishContext() call is preventing
to call the routing from the parallel loop.  Is that a bug, a
limitation of current implementation or it is just that I'm missing
something?

Thanks,

-- 
Francesc Alted
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mandel.pyx
Type: application/octet-stream
Size: 1631 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110803/cb6d697d/attachment.obj>

From markflorisson88 at gmail.com  Wed Aug  3 14:23:07 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Aug 2011 14:23:07 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
In-Reply-To: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
References: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
Message-ID: <CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>

On 3 August 2011 14:18, Francesc Alted <faltet at pytables.org> wrote:
> Hi,
>
> I'm trying to take advantage of the exciting new parallelizing
> capabilities recently introduced in forthcoming 0.15 version, but I'm
> having a small difficulty. ?When I try to compile a small demo routing
> (attached), I'm getting this error:
>
> $ cython -a mandel.pyx
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> ? ?for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
> ? ? ? ?x = pix % width
> ? ? ? ?y = pix // width
> ? ? ? ?cr = begin_r + (x * span_r / (width + 1.0))
> ? ? ? ?ci = begin_i + (y * span_i / (height + 1.0))
> ? ? ? ?n = MandelbrotCalculate(cr, ci, maxiter)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
> ------------------------------------------------------------
>
> mandel.pyx:50:31: Calling gil-requiring function not allowed without gil
>
> While trying to figure out why MandelbrotCalculate does require GIL, I
> replaced the "for ?prange", by a trivial "for range", and the
> annotated HTML source reveals that the only line in (light) yellow in
> this function is
>
> ? ?return n
>
> which is translated as:
>
> ?/* "mandel.pyx":22
> ?* ? ? ? ? ti = 2*zr*zi + ci
> ?* ? ? ? ? zr, zi = tr, ti
> ?* ? ? return n ? ? ? ? ? ? # <<<<<<<<<<<<<<
> ?*
> ?* @cython.boundscheck(False)
> ?*/
> ?__pyx_r = __pyx_v_n;
> ?goto __pyx_L0;
>
> ?__pyx_r = 0;
> ?__pyx_L0:;
> ?__Pyx_RefNannyFinishContext();
> ?return __pyx_r;
> }
>
> My guess is that this __Pyx_RefNannyFinishContext() call is preventing
> to call the routing from the parallel loop. ?Is that a bug, a
> limitation of current implementation or it is just that I'm missing
> something?
>
> Thanks,
>
> --
> Francesc Alted
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

Hey Francesc!

The problem is that you didn't declare MandelbrotCalculate 'nogil'.
You have to write

    cdef long MandelbrotCalculate(double cr, double ci, long maxiter) nogil:
        ...

That's all, and it will compile :)

Cheers,

Mark

From stefan_ml at behnel.de  Wed Aug  3 14:27:02 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Aug 2011 14:27:02 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
In-Reply-To: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
References: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
Message-ID: <4E393E96.5000603@behnel.de>

Francesc Alted, 03.08.2011 14:18:
> I'm trying to take advantage of the exciting new parallelizing
> capabilities recently introduced in forthcoming 0.15 version, but I'm
> having a small difficulty.  When I try to compile a small demo routing
> (attached), I'm getting this error:
>
> $ cython -a mandel.pyx
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>      for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
>          x = pix % width
>          y = pix // width
>          cr = begin_r + (x * span_r / (width + 1.0))
>          ci = begin_i + (y * span_i / (height + 1.0))
>          n = MandelbrotCalculate(cr, ci, maxiter)
>                                ^
> ------------------------------------------------------------
>
> mandel.pyx:50:31: Calling gil-requiring function not allowed without gil
>
> While trying to figure out why MandelbrotCalculate does require GIL

That's because it does not declare that it does not require the GIL. Use 
the "nogil" annotation at the end of the function signature.

Stefan

From faltet at pytables.org  Wed Aug  3 14:33:00 2011
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 3 Aug 2011 14:33:00 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
In-Reply-To: <CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>
References: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
	<CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>
Message-ID: <CAFrp1vrrr1CmPcbpeSQfMp89+t5MOx9WjrYhXnd74tANR=RyWg@mail.gmail.com>

Sure.  And I'm seeing a good speed-up on my 2-core machine indeed:

Without parallel loop:

real    0m0.923s
user    0m0.875s
sys     0m0.045s

With parallel loop:

real    0m0.544s
user    0m0.876s
sys     0m0.045s

Which is pretty awesome, given the simplicity of Cython parallel
implementation :)

Thanks a lot Mark!


2011/8/3, mark florisson <markflorisson88 at gmail.com>:
> On 3 August 2011 14:18, Francesc Alted <faltet at pytables.org> wrote:
>> Hi,
>>
>> I'm trying to take advantage of the exciting new parallelizing
>> capabilities recently introduced in forthcoming 0.15 version, but I'm
>> having a small difficulty. ?When I try to compile a small demo routing
>> (attached), I'm getting this error:
>>
>> $ cython -a mandel.pyx
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ?for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
>> ? ? ? ?x = pix % width
>> ? ? ? ?y = pix // width
>> ? ? ? ?cr = begin_r + (x * span_r / (width + 1.0))
>> ? ? ? ?ci = begin_i + (y * span_i / (height + 1.0))
>> ? ? ? ?n = MandelbrotCalculate(cr, ci, maxiter)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>> ------------------------------------------------------------
>>
>> mandel.pyx:50:31: Calling gil-requiring function not allowed without gil
>>
>> While trying to figure out why MandelbrotCalculate does require GIL, I
>> replaced the "for ?prange", by a trivial "for range", and the
>> annotated HTML source reveals that the only line in (light) yellow in
>> this function is
>>
>> ? ?return n
>>
>> which is translated as:
>>
>> ?/* "mandel.pyx":22
>> ?* ? ? ? ? ti = 2*zr*zi + ci
>> ?* ? ? ? ? zr, zi = tr, ti
>> ?* ? ? return n ? ? ? ? ? ? # <<<<<<<<<<<<<<
>> ?*
>> ?* @cython.boundscheck(False)
>> ?*/
>> ?__pyx_r = __pyx_v_n;
>> ?goto __pyx_L0;
>>
>> ?__pyx_r = 0;
>> ?__pyx_L0:;
>> ?__Pyx_RefNannyFinishContext();
>> ?return __pyx_r;
>> }
>>
>> My guess is that this __Pyx_RefNannyFinishContext() call is preventing
>> to call the routing from the parallel loop. ?Is that a bug, a
>> limitation of current implementation or it is just that I'm missing
>> something?
>>
>> Thanks,
>>
>> --
>> Francesc Alted
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
> Hey Francesc!
>
> The problem is that you didn't declare MandelbrotCalculate 'nogil'.
> You have to write
>
>     cdef long MandelbrotCalculate(double cr, double ci, long maxiter) nogil:
>         ...
>
> That's all, and it will compile :)
>
> Cheers,
>
> Mark
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>


-- 
Francesc Alted

From faltet at pytables.org  Wed Aug  3 14:33:43 2011
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 3 Aug 2011 14:33:43 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
In-Reply-To: <CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>
References: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
	<CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>
Message-ID: <CAFrp1vrfv4pfkPgC5+9Qsx=g6uj1RuJ4+OhXiZ9j7mbXNyQLrA@mail.gmail.com>

Hey Mark,

Sure.  And I'm seeing a good speed-up on my 2-core machine indeed:

Without parallel loop:

real    0m0.923s
user    0m0.875s
sys     0m0.045s

With parallel loop:

real    0m0.544s
user    0m0.876s
sys     0m0.045s

Which is pretty awesome, given the simplicity of the Cython parallel
implementation :)

Thanks a lot!


2011/8/3, mark florisson <markflorisson88 at gmail.com>:
> On 3 August 2011 14:18, Francesc Alted <faltet at pytables.org> wrote:
>> Hi,
>>
>> I'm trying to take advantage of the exciting new parallelizing
>> capabilities recently introduced in forthcoming 0.15 version, but I'm
>> having a small difficulty. ?When I try to compile a small demo routing
>> (attached), I'm getting this error:
>>
>> $ cython -a mandel.pyx
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ?for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
>> ? ? ? ?x = pix % width
>> ? ? ? ?y = pix // width
>> ? ? ? ?cr = begin_r + (x * span_r / (width + 1.0))
>> ? ? ? ?ci = begin_i + (y * span_i / (height + 1.0))
>> ? ? ? ?n = MandelbrotCalculate(cr, ci, maxiter)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>> ------------------------------------------------------------
>>
>> mandel.pyx:50:31: Calling gil-requiring function not allowed without gil
>>
>> While trying to figure out why MandelbrotCalculate does require GIL, I
>> replaced the "for ?prange", by a trivial "for range", and the
>> annotated HTML source reveals that the only line in (light) yellow in
>> this function is
>>
>> ? ?return n
>>
>> which is translated as:
>>
>> ?/* "mandel.pyx":22
>> ?* ? ? ? ? ti = 2*zr*zi + ci
>> ?* ? ? ? ? zr, zi = tr, ti
>> ?* ? ? return n ? ? ? ? ? ? # <<<<<<<<<<<<<<
>> ?*
>> ?* @cython.boundscheck(False)
>> ?*/
>> ?__pyx_r = __pyx_v_n;
>> ?goto __pyx_L0;
>>
>> ?__pyx_r = 0;
>> ?__pyx_L0:;
>> ?__Pyx_RefNannyFinishContext();
>> ?return __pyx_r;
>> }
>>
>> My guess is that this __Pyx_RefNannyFinishContext() call is preventing
>> to call the routing from the parallel loop. ?Is that a bug, a
>> limitation of current implementation or it is just that I'm missing
>> something?
>>
>> Thanks,
>>
>> --
>> Francesc Alted
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
> Hey Francesc!
>
> The problem is that you didn't declare MandelbrotCalculate 'nogil'.
> You have to write
>
>     cdef long MandelbrotCalculate(double cr, double ci, long maxiter) nogil:
>         ...
>
> That's all, and it will compile :)
>
> Cheers,
>
> Mark
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>


-- 
Francesc Alted

From markflorisson88 at gmail.com  Wed Aug  3 14:57:33 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Aug 2011 14:57:33 +0200
Subject: [Cython] Cython 0.15rc2 and parallelization issue
In-Reply-To: <CAFrp1vrrr1CmPcbpeSQfMp89+t5MOx9WjrYhXnd74tANR=RyWg@mail.gmail.com>
References: <CAFrp1vqJEaCqvSD1xO5qZxh26C5-31uX2OTgYyW2UWmkWRviQw@mail.gmail.com>
	<CANg26EUUmWeZ22Hjme=jNLtq+pH6BSyL4pE0BzcpPY3esJbRxQ@mail.gmail.com>
	<CAFrp1vrrr1CmPcbpeSQfMp89+t5MOx9WjrYhXnd74tANR=RyWg@mail.gmail.com>
Message-ID: <CANg26EVaw4XgN+UyVR1xV3bo66UXoLj=ZGibG4HPuxtgoTKOTQ@mail.gmail.com>

On 3 August 2011 14:33, Francesc Alted <faltet at pytables.org> wrote:
> Sure. ?And I'm seeing a good speed-up on my 2-core machine indeed:
>
> Without parallel loop:
>
> real ? ?0m0.923s
> user ? ?0m0.875s
> sys ? ? 0m0.045s
>
> With parallel loop:
>
> real ? ?0m0.544s
> user ? ?0m0.876s
> sys ? ? 0m0.045s
>
> Which is pretty awesome, given the simplicity of Cython parallel
> implementation :)
>
> Thanks a lot Mark!
>

Glad you like it, nice speedup!

> 2011/8/3, mark florisson <markflorisson88 at gmail.com>:
>> On 3 August 2011 14:18, Francesc Alted <faltet at pytables.org> wrote:
>>> Hi,
>>>
>>> I'm trying to take advantage of the exciting new parallelizing
>>> capabilities recently introduced in forthcoming 0.15 version, but I'm
>>> having a small difficulty. ?When I try to compile a small demo routing
>>> (attached), I'm getting this error:
>>>
>>> $ cython -a mandel.pyx
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>> ? ?for pix in prange(num_pixels, nogil=True, schedule="dynamic"):
>>> ? ? ? ?x = pix % width
>>> ? ? ? ?y = pix // width
>>> ? ? ? ?cr = begin_r + (x * span_r / (width + 1.0))
>>> ? ? ? ?ci = begin_i + (y * span_i / (height + 1.0))
>>> ? ? ? ?n = MandelbrotCalculate(cr, ci, maxiter)
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>>> ------------------------------------------------------------
>>>
>>> mandel.pyx:50:31: Calling gil-requiring function not allowed without gil
>>>
>>> While trying to figure out why MandelbrotCalculate does require GIL, I
>>> replaced the "for ?prange", by a trivial "for range", and the
>>> annotated HTML source reveals that the only line in (light) yellow in
>>> this function is
>>>
>>> ? ?return n
>>>
>>> which is translated as:
>>>
>>> ?/* "mandel.pyx":22
>>> ?* ? ? ? ? ti = 2*zr*zi + ci
>>> ?* ? ? ? ? zr, zi = tr, ti
>>> ?* ? ? return n ? ? ? ? ? ? # <<<<<<<<<<<<<<
>>> ?*
>>> ?* @cython.boundscheck(False)
>>> ?*/
>>> ?__pyx_r = __pyx_v_n;
>>> ?goto __pyx_L0;
>>>
>>> ?__pyx_r = 0;
>>> ?__pyx_L0:;
>>> ?__Pyx_RefNannyFinishContext();
>>> ?return __pyx_r;
>>> }
>>>
>>> My guess is that this __Pyx_RefNannyFinishContext() call is preventing
>>> to call the routing from the parallel loop. ?Is that a bug, a
>>> limitation of current implementation or it is just that I'm missing
>>> something?
>>>
>>> Thanks,
>>>
>>> --
>>> Francesc Alted
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>> Hey Francesc!
>>
>> The problem is that you didn't declare MandelbrotCalculate 'nogil'.
>> You have to write
>>
>> ? ? cdef long MandelbrotCalculate(double cr, double ci, long maxiter) nogil:
>> ? ? ? ? ...
>>
>> That's all, and it will compile :)
>>
>> Cheers,
>>
>> Mark
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
>
> --
> Francesc Alted
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Wed Aug  3 16:28:32 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Aug 2011 16:28:32 +0200
Subject: [Cython] object to struct conversion
Message-ID: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>

Is there any specific reason objects cannot coerce to structs (from
e.g. dicts?). It would be convenient for memoryviews, then you could
assign dicts (or any mapping) to items in the memoryview from Python
space. You could also have structs as argument to def functions etc.
Any objection to this addition? The implementation is pretty
straightforward.

From stefan_ml at behnel.de  Wed Aug  3 18:20:55 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Aug 2011 18:20:55 +0200
Subject: [Cython] object to struct conversion
In-Reply-To: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
Message-ID: <4E397567.1060708@behnel.de>

mark florisson, 03.08.2011 16:28:
> Is there any specific reason objects cannot coerce to structs (from
> e.g. dicts?). It would be convenient for memoryviews, then you could
> assign dicts (or any mapping) to items in the memoryview from Python
> space. You could also have structs as argument to def functions etc.
> Any objection to this addition? The implementation is pretty
> straightforward.

Robert implemented these things a while ago. I think I recall that there 
were unsupported use cases, but I'm not sure what exactly is missing here.

Stefan

From robertwb at math.washington.edu  Wed Aug  3 19:36:14 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Aug 2011 10:36:14 -0700
Subject: [Cython] object to struct conversion
In-Reply-To: <4E397567.1060708@behnel.de>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
	<4E397567.1060708@behnel.de>
Message-ID: <CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>

On Wed, Aug 3, 2011 at 9:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 03.08.2011 16:28:
>>
>> Is there any specific reason objects cannot coerce to structs (from
>> e.g. dicts?). It would be convenient for memoryviews, then you could
>> assign dicts (or any mapping) to items in the memoryview from Python
>> space. You could also have structs as argument to def functions etc.
>> Any objection to this addition? The implementation is pretty
>> straightforward.
>
> Robert implemented these things a while ago. I think I recall that there
> were unsupported use cases, but I'm not sure what exactly is missing here.

It just simply wasn't implemented yet.

- Robert

From vitja.makarov at gmail.com  Wed Aug  3 21:07:46 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Aug 2011 23:07:46 +0400
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <CAKGHGPRdk+su2xh=xXXwC8tRXy8CM85-ZRLU59cq5QFV3LSZeg@mail.gmail.com>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>
	<4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org>
	<4DF5C85D.3010306@behnel.de>
	<878vt54w22.fsf@inspiron.ap.columbia.edu>
	<4E323FCC.8010002@behnel.de>
	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>
	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>
	<4E325853.5070207@behnel.de>
	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>
	<4E326DFF.2090307@behnel.de>
	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>
	<4E32745B.7040501@behnel.de>
	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>
	<4E32794F.5040007@behnel.de>
	<CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>
	<CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>
	<CAKGHGPRdk+su2xh=xXXwC8tRXy8CM85-ZRLU59cq5QFV3LSZeg@mail.gmail.com>
Message-ID: <CAKGHGPQg1LYSnW57bXsXwVCsGp8tOK8ANpsd2R2g_VMDyefJpg@mail.gmail.com>

2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/2 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/7/29 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Vitja Makarov, 29.07.2011 10:55:
>>>>>
>>>>> 2011/7/29 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>
>>>>>> Vitja Makarov, 29.07.2011 10:44:
>>>>>>>
>>>>>>> 2011/7/29 Stefan Behnel:
>>>>>>>>
>>>>>>>> Vitja Makarov, 29.07.2011 10:08:
>>>>>>>>>
>>>>>>>>> this issue isn't critical and even isn't a bug at all.
>>>>>>>>
>>>>>>>> Agreed. It's nothing that needs to be done for 0.15. I just thought you
>>>>>>>> might be interested. :D
>>>>>>>>
>>>>>>>
>>>>>>> Yeah, I tried to do this once but I've found some problems with buffer
>>>>>>> variables.
>>>>>>>
>>>>>>> What to do about local variables:
>>>>>>>
>>>>>>> def foo():
>>>>>>> ? ? a = 1
>>>>>>>
>>>>>>> 'a' is unused here
>>>>>>
>>>>>> That's up to the user to fix. However, there may be restrictions
>>>>>> regarding
>>>>>> the signature (inheritance etc.) that the users cannot control, so unused
>>>>>> *parameters* must not produce warnings.
>>>>>>
>>>>>
>>>>> Sure. Because of that there is separate warn.unused_args option :)
>>>>
>>>> With the caveat that gcc 4.6 produces a warning with -Wall for them, because
>>>> it cannot know that they originally were parameters in the Cython code.
>>>>
>>>
>>>
>>> I tried to implement this for c[p]def functions:
>>>
>>> For required args I've added CYTHON_UNUSED qualifier and removed
>>> unused optionals.
>>>
>>
>>
>> Stefan, do you know why skip_dispatch argument is used for
>> module-level cpdef function?
>>
>> There is warning about that too.
>>
>
> I tried to fix the issue here:
> https://github.com/vitek/cython/commit/dc87be3f546ffe609fdce9514d939c147c99cf35
>
> And here is my branch for unused_arg:
>
> https://github.com/vitek/cython/commits/_unused_args
>

I've reverted skip_dispatch argument removal, that doesn't work this
way and is more general issue.
Now I mark all unused local entries and args with CYTHON_UNUSED.

That actually isn't an optimization just some warnings removal.

I think real optimization should be done when we start with DefNode refactoring.

If you like that I'm gonna start pull request.

-- 
vitja.

From stefano.k.sanfilippo at gmail.com  Wed Aug  3 22:53:36 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Wed, 3 Aug 2011 22:53:36 +0200
Subject: [Cython] Willing to contribute
Message-ID: <201108032253.36332@a>

Hi everybody,

I've been a enthusiast Python programmer for 3 years now, and I've hailed 
Cython project with great interest. Now, I'm willing to contribute. I've would 
say I'm an quite an expert Python programmer and I've some-year-long 
experience with C, modern C++ and its cousin Java. Sidenote, I'm a student at 
Politecnico di Milano, Italy.

Everything started with trying to speed up a Mandelbrot generator, which 
heavily relies on complex numbers. Eventually I've found in "Mini Projects (or 
something similar)" wiki that C99 `complex' is still to be implemented. So, I 
decide to make a step further and try to hop into development.

I've would be grateful if a core developer could point me in the right 
direction concerning coding style, repositories, deadlines (I'm still a 
student, so I may have 'offline' days this winter) and other issues. I may 
start with something easy (`complex' implementation, or command line API 
restructuration), and I'd like someone from the team to discuss what I'll be 
implementing (ok, just for the first days), just to see if  "I'm doing it 
right" (TM) and, mostly, if I'm the right man or I'm just making you loose 
your time.

You can see an example of what I did in the past here: 
http://code.google.com/p/libsmth/ - for instance (still a WIP).

I really appreciate your work :)

--SKS

From markflorisson88 at gmail.com  Wed Aug  3 23:09:19 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Aug 2011 23:09:19 +0200
Subject: [Cython] object to struct conversion
In-Reply-To: <CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
	<4E397567.1060708@behnel.de>
	<CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>
Message-ID: <CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>

On 3 August 2011 19:36, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Aug 3, 2011 at 9:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 03.08.2011 16:28:
>>>
>>> Is there any specific reason objects cannot coerce to structs (from
>>> e.g. dicts?). It would be convenient for memoryviews, then you could
>>> assign dicts (or any mapping) to items in the memoryview from Python
>>> space. You could also have structs as argument to def functions etc.
>>> Any objection to this addition? The implementation is pretty
>>> straightforward.
>>
>> Robert implemented these things a while ago. I think I recall that there
>> were unsupported use cases, but I'm not sure what exactly is missing here.
>
> It just simply wasn't implemented yet.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Ok, thanks guys, it's implemented in _memview:
https://github.com/markflorisson88/cython/commit/6a2560b5b151a788ed11671663c833b6de6daccd#L4R1

BTW, my hudson sdist keeps failing with

writing manifest file 'MANIFEST'
making hard links in Cython-0.14.1+...
hard linking COPYING.txt -> Cython-0.14.1+
error: File exists

any idea what that is all about? Are these hudson scripts available for perusal?

From dalcinl at gmail.com  Thu Aug  4 01:32:46 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 3 Aug 2011 20:32:46 -0300
Subject: [Cython] Willing to contribute
In-Reply-To: <201108032253.36332@a>
References: <201108032253.36332@a>
Message-ID: <CAEcYPwB_GTjTSJAnh7XMaV=ie1BcenaPPSwkZMNEevHKpAEO=w@mail.gmail.com>

On 3 August 2011 17:53, Stefano <stefano.k.sanfilippo at gmail.com> wrote:
>
> Everything started with trying to speed up a Mandelbrot generator, which
> heavily relies on complex numbers. Eventually I've found in "Mini Projects (or
> something similar)" wiki that C99 `complex' is still to be implemented. So, I
> decide to make a step further and try to hop into development.
>

That comment is outdated. Cython do support complex numbers in many
different flavors (native C99 complex, C++ std::complex, raw C structs
for pre-C99 compilers)


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Thu Aug  4 03:27:56 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Aug 2011 18:27:56 -0700
Subject: [Cython] Willing to contribute
In-Reply-To: <201108032253.36332@a>
References: <201108032253.36332@a>
Message-ID: <CADiQ+QDKt4SwHYL5TgHT0ZrccpputO7vgCMB3yoqyEnxtSOu1Q@mail.gmail.com>

On Wed, Aug 3, 2011 at 1:53 PM, Stefano <stefano.k.sanfilippo at gmail.com> wrote:
> Hi everybody,
>
> I've been a enthusiast Python programmer for 3 years now, and I've hailed
> Cython project with great interest. Now, I'm willing to contribute. I've would
> say I'm an quite an expert Python programmer and I've some-year-long
> experience with C, modern C++ and its cousin Java. Sidenote, I'm a student at
> Politecnico di Milano, Italy.
>
> Everything started with trying to speed up a Mandelbrot generator, which
> heavily relies on complex numbers. Eventually I've found in "Mini Projects (or
> something similar)" wiki that C99 `complex' is still to be implemented. So, I
> decide to make a step further and try to hop into development.

As Lisandro mentioned, complex numbers have been implemented, but
there's still plenty to do :).

> I've would be grateful if a core developer could point me in the right
> direction concerning coding style, repositories, deadlines (I'm still a
> student, so I may have 'offline' days this winter) and other issues.

Our process is pretty simple. Our code is up at
https://github.com/cython/ and we prefer contributions via pull
requests; the documentation is in the same repository. As for coding
guildelines, we try to follow http://www.python.org/dev/peps/pep-0008/
and support Python 2.4+. Everyone here is working on Cython on the
side (with, I suppose, the exception of Google Summer of Code
contributors) so development is done in a pretty informal
as-you-have-the-time manner. For substantially new features, a Cython
Enhancement Proposal http://wiki.cython.org/enhancements is usually in
order.

> I may
> start with something easy (`complex' implementation, or command line API
> restructuration), and I'd like someone from the team to discuss what I'll be
> implementing (ok, just for the first days), just to see if ?"I'm doing it
> right" (TM) and, mostly, if I'm the right man or I'm just making you loose
> your time.

We'll try to answer your questions, but the more specific the better.
You might also want to play around with "cython -a" and the options in
Cython/Compiler/DebugFlags.py to find your way around the codebase.
There is also some good documentation on the wiki, but as has been
pointed out, there's some really old stuff there as well. (There's a
lot of non-coding work to be done too :-).

Everyone scratches their own itch, but all of us here would agree that
this is a pretty fun one.

> You can see an example of what I did in the past here:
> http://code.google.com/p/libsmth/ - for instance (still a WIP).
>
> I really appreciate your work :)

Thanks.

- Robert

From stefan_ml at behnel.de  Thu Aug  4 08:41:21 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Aug 2011 08:41:21 +0200
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <CAKGHGPQg1LYSnW57bXsXwVCsGp8tOK8ANpsd2R2g_VMDyefJpg@mail.gmail.com>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>	<4DF51F03.8010806@behnel.de>
	<87d3iihc5p.fsf@vostro.rath.org>	<4DF5C85D.3010306@behnel.de>	<878vt54w22.fsf@inspiron.ap.columbia.edu>	<4E323FCC.8010002@behnel.de>	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>	<4E325853.5070207@behnel.de>	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>	<4E326DFF.2090307@behnel.de>	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>	<4E32745B.7040501@behnel.de>	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>	<4E32794F.5040007@behnel.de>	<CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>	<CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>	<CAKGHGPRdk+su2xh=xXXwC8tRXy8CM85-ZRLU59cq5QFV3LSZeg@mail.gmail.com>
	<CAKGHGPQg1LYSnW57bXsXwVCsGp8tOK8ANpsd2R2g_VMDyefJpg@mail.gmail.com>
Message-ID: <4E3A3F11.5040901@behnel.de>

Vitja Makarov, 03.08.2011 21:07:
>>> Stefan, do you know why skip_dispatch argument is used for
>>> module-level cpdef function?
>>>
>>> There is warning about that too.

It seems you already found a way to handle it.


>> And here is my branch for unused_arg:
>>
>> https://github.com/vitek/cython/commits/_unused_args
>
> I've reverted skip_dispatch argument removal, that doesn't work this
> way and is more general issue.
> Now I mark all unused local entries and args with CYTHON_UNUSED.

Cool.


> That actually isn't an optimization just some warnings removal.
>
> I think real optimization should be done when we start with DefNode
> refactoring.

Absolutely.


> If you like that I'm gonna start pull request.

It looks good to me, so please do.

Stefan

From vitja.makarov at gmail.com  Thu Aug  4 09:43:33 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Aug 2011 11:43:33 +0400
Subject: [Cython] GCC 4.6 unused-but-set-variable warnings
In-Reply-To: <4E3A3F11.5040901@behnel.de>
References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu>
	<4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org>
	<4DF5C85D.3010306@behnel.de>
	<878vt54w22.fsf@inspiron.ap.columbia.edu>
	<4E323FCC.8010002@behnel.de>
	<CAKGHGPRu2nxEYbRYAi3WH+REd1ghk-3XB0iRkYrBdJ2tAqmurg@mail.gmail.com>
	<CAKGHGPT=yuT_Bm84b-hT+oApQtUP3x8yxSf-5z38uf8jAPiJEg@mail.gmail.com>
	<4E325853.5070207@behnel.de>
	<CAKGHGPSWhVOa8ARt+fzOHOdoqSFc9Yr8j31Sc-6fj96upUa36A@mail.gmail.com>
	<4E326DFF.2090307@behnel.de>
	<CAKGHGPQMKPywPqOB6j8FYk-z9YxkQXmBXM1sYqpevM+nXf9JnQ@mail.gmail.com>
	<4E32745B.7040501@behnel.de>
	<CAKGHGPSbEuOReFkev=+iqdqJTY4vjH-YBjamCWyrjPp1yw7T-Q@mail.gmail.com>
	<4E32794F.5040007@behnel.de>
	<CAKGHGPTBeR8-V3gEBfhZwEj29y4PdmnGLuKoVEfYsC=pG5onkQ@mail.gmail.com>
	<CAKGHGPRxFoAuTr1w0g-46UrY6qHy_ZrwPU3HNx9ggowoLecSog@mail.gmail.com>
	<CAKGHGPRdk+su2xh=xXXwC8tRXy8CM85-ZRLU59cq5QFV3LSZeg@mail.gmail.com>
	<CAKGHGPQg1LYSnW57bXsXwVCsGp8tOK8ANpsd2R2g_VMDyefJpg@mail.gmail.com>
	<4E3A3F11.5040901@behnel.de>
Message-ID: <CAKGHGPTBV0VvuZJ6XOc6p7JXLDMyXCfKaCUnNSLVcT7UzKWrag@mail.gmail.com>

2011/8/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 03.08.2011 21:07:
>>>>
>>>> Stefan, do you know why skip_dispatch argument is used for
>>>> module-level cpdef function?
>>>>
>>>> There is warning about that too.
>
> It seems you already found a way to handle it.
>
>
>>> And here is my branch for unused_arg:
>>>
>>> https://github.com/vitek/cython/commits/_unused_args
>>
>> I've reverted skip_dispatch argument removal, that doesn't work this
>> way and is more general issue.
>> Now I mark all unused local entries and args with CYTHON_UNUSED.
>
> Cool.
>
>
>> That actually isn't an optimization just some warnings removal.
>>
>> I think real optimization should be done when we start with DefNode
>> refactoring.
>
> Absolutely.
>
>
>> If you like that I'm gonna start pull request.
>
> It looks good to me, so please do.
>

Thanks, here it is
https://github.com/cython/cython/pull/50


-- 
vitja.

From d.s.seljebotn at astro.uio.no  Thu Aug  4 10:15:27 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Aug 2011 10:15:27 +0200
Subject: [Cython] Willing to contribute
In-Reply-To: <201108032253.36332@a>
References: <201108032253.36332@a>
Message-ID: <4E3A551F.2000601@astro.uio.no>

On 08/03/2011 10:53 PM, Stefano wrote:
> Hi everybody,
>
> I've been a enthusiast Python programmer for 3 years now, and I've hailed
> Cython project with great interest. Now, I'm willing to contribute. I've would
> say I'm an quite an expert Python programmer and I've some-year-long
> experience with C, modern C++ and its cousin Java. Sidenote, I'm a student at
> Politecnico di Milano, Italy.
>
> Everything started with trying to speed up a Mandelbrot generator, which
> heavily relies on complex numbers. Eventually I've found in "Mini Projects (or
> something similar)" wiki that C99 `complex' is still to be implemented. So, I
> decide to make a step further and try to hop into development.
>
> I've would be grateful if a core developer could point me in the right
> direction concerning coding style, repositories, deadlines (I'm still a
> student, so I may have 'offline' days this winter) and other issues. I may
> start with something easy (`complex' implementation, or command line API
> restructuration), and I'd like someone from the team to discuss what I'll be
> implementing (ok, just for the first days), just to see if  "I'm doing it
> right" (TM) and, mostly, if I'm the right man or I'm just making you loose
> your time.

Great!

A history lesson: On a higher level, the Cython code base originated in 
Pyrex, and has then gone through a number of refactorings (introduce a 
pipeline with transforms etc.). While refactoring, one always made sure 
Cython was shipable (at least within a couple of months time), so all 
refactorings were small an isolated. I.e., (like most code bases), 
there's no "grand design idea". I think there used to be one in Pyrex, 
but we've patched ourself far away from that now.

What I'm saying is: If something appears strange, chances are it is 
simply an artifact of something that made sense before a refactor, but 
wasn't taken into account while refactoring. So any cleanups (such as 
command line API restructuring) are welcome.

To really get drawn in Cython development though (and to get motivated 
and think it is fun), what you should eventually do is find a new 
feature that you want to work on -- that's when Cython development 
becomes really fun! Have a look at the enhancements wiki page and see if 
anything catches your interest, if you don't get very interested in 
seeing a feature working then things get much harder. Some ideas:

  - Python compatability bugs are the most important ones to get to 
Cython 1.0. OTOH, the low-hanging ones are mostly taken I think, so that 
the remaining ones may require a detailed knowledge of the inner 
workings of Cython and CPython. But have a look through trac.cython.org...

  - One idea is coercion of C pointers to ctypes Python objects and back 
again. This would allow any method to be "cpdef", so that "cdef" methods 
doesn't need to be learned (unless you want to keep things private). 
This may in time lead to simplifying the Cython language by merging cdef 
and cpdef functions/methods.

  - Supporting C++/C99 "const"...

  - Some CEPs I'd like to see implemented personally:

http://wiki.cython.org/enhancements/nativecall
http://wiki.cython.org/enhancements/signals


Dag Sverre

From stefan_ml at behnel.de  Thu Aug  4 10:22:13 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Aug 2011 10:22:13 +0200
Subject: [Cython] Willing to contribute
In-Reply-To: <201108032253.36332@a>
References: <201108032253.36332@a>
Message-ID: <4E3A56B5.4050800@behnel.de>

Stefano, 03.08.2011 22:53:
> I've been a enthusiast Python programmer for 3 years now, and I've hailed
> Cython project with great interest. Now, I'm willing to contribute.

Cool. Since no-one pointed you there yet, there's a hacking guide with some 
hints to get you started:

http://wiki.cython.org/HackerGuide

Stefan

From greg.ewing at canterbury.ac.nz  Thu Aug  4 23:51:44 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 05 Aug 2011 09:51:44 +1200
Subject: [Cython] Willing to contribute
In-Reply-To: <4E3A551F.2000601@astro.uio.no>
References: <201108032253.36332@a> <4E3A551F.2000601@astro.uio.no>
Message-ID: <4E3B1470.8080709@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

>  - One idea is coercion of C pointers to ctypes Python objects and back 
> again.

Some way of requesting this manually might be useful, but
I don't think I'd like it to happen automatically. Slinging
raw pointers around in Python isn't something to be done
lightly -- even if all the code that dereferences it is
in Cython, there are problems with managing the lifetime
of whatever it references.

-- 
Greg

From robertwb at math.washington.edu  Fri Aug  5 01:31:35 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Aug 2011 16:31:35 -0700
Subject: [Cython] Ctypes object <-> pointer conversion.
Message-ID: <CADiQ+QAsAdu46EvJ2ASHYOKLML-u4PDZSO0=6qH4+VRquxQsow@mail.gmail.com>

This is getting a bit OT but is worth discussing, so I'm starting a new thread.

On Thu, Aug 4, 2011 at 2:51 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Dag Sverre Seljebotn wrote:
>
>> ?- One idea is coercion of C pointers to ctypes Python objects and back
>> again.
>
> Some way of requesting this manually might be useful, but
> I don't think I'd like it to happen automatically. Slinging
> raw pointers around in Python isn't something to be done
> lightly -- even if all the code that dereferences it is
> in Cython, there are problems with managing the lifetime
> of whatever it references.

You know, C has the same problem with slinging around raw pointers and
managing their lifetimes :). Of course, with C, the user is all to
painfully aware of the situation.

I think this would be most useful for providing data from Python (even
an interactive prompt) to a Cython module. I agree about it being
dangerous to do implicitly, but something explicit like

def entry_point(ctypes.CData x):
    cdef double* result = func(<int*>x)
    return <ctypes.CData>result   # or cytpes.CData(result,
option=...) to control deallocation

could be really nice. Maybe CData could even be parameterized.

- Robert

From robertwb at math.washington.edu  Fri Aug  5 08:53:12 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Aug 2011 23:53:12 -0700
Subject: [Cython] object to struct conversion
In-Reply-To: <CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
	<4E397567.1060708@behnel.de>
	<CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>
	<CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>
Message-ID: <CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>

On Wed, Aug 3, 2011 at 2:09 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 3 August 2011 19:36, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Wed, Aug 3, 2011 at 9:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 03.08.2011 16:28:
>>>>
>>>> Is there any specific reason objects cannot coerce to structs (from
>>>> e.g. dicts?). It would be convenient for memoryviews, then you could
>>>> assign dicts (or any mapping) to items in the memoryview from Python
>>>> space. You could also have structs as argument to def functions etc.
>>>> Any objection to this addition? The implementation is pretty
>>>> straightforward.
>>>
>>> Robert implemented these things a while ago. I think I recall that there
>>> were unsupported use cases, but I'm not sure what exactly is missing here.
>>
>> It just simply wasn't implemented yet.
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> Ok, thanks guys, it's implemented in _memview:
> https://github.com/markflorisson88/cython/commit/6a2560b5b151a788ed11671663c833b6de6daccd#L4R1
>
> BTW, my hudson sdist keeps failing with
>
> writing manifest file 'MANIFEST'
> making hard links in Cython-0.14.1+...
> hard linking COPYING.txt -> Cython-0.14.1+
> error: File exists

I'm not sure. I removed that directory entirely and scheduled another
build; we'll see if that helps.

> any idea what that is all about? Are these hudson scripts available for perusal?

Yes, but you need to have an account. We'd be happy to set you up with
one (though I thought you already had access).

- Robert

From d.s.seljebotn at astro.uio.no  Fri Aug  5 08:58:20 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 05 Aug 2011 08:58:20 +0200
Subject: [Cython] Ctypes object <-> pointer conversion.
In-Reply-To: <CADiQ+QAsAdu46EvJ2ASHYOKLML-u4PDZSO0=6qH4+VRquxQsow@mail.gmail.com>
References: <CADiQ+QAsAdu46EvJ2ASHYOKLML-u4PDZSO0=6qH4+VRquxQsow@mail.gmail.com>
Message-ID: <4E3B948C.2080005@astro.uio.no>

On 08/05/2011 01:31 AM, Robert Bradshaw wrote:
> This is getting a bit OT but is worth discussing, so I'm starting a new thread.
>
> On Thu, Aug 4, 2011 at 2:51 PM, Greg Ewing<greg.ewing at canterbury.ac.nz>  wrote:
>> Dag Sverre Seljebotn wrote:
>>
>>>   - One idea is coercion of C pointers to ctypes Python objects and back
>>> again.
>>
>> Some way of requesting this manually might be useful, but
>> I don't think I'd like it to happen automatically. Slinging
>> raw pointers around in Python isn't something to be done
>> lightly -- even if all the code that dereferences it is
>> in Cython, there are problems with managing the lifetime
>> of whatever it references.
>
> You know, C has the same problem with slinging around raw pointers and
> managing their lifetimes :). Of course, with C, the user is all to
> painfully aware of the situation.
>
> I think this would be most useful for providing data from Python (even
> an interactive prompt) to a Cython module. I agree about it being
> dangerous to do implicitly, but something explicit like
>
> def entry_point(ctypes.CData x):
>      cdef double* result = func(<int*>x)
>      return<ctypes.CData>result   # or cytpes.CData(result,
> option=...) to control deallocation
>
> could be really nice. Maybe CData could even be parameterized.

This doesn't let you write

cpdef int func(int *a): ...

though. My motivation for this was mainly to remove an exception from 
the rules, and make sure that every C level object .

Although, C++ classes are another gaping exception to the rules. Perhaps 
just allowing the extension to 'cpdef' above but instead generate a stub 
that always raises a TypeError for the 'def' version is a better idea? 
That would allow fetching the docstring etc. from Python space, while 
the function remains uncallable. Even if we get int*<->ctypes.pointer, I 
propose this idea for C++ classes.

But I can also see that it'd be really handy if you, say, have a huge 
API wrapped with ctypes, and want to gradually and piecewise move over 
to Cython. Or do some low-level integration between a ctypes wrapper and 
Cython wrapper.

What's ctypes.CData? It's not in my ctypes module...

Dag Sverre

From robertwb at math.washington.edu  Fri Aug  5 09:07:23 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Aug 2011 00:07:23 -0700
Subject: [Cython] Ctypes object <-> pointer conversion.
In-Reply-To: <4E3B948C.2080005@astro.uio.no>
References: <CADiQ+QAsAdu46EvJ2ASHYOKLML-u4PDZSO0=6qH4+VRquxQsow@mail.gmail.com>
	<4E3B948C.2080005@astro.uio.no>
Message-ID: <CADiQ+QDPUjyCKjXnyEgC5QtEbCqNHHzrjb+28_VNAOh2obYfHQ@mail.gmail.com>

On Thu, Aug 4, 2011 at 11:58 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/05/2011 01:31 AM, Robert Bradshaw wrote:
>>
>> This is getting a bit OT but is worth discussing, so I'm starting a new
>> thread.
>>
>> On Thu, Aug 4, 2011 at 2:51 PM, Greg Ewing<greg.ewing at canterbury.ac.nz>
>> ?wrote:
>>>
>>> Dag Sverre Seljebotn wrote:
>>>
>>>> ?- One idea is coercion of C pointers to ctypes Python objects and back
>>>> again.
>>>
>>> Some way of requesting this manually might be useful, but
>>> I don't think I'd like it to happen automatically. Slinging
>>> raw pointers around in Python isn't something to be done
>>> lightly -- even if all the code that dereferences it is
>>> in Cython, there are problems with managing the lifetime
>>> of whatever it references.
>>
>> You know, C has the same problem with slinging around raw pointers and
>> managing their lifetimes :). Of course, with C, the user is all to
>> painfully aware of the situation.
>>
>> I think this would be most useful for providing data from Python (even
>> an interactive prompt) to a Cython module. I agree about it being
>> dangerous to do implicitly, but something explicit like
>>
>> def entry_point(ctypes.CData x):
>> ? ? cdef double* result = func(<int*>x)
>> ? ? return<ctypes.CData>result ? # or cytpes.CData(result,
>> option=...) to control deallocation
>>
>> could be really nice. Maybe CData could even be parameterized.
>
> This doesn't let you write
>
> cpdef int func(int *a): ...
>
> though.

I wasn't thinking of excluding this, but I'd was thinking

    cdef object o = ...
    int* x = o
    o = x

might be a compile-time error, forcing the user to be a bit more
explicit (especially the latter).

> My motivation for this was mainly to remove an exception from the
> rules, and make sure that every C level object .

?

> Although, C++ classes are another gaping exception to the rules.

On this note, what about SWIG-wrapped types?

> Perhaps
> just allowing the extension to 'cpdef' above but instead generate a stub
> that always raises a TypeError for the 'def' version is a better idea? That
> would allow fetching the docstring etc. from Python space, while the
> function remains uncallable. Even if we get int*<->ctypes.pointer, I propose
> this idea for C++ classes.

That's an interesting idea.

> But I can also see that it'd be really handy if you, say, have a huge API
> wrapped with ctypes, and want to gradually and piecewise move over to
> Cython. Or do some low-level integration between a ctypes wrapper and Cython
> wrapper.
>
> What's ctypes.CData? It's not in my ctypes module...

It's really _ctypes._CData.

- Robert

From markflorisson88 at gmail.com  Fri Aug  5 10:49:18 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 5 Aug 2011 10:49:18 +0200
Subject: [Cython] object to struct conversion
In-Reply-To: <CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
	<4E397567.1060708@behnel.de>
	<CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>
	<CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>
	<CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>
Message-ID: <CANg26EWWM5_jNxnH3_yjE0ux=sHS5DtxyuWs4_=Uoh-EAKWW=Q@mail.gmail.com>

On 5 August 2011 08:53, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Aug 3, 2011 at 2:09 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 3 August 2011 19:36, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Wed, Aug 3, 2011 at 9:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 03.08.2011 16:28:
>>>>>
>>>>> Is there any specific reason objects cannot coerce to structs (from
>>>>> e.g. dicts?). It would be convenient for memoryviews, then you could
>>>>> assign dicts (or any mapping) to items in the memoryview from Python
>>>>> space. You could also have structs as argument to def functions etc.
>>>>> Any objection to this addition? The implementation is pretty
>>>>> straightforward.
>>>>
>>>> Robert implemented these things a while ago. I think I recall that there
>>>> were unsupported use cases, but I'm not sure what exactly is missing here.
>>>
>>> It just simply wasn't implemented yet.
>>>
>>> - Robert
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> Ok, thanks guys, it's implemented in _memview:
>> https://github.com/markflorisson88/cython/commit/6a2560b5b151a788ed11671663c833b6de6daccd#L4R1
>>
>> BTW, my hudson sdist keeps failing with
>>
>> writing manifest file 'MANIFEST'
>> making hard links in Cython-0.14.1+...
>> hard linking COPYING.txt -> Cython-0.14.1+
>> error: File exists
>
> I'm not sure. I removed that directory entirely and scheduled another
> build; we'll see if that helps.

Thanks! That did the trick.

>> any idea what that is all about? Are these hudson scripts available for perusal?
>
> Yes, but you need to have an account. We'd be happy to set you up with
> one (though I thought you already had access).
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Yeah I have an account. I suppose I should read some documentation to
find out where to find the scripts :)

From stefan_ml at behnel.de  Fri Aug  5 11:05:18 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 05 Aug 2011 11:05:18 +0200
Subject: [Cython] object to struct conversion
In-Reply-To: <CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>	<4E397567.1060708@behnel.de>	<CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>	<CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>
	<CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>
Message-ID: <4E3BB24E.40509@behnel.de>

Robert Bradshaw, 05.08.2011 08:53:
> On Wed, Aug 3, 2011 at 2:09 PM, mark florisson wrote:
>> my hudson sdist keeps failing with
>>
>> writing manifest file 'MANIFEST'
>> making hard links in Cython-0.14.1+...
>> hard linking COPYING.txt ->  Cython-0.14.1+
>> error: File exists
>
> I'm not sure. I removed that directory entirely and scheduled another
> build; we'll see if that helps.

BTW, just in case: this can be done through the web interface by clicking 
on the "workspace" of the job and then the "clear" link that appears below 
that.

Stefan

From markflorisson88 at gmail.com  Fri Aug  5 11:40:21 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 5 Aug 2011 11:40:21 +0200
Subject: [Cython] object to struct conversion
In-Reply-To: <4E3BB24E.40509@behnel.de>
References: <CANg26EUBMuKVSufqeu2+gCdQw7F+MNKG2=NoQ1j0kZMC7zNADw@mail.gmail.com>
	<4E397567.1060708@behnel.de>
	<CADiQ+QA7abP+i860ZQ01YHtVq_kq3Ki16JfscTabZSFecV3C9Q@mail.gmail.com>
	<CANg26EWL-XtUx78QARj69+5jojSJayvWtZZhSKVTj-QRJUn7ng@mail.gmail.com>
	<CADiQ+QBTmouVAQcu9x-q2o9z6Ygr_emNGgqSHMUKyRm5fF77Aw@mail.gmail.com>
	<4E3BB24E.40509@behnel.de>
Message-ID: <CANg26EUmu=b-36CGvrzRtJxkdtMkZ4Rxe4+JXqMUgoEq982FsQ@mail.gmail.com>

On 5 August 2011 11:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 05.08.2011 08:53:
>>
>> On Wed, Aug 3, 2011 at 2:09 PM, mark florisson wrote:
>>>
>>> my hudson sdist keeps failing with
>>>
>>> writing manifest file 'MANIFEST'
>>> making hard links in Cython-0.14.1+...
>>> hard linking COPYING.txt -> ?Cython-0.14.1+
>>> error: File exists
>>
>> I'm not sure. I removed that directory entirely and scheduled another
>> build; we'll see if that helps.
>
> BTW, just in case: this can be done through the web interface by clicking on
> the "workspace" of the job and then the "clear" link that appears below
> that.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

I see, thanks Stefan!

From robertwb at math.washington.edu  Sat Aug  6 01:12:18 2011
From: robertwb at math.washington.edu (robertwb)
Date: Fri, 5 Aug 2011 16:12:18 -0700 (PDT)
Subject: [Cython] Cython 0.15 released
Message-ID: <a22f3bde-0758-4f33-a1c8-f5d171432952@e2g2000prj.googlegroups.com>

We are excited to announce the release of Cython 0.15, which is a huge
step forward in achieving full Python language coverage as well as
many new features, optimizations, and bugfixes.

Download: http://cython.org/ or http://pypi.python.org/pypi/Cython

== Major Features ==

 * Generators (yield) - Cython has full support for generators,
generator expressions and coroutines. http://www.python.org/dev/peps/pep-0342/

 * The nonlocal keyword is supported.

 * Re-acquiring the gil: with gil - works as expected within a nogil
context.

 * OpenMP support: http://docs.cython.org/0.15/src/userguide/parallelism.html

 * Control flow analysis prunes dead code and emits warnings and
errors about uninitialised variables.

 * Debugger command cy set to assign values of expressions to Cython
variables and cy exec counterpart cy_eval().

 * Exception chaining http://www.python.org/dev/peps/pep-3134/

 * Relative imports http://www.python.org/dev/peps/pep-0328/

 * The with statement has its own dedicated and faster C
implementation.

 * Improved pure syntax including cython.cclass, cython.cfunc, and
cython.ccall. http://docs.cython.org/0.15/src/tutorial/pure.html

 * Support for del.

 * Boundschecking directives implemented for builtin Python sequence
types.

 * Several updates and additions to the shipped standard library pxd
files https://github.com/cython/cython/tree/master/Cython/Includes

 * Forward declaration of types is no longer required for circular
references.

Note: this will be the last release to support Python 2.3; Python 2.4
will be supported for at least one more release.

== General improvements and bug fixes ==

This release contains over a thousand commits including hundreds of
bugfixes and optimizations.  The bug tracker has not been as heavily
used this release cycle, but is still useful
http://trac.cython.org/cython_trac/query?status=closed&group=component&order=id&col=id&col=summary&col=milestone&col=status&col=type&col=priority&col=owner&col=component&milestone=0.15&desc=1

== Incompatible changes ==

 * Uninitialized variables are no longer initialized to None and
accessing them has the same semantics as standard Python.

 * globals() now returns a read-only dict of the Cython module's
globals, rather than the globals
   of the first non-Cython module in the stack

 * Many C++ exceptions are now special cases to give closer Python
counterparts.  This means that except+ functions that formally raised
generic RuntimeErrors may raise something else such as
ArithmaticError.

== Known regressions ==

 * The inlined generator expressions (introduced in Cython 0.13) were
disabled in favour of full generator expression support. This induces
a performance regression for cases that were previously inlined.

== Contributors ==

Many thanks to:

Francesc Alted,
Haoyu Bai,
Stefan Behnel,
Robert Bradshaw,
Lars Buitinck,
Lisandro Dalcin,
John Ehresman,
Mark Florisson,
Christoph Gohlke,
Jason Grout,
Chris Lasher,
Vitja Makarov,
Brent Pedersen,
Dag Sverre Seljebotn,
Nathaniel Smith,
and Pauli Virtanen


From stefano.k.sanfilippo at gmail.com  Sun Aug  7 14:06:19 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Sun, 7 Aug 2011 14:06:19 +0200
Subject: [Cython] CmdLine.py refactoring
Message-ID: <201108071406.19312@a>

Hi,

I've cloned GitHub repo and started working on it. First, I'm rewriting 
Compiler/CmdLine.py to use the optparse/argparse module. Then I'll try to 
address the Bugzilla ticket mentioned in the source file.

Question: optparse has been deprecated since 2.7 (which adopts optparse 
module). So, which one to choose? Old one, or new one? Or should I work on a 
fallback system?

--SKS

From stefano.k.sanfilippo at gmail.com  Sun Aug  7 14:16:21 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Sun, 7 Aug 2011 14:16:21 +0200
Subject: [Cython]  CommandLine.py refactoring
Message-ID: <201108071416.21322@a>

> Then I'll try to  address the Bugzilla ticket mentioned in the source file.

Well, I meant Trac ticket.

--SKS

From stefan_ml at behnel.de  Sun Aug  7 16:21:06 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Aug 2011 16:21:06 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <201108071406.19312@a>
References: <201108071406.19312@a>
Message-ID: <4E3E9F52.7010403@behnel.de>

Stefano, 07.08.2011 14:06:
> I've cloned GitHub repo and started working on it.

Will you put it up somewhere on github?


> First, I'm rewriting
> Compiler/CmdLine.py to use the optparse/argparse module.

That would help in keeping the command line help messages up to date with 
the actually supported options, but given that the current implementation 
"ain't broken", it's certainly not a priority to us.

I'm also not sure how well this will fit in, given that the default options 
are stored in Main.py and don't belong in CmdLine.py as the command line is 
only one of the ways to run Cython. I'd prefer not to duplicate too much of 
that setup.


> Then I'll try to
> address the Bugzilla ticket mentioned in the source file.

That's the "-r" option for recursive compilation of dependencies. I don't 
think it'll be all that easy to fix, but I think this would totally be 
worth a try. It may be easier to implement now that we have 
Cython.Build.Dependencies and friends.

Personally, I'd recommend starting with something like that, instead of 
putting too much effort into the command line parser (which changes pretty 
rarely these days).

BTW, it seems that trac is currently broken, so I can't actually look up 
the ticket.


> Question: optparse has been deprecated since 2.7 (which adopts optparse
> module). So, which one to choose? Old one, or new one? Or should I work on a
> fallback system?

argparse is new and not supported by anything but 2.7 and recent 3.x 
versions. Cython currently runs on Python 2.4, which rules out a dependency 
on argparse. I never tried it, so I can't tell what exactly the advantages 
over optparse are. They certainly can't be big enough to make up for a 
duplicate implementation of Cython's cmd line parser.

Stefan

From stefano.k.sanfilippo at gmail.com  Sun Aug  7 21:58:46 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Sun, 7 Aug 2011 21:58:46 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E3E9F52.7010403@behnel.de>
References: <201108071406.19312@a> <4E3E9F52.7010403@behnel.de>
Message-ID: <201108072158.46235@a>

Sunday 7th August 2011 16:21:06, Stefan Behnel wrote:
> Will you put it up somewhere on github?

Here's my branch for command line refactoring. There's still nothing 
interesting on, but changes will be pushed soon.

https://github.com/satufk/cython/tree/_commandline

> I'm also not sure how well this will fit in, given that the default options
> are stored in Main.py and don't belong in CmdLine.py as the command line is
> only one of the ways to run Cython. I'd prefer not to duplicate too much of
> that setup.

I agree. That's the reason while I'm thinking about a partial refactoring of 
all the Options&co. system, to provide a more coherent interface. Well, at 
least, I'm just evaluating at what extent it could be possible.

> argparse is new and not supported by anything but 2.7 and recent 3.x
> versions. Cython currently runs on Python 2.4, which rules out a dependency
> on argparse. I never tried it, so I can't tell what exactly the advantages
> over optparse are. They certainly can't be big enough to make up for a
> duplicate implementation of Cython's cmd line parser.

Fixed the issue. Simply, try: catch the import. The modules are very similar, 
so no more than 10 lines of overhead.  When the old module will disappear, 
I'll just remove that lines :)

--SKS

From stefano.k.sanfilippo at gmail.com  Mon Aug  8 22:36:13 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Mon, 8 Aug 2011 22:36:13 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E3E9F52.7010403@behnel.de>
References: <201108071406.19312@a> <4E3E9F52.7010403@behnel.de>
Message-ID: <201108082236.13979@a>

Sunday 7th August 2011 16:21:06, Stefan Behnel wrote:
> argparse is new and not supported by anything but 2.7 and recent 3.x
> versions. Cython currently runs on Python 2.4, which rules out a dependency
> on argparse. I never tried it, so I can't tell what exactly the advantages
> over optparse are. They certainly can't be big enough to make up for a
> duplicate implementation of Cython's cmd line parser.

Work on commandline is almost over, then I'll expose my refactoring proposal 
:)

Working with argparse vs. optparse showed me that, indeed, the two things are 
much different (and, in this sense, argparse is more straightforward than 
optparse), moreover that the optparse (nor the old if: elif: approach) version 
cannot parse complicated commandlines (see Tests/TestCommandLine.py)  - a 
thing which we may expect with some exoteric build system producing 10-line-
long command lines (a.k.a. autotools) . 

As a proof of its quality, Fedora, Ubuntu and SuSe all ship a python-argparse 
package for older versions of python (<2.7). Given that the module is self-
contained and as small as 80KB, wouldn't be feasible to add it as a build 
dependency?

So that I could drop the alternate implementation - which, sadly, makes one of 
the tests from the testsuite (apart from my own) fail.

Find all commits at: https://github.com/satufk/cython/commits/_commandline

--SKS

From stefan_ml at behnel.de  Tue Aug  9 08:33:42 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 09 Aug 2011 08:33:42 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <201108082236.13979@a>
References: <201108071406.19312@a> <4E3E9F52.7010403@behnel.de>
	<201108082236.13979@a>
Message-ID: <4E40D4C6.5090709@behnel.de>

Stefano, 08.08.2011 22:36:
> As a proof of its quality, Fedora, Ubuntu and SuSe all ship a python-argparse
> package for older versions of python (<2.7). Given that the module is self-
> contained and as small as 80KB, wouldn't be feasible to add it as a build
> dependency?

Note that that's huge compared to the current command line parser. The 
current implementation doesn't even use 8K, and it's perfectly 
self-contained as well.

Could you try to explain what the advantages of your new implementation are?

Stefan

From vitja.makarov at gmail.com  Tue Aug  9 08:44:03 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 9 Aug 2011 10:44:03 +0400
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E40D4C6.5090709@behnel.de>
References: <201108071406.19312@a> <4E3E9F52.7010403@behnel.de>
	<201108082236.13979@a> <4E40D4C6.5090709@behnel.de>
Message-ID: <CAKGHGPSviJyS_E_XjGm5DDBeVJRLNp3v+7ype5CvWiaNXdk6JA@mail.gmail.com>

2011/8/9 Stefan Behnel <stefan_ml at behnel.de>:
> Stefano, 08.08.2011 22:36:
>>
>> As a proof of its quality, Fedora, Ubuntu and SuSe all ship a
>> python-argparse
>> package for older versions of python (<2.7). Given that the module is
>> self-
>> contained and as small as 80KB, wouldn't be feasible to add it as a build
>> dependency?
>
> Note that that's huge compared to the current command line parser. The
> current implementation doesn't even use 8K, and it's perfectly
> self-contained as well.
>
> Could you try to explain what the advantages of your new implementation are?
>

One more note: with optparser it would be harder to handle options like -Werror,
that's easy to remove them at all, or they could be handled as "W:"

Don't know how it's made in gcc.

-- 
vitja.

From d.s.seljebotn at astro.uio.no  Tue Aug  9 08:51:52 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Aug 2011 08:51:52 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E40D4C6.5090709@behnel.de>
References: <201108071406.19312@a>
	<4E3E9F52.7010403@behnel.de>	<201108082236.13979@a>
	<4E40D4C6.5090709@behnel.de>
Message-ID: <4E40D908.1040303@astro.uio.no>

On 08/09/2011 08:33 AM, Stefan Behnel wrote:
> Stefano, 08.08.2011 22:36:
>> As a proof of its quality, Fedora, Ubuntu and SuSe all ship a
>> python-argparse
>> package for older versions of python (<2.7). Given that the module is
>> self-
>> contained and as small as 80KB, wouldn't be feasible to add it as a build
>> dependency?
>
> Note that that's huge compared to the current command line parser. The
> current implementation doesn't even use 8K, and it's perfectly
> self-contained as well.

I'd prefer if optparse was sufficient. The main advantage of argparse to 
me seems to be in parsing command structures, such as "hg add ...", "hg 
init ...". Since we don't do that in Cython I'd think optparse would be 
sufficient, and that we should not ship argparse.

> Could you try to explain what the advantages of your new implementation
> are?

Assuming one switches to optparse, I find this line of reasoning highly 
dubious... and think the burden should be the other way: Please argue 
why custom, specific, low-level code is necesarry in Cython's command 
line parser.

Using optparse/argparse have the obvious benefits that they are higher 
level and so more stuff can be automated. In particular, better 
automatically up-to-date command line help tracking changes elsewhere in 
the codebase (such as directives that can be passed to -X).

Dag Sverre

From stefan_ml at behnel.de  Tue Aug  9 10:15:14 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 09 Aug 2011 10:15:14 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E40D908.1040303@astro.uio.no>
References: <201108071406.19312@a>	<4E3E9F52.7010403@behnel.de>	<201108082236.13979@a>	<4E40D4C6.5090709@behnel.de>
	<4E40D908.1040303@astro.uio.no>
Message-ID: <4E40EC92.1050901@behnel.de>

Dag Sverre Seljebotn, 09.08.2011 08:51:
> On 08/09/2011 08:33 AM, Stefan Behnel wrote:
>> Stefano, 08.08.2011 22:36:
>>> As a proof of its quality, Fedora, Ubuntu and SuSe all ship a
>>> python-argparse
>>> package for older versions of python (<2.7). Given that the module is
>>> self-
>>> contained and as small as 80KB, wouldn't be feasible to add it as a build
>>> dependency?
>>
>> Note that that's huge compared to the current command line parser. The
>> current implementation doesn't even use 8K, and it's perfectly
>> self-contained as well.
>
> I'd prefer if optparse was sufficient. The main advantage of argparse to me
> seems to be in parsing command structures, such as "hg add ...", "hg init
> ...". Since we don't do that in Cython I'd think optparse would be
> sufficient, and that we should not ship argparse.

Agreed. I never understood why the CPython devs decided to deprecate 
optparse in 2.7.


>> Could you try to explain what the advantages of your new implementation
>> are?
>
> Assuming one switches to optparse, I find this line of reasoning highly
> dubious... and think the burden should be the other way: Please argue why
> custom, specific, low-level code is necesarry in Cython's command line parser.

Since Stefano has decided to write the code, I think the "it's there and it 
works" argument is basically dead already. And I do see the advantage of 
using the "standard" tool for the job.

However, I also see the problem that optparse is deprecated and may get 
removed in a future 3.x version. So, switching to optparse now means that 
we may have to switch to something else later, whereas that's not the case 
with the current code, however clumsy and low-level it may be.

 From a look at Stefano's code, it seems that it's not all that hard to 
support both argparse and optparse. If we can keep that up, that would 
solve the future upgrade problem.

I actually think that the new implementation looks much clearer than what 
we currently have:

https://github.com/satufk/cython/blob/13309377dd3fd90761e97718fc3297efe1e962dc/Cython/Compiler/CmdLine.py


> Using optparse/argparse have the obvious benefits that they are higher
> level and so more stuff can be automated. In particular, better
> automatically up-to-date command line help tracking changes elsewhere in
> the codebase (such as directives that can be passed to -X).

Yes, the list of "-X" options in the cmd line help text should be 
automatically generated from the Options. And, yes, the current status of 
the help text suggests that it does represent a notable maintenance 
overhead to keep it up to date with the implementation.

Stefan

From stefano.k.sanfilippo at gmail.com  Tue Aug  9 12:13:52 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Tue, 9 Aug 2011 12:13:52 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <4E40EC92.1050901@behnel.de>
References: <201108071406.19312@a> <4E40D908.1040303@astro.uio.no>
	<4E40EC92.1050901@behnel.de>
Message-ID: <201108091213.52282@a>

Tuesday 9th August 2011 10:15:14, Stefan Behnel wrote:
> However, I also see the problem that optparse is deprecated and may get 
> removed in a future 3.x version. So, switching to optparse now means that 
> we may have to switch to something else later, whereas that's not the case 
> with the current code, however clumsy and low-level it may be.

I've fixed the glitches, now both versions (argparse and optparse) are 
supported. Still, optparse's code performs a bit faster and doesn't need a 
couple of HACKs that are necessary for optparse's code to run. If all tests 
perform correctly, I'll merge the branch into master.

So, no upgrade problems, nor optparse embedding issues :)

I did want to rewrite the module beacuse I think that clarity of 
implementation should win over milliseconds of speed. Which, I think, is the 
very same spirit on which Python is founded.

Regards,

--SKS

From stefan_ml at behnel.de  Tue Aug  9 13:22:58 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 09 Aug 2011 13:22:58 +0200
Subject: [Cython] CmdLine.py refactoring
In-Reply-To: <201108091213.52282@a>
References: <201108071406.19312@a>
	<4E40D908.1040303@astro.uio.no>	<4E40EC92.1050901@behnel.de>
	<201108091213.52282@a>
Message-ID: <4E411892.5080007@behnel.de>

Stefano, 09.08.2011 12:13:
> Tuesday 9th August 2011 10:15:14, Stefan Behnel wrote:
>> However, I also see the problem that optparse is deprecated and may get
>> removed in a future 3.x version. So, switching to optparse now means that
>> we may have to switch to something else later, whereas that's not the case
>> with the current code, however clumsy and low-level it may be.
>
> I've fixed the glitches, now both versions (argparse and optparse) are
> supported. Still, optparse's code performs a bit faster and doesn't need a
> couple of HACKs that are necessary for optparse's code to run. If all tests
> perform correctly, I'll merge the branch into master.
>
> So, no upgrade problems, nor optparse embedding issues :)

Thanks!


> I did want to rewrite the module beacuse I think that clarity of
> implementation should win over milliseconds of speed.

This is really old code and performance certainly is not an issue. I 
wouldn't be surprised if the implementation simply predated the addition of 
command line parsers to the stdlib.

In any case, we are happy about all contributions, not only those that 
address our most urgent needs. Your changes clearly make the command line 
parser more readable.

When you're done with it, please open a pull request.

Stefan

From stefan_ml at behnel.de  Thu Aug 11 16:00:02 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Aug 2011 16:00:02 +0200
Subject: [Cython] [cython-users] Calling gil-requiring function not
 allowed without gil
In-Reply-To: <4E43D0CA.1060900@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
Message-ID: <4E43E062.5000502@behnel.de>

[moving this away from cython-users]

Dag Sverre Seljebotn, 11.08.2011 14:53:
> On 08/11/2011 02:13 PM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 11.08.2011 13:58:
>>> On 08/11/2011 01:43 PM, Ting Zhou wrote:
>>>> here is my pyx file. When I compile it, it says "Calling gil-requiring
>>>> function not allowed without gil" at calling dabs(x[i]). Why my dabs
>>>> function is a gil-requiring function?
>>>>
>>>> ====================================
>>>> from cython.parallel import *
>>>> import numpy as np
>>>> cimport numpy as np
>>>> cimport cython
>>>>
>>>> cdef double dabs(double x):
>>>
>>> This should be
>>>
>>> cdef double dabs(double x) nogil:
>>
>> Note that Cython cannot infer this automatically. Even a trivial
>> function may require an exclusive global lock for some reason, and it's
>> common to use the GIL for that. So the programmer must be explicit here.
>
> Are you still against this mini-CEP?:
>
> with cython.global_lock():
> ...
>
> Where global_lock() is GIL-requiring noop.
>
> This
>
> a) Improves code readability vastly. Having a critical section take effect
> because of the *lack* of a keyword is just very odd to anyone who's not
> shoulder deep in CPython internals

The GIL is more than a critical section. It's just there - and that's a 
good thing. Just take it as a part of the runtime environment, and disable 
it when you are sure you can *safely* do without it. A global lock makes 
life a lot easier, as it prevents unconditional (and unexpected) concurrency.


> b) Allows inferring 'nogil'

No it doesn't, because existing code doesn't use it. Only because you use 
one little statement in one part of your sources doesn't mean you know what 
you're doing, and that Cython can happily assume reversed semantics for the 
rest. Don't forget that threading is a dangerous abstraction, hard to get 
right and likely even the most error prone concurrency mechanism in 
widespread use. Not everything is "trivially parallelisable", and even 
those cases that are still produce enough problems.

Actually, it's worth asking if even the prange loop was a good idea, given 
the amount of confusion driven traffic that it currently produces on the 
mailing list. I'm not arguing against it, but it seems to be a trickier 
concept than it appears at first sight. IMHO, that's worth working on 
*before* we introduce even more hard-to-understand concepts and 
hard-to-control compiler trickery.

Stefan

From d.s.seljebotn at astro.uio.no  Thu Aug 11 16:16:36 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Aug 2011 16:16:36 +0200
Subject: [Cython] [cython-users] Calling gil-requiring function not
 allowed without gil
In-Reply-To: <4E43E062.5000502@behnel.de>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no> <4E43E062.5000502@behnel.de>
Message-ID: <4E43E444.1090501@astro.uio.no>

On 08/11/2011 04:00 PM, Stefan Behnel wrote:
> Actually, it's worth asking if even the prange loop was a good idea,
> given the amount of confusion driven traffic that it currently produces
> on the mailing list. I'm not arguing against it, but it seems to be a
> trickier concept than it appears at first sight. IMHO, that's worth
> working on *before* we introduce even more hard-to-understand concepts
> and hard-to-control compiler trickery.

Honestly. Have you even read the posts regarding the prange statements?

One was a missing DLL (similar in nature to "I can't find my Python.h"), 
and another tried to implement something which was fundamentally not 
parallizable using *any* conceivable framework for expressing any 
parallelism at all.

Nobody has as far as I remember had a problem that wouldn't be a problem 
using multiprocessing or some such instead.

I think you're very far from arguing rational in any sense about this, 
so I'll try my best to just shut up.

(Sorry all, but the statement about the prange loop blew my fuses...)

Dag Sverre

From stefano.k.sanfilippo at gmail.com  Thu Aug 11 16:24:37 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Thu, 11 Aug 2011 16:24:37 +0200
Subject: [Cython]  What now?
Message-ID: <201108111624.37503@a>

Hi,

now that I've nailed Cython code, I'd like to get into something more funny.
Currently, I'm working on a set of macros to seamlessy integrate Cython into 
CMake build process (in fact, I love CMake). But, I'd like to work also on 
something more essential, so... Is there any updated TODO list? (ToDo.txt in 
git repo dates back to more than 8 months ago). I may also work on the 
wiki....

Cheers,

--SKS

From stefan_ml at behnel.de  Thu Aug 11 18:34:06 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Aug 2011 18:34:06 +0200
Subject: [Cython] What now?
In-Reply-To: <201108111624.37503@a>
References: <201108111624.37503@a>
Message-ID: <4E44047E.1040402@behnel.de>

Stefano, 11.08.2011 16:24:
> I'd like to work also on
> something more essential, so... Is there any updated TODO list? (ToDo.txt in
> git repo dates back to more than 8 months ago).

The most up-to-date list of things to do is the bug tracker.

http://trac.cython.org/cython_trac/report/1?asc=0&sort=ticket

It contains a broad list of bugs and enhancements, anything from small 
issues to larger language features. Just give it a look and see if you find 
something that sounds interesting to you. You can also use the query page 
to select by components, such as potential optimisations, Python 
compatibility issues or Cython specific language features.

http://trac.cython.org/cython_trac/query

Note that some tickets may be incomprehensible without some background 
knowledge, but most of them should be ok. If you have any questions or if 
you find something that you want to tackle, please ask on this list about 
it. We'll try to get you going.

There's also the enhancement proposal list with some bigger topics, but 
most of them haven't been decided upon (just some wild ideas) and even the 
acceptable ones may be too large to get started. Also, the page is somewhat 
outdated in corners.

http://wiki.cython.org/enhancements

Hope that helps,

Stefan

From d.s.seljebotn at astro.uio.no  Thu Aug 11 19:51:14 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Aug 2011 19:51:14 +0200
Subject: [Cython] What now?
In-Reply-To: <201108111624.37503@a>
References: <201108111624.37503@a>
Message-ID: <4E441692.7080400@astro.uio.no>

On 08/11/2011 04:24 PM, Stefano wrote:
> Hi,
>
> now that I've nailed Cython code, I'd like to get into something more funny.
> Currently, I'm working on a set of macros to seamlessy integrate Cython into
> CMake build process (in fact, I love CMake). But, I'd like to work also on
> something more essential, so... Is there any updated TODO list? (ToDo.txt in
> git repo dates back to more than 8 months ago). I may also work on the
> wiki....
>

In addition to Stefan's suggestions, here's a small mini-CEP:

Error testcases in tests/errors are currently rather inconvenient, since 
changing anything often shifts a lot of error message line numbers. 
Something like this format might be better, where the line number is 
implied from the position of the comment:

# mode: error

with nogil:
     print 1 #ERROR:4:print statement not allowed in nogil block

I just filed this as http://trac.cython.org/cython_trac/ticket/710

Dag Sverre

From stefano.k.sanfilippo at gmail.com  Thu Aug 11 22:45:19 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Thu, 11 Aug 2011 22:45:19 +0200
Subject: [Cython] What now?
In-Reply-To: <4E441692.7080400@astro.uio.no>
References: <201108111624.37503@a> <4E441692.7080400@astro.uio.no>
Message-ID: <201108112245.19952@a>

Thursday 11th August 2011 19:51:14, Dag Sverre Seljebotn wrote:
> Error testcases in tests/errors are currently rather inconvenient, since
> changing anything often shifts a lot of error message line numbers.
> Something like this format might be better, where the line number is
> implied from the position of the comment:
> 
> # mode: error
> 
> with nogil:
>      print 1 #ERROR:4:print statement not allowed in nogil block
> 
> I just filed this as http://trac.cython.org/cython_trac/ticket/710

That's a good point. I've started working on it. Actually, runtests.py should 
be split into a module and a public run-main stub, to make it load faster. 
I'll try to cope with this too.

--SKS

From robertwb at math.washington.edu  Fri Aug 12 01:44:54 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Aug 2011 16:44:54 -0700
Subject: [Cython] What now?
In-Reply-To: <201108112245.19952@a>
References: <201108111624.37503@a> <4E441692.7080400@astro.uio.no>
	<201108112245.19952@a>
Message-ID: <CADiQ+QCirSOTJbTCcQV14J75AkMZFqLcBXMv0Lp01TnZXrYyiA@mail.gmail.com>

On Thu, Aug 11, 2011 at 1:45 PM, Stefano <stefano.k.sanfilippo at gmail.com> wrote:
> Thursday 11th August 2011 19:51:14, Dag Sverre Seljebotn wrote:
>> Error testcases in tests/errors are currently rather inconvenient, since
>> changing anything often shifts a lot of error message line numbers.
>> Something like this format might be better, where the line number is
>> implied from the position of the comment:
>>
>> # mode: error
>>
>> with nogil:
>> ? ? ?print 1 #ERROR:4:print statement not allowed in nogil block
>>
>> I just filed this as http://trac.cython.org/cython_trac/ticket/710
>
> That's a good point. I've started working on it.

That would be really nice.

> Actually, runtests.py should
> be split into a module and a public run-main stub, to make it load faster.
> I'll try to cope with this too.

I don't think this is really a concern.

In terms of the enhancements wiki, it's always a good idea to look at
the history to get an idea of how current/out of date a given CEP is.

- Robert

From romain.py at gmail.com  Fri Aug 12 03:19:01 2011
From: romain.py at gmail.com (Romain Guillebert)
Date: Fri, 12 Aug 2011 03:19:01 +0200
Subject: [Cython] Cython bug ?
Message-ID: <20110812011901.GA9596@ubuntu>

Hi

I tried to compiled Demos/primes.pyx using the ctypes backend and I
think I've found a bug :
The Entry for the kmax parameter does not set is_arg to 1. However if I
turn the def into a cdef, it works fine.

Is this a bug or a known hack ?

Thanks
Romain

From stefan_ml at behnel.de  Fri Aug 12 05:50:02 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Aug 2011 05:50:02 +0200
Subject: [Cython] Cython bug ? - Entry.is_arg flag not set for all
	arguments
In-Reply-To: <20110812011901.GA9596@ubuntu>
References: <20110812011901.GA9596@ubuntu>
Message-ID: <4E44A2EA.8010700@behnel.de>

[fixed subject]

Romain Guillebert, 12.08.2011 03:19:
> I tried to compiled Demos/primes.pyx using the ctypes backend and I
> think I've found a bug :
> The Entry for the kmax parameter does not set is_arg to 1. However if I
> turn the def into a cdef, it works fine.
>
> Is this a bug or a known hack ?

Vitja already brought this up, too:

http://article.gmane.org/gmane.comp.python.cython.devel/12385

A quick grep on the sources gives me this:

"""
Cython/Compiler/Buffer.py:            if entry.is_arg:
Cython/Compiler/Buffer.py:                if entry.is_arg:
Cython/Compiler/ExprNodes.py:        return entry and (entry.is_local or 
entry.is_arg) and not entry.in_closure
Cython/Compiler/FlowControl.py:        return (entry.is_local or 
entry.is_pyclass_attr or entry.is_arg or
Cython/Compiler/FlowControl.py:        self.is_arg = False
Cython/Compiler/FlowControl.py:        self.is_arg = True
Cython/Compiler/FlowControl.py:                if assmt.is_arg:
Cython/Compiler/FlowControl.py:            # TODO: starred args entries are 
not marked with is_arg flag
Cython/Compiler/FlowControl.py:                if assmt.is_arg:
Cython/Compiler/FlowControl.py:                    is_arg = True
Cython/Compiler/FlowControl.py:                is_arg = False
Cython/Compiler/FlowControl.py:            if is_arg:
Cython/Compiler/Symtab.py:    # is_arg           boolean    Is the arg of a 
method
Cython/Compiler/Symtab.py:    is_arg = 0
Cython/Compiler/Symtab.py:        entry.is_arg = 1
"""

This doesn't look like it would be wrong (or even just unsafe) to consider 
the unset flag a bug and fix it. Basically, it's almost unused in the 
original sources, all places where the flag is being read currently are 
somewhat recent code that looks reasonable to me and that appear to assume 
that the flag is actually set.

So, I'd say, if anyone wants to properly clean this up, please go ahead and 
do so, but please do it right on the master branch and send it through Jenkins.

Stefan

From vitja.makarov at gmail.com  Fri Aug 12 08:21:53 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Aug 2011 10:21:53 +0400
Subject: [Cython] Cython bug ? - Entry.is_arg flag not set for all
	arguments
In-Reply-To: <4E44A2EA.8010700@behnel.de>
References: <20110812011901.GA9596@ubuntu>
	<4E44A2EA.8010700@behnel.de>
Message-ID: <CAKGHGPSLaP59SHkkdqrmg=khHJxwRpxkaHDqsyZF65nwDu54eg@mail.gmail.com>

2011/8/12 Stefan Behnel <stefan_ml at behnel.de>:
> [fixed subject]
>
> Romain Guillebert, 12.08.2011 03:19:
>>
>> I tried to compiled Demos/primes.pyx using the ctypes backend and I
>> think I've found a bug :
>> The Entry for the kmax parameter does not set is_arg to 1. However if I
>> turn the def into a cdef, it works fine.
>>
>> Is this a bug or a known hack ?
>
> Vitja already brought this up, too:
>
> http://article.gmane.org/gmane.comp.python.cython.devel/12385
>
> A quick grep on the sources gives me this:
>
> """
> Cython/Compiler/Buffer.py: ? ? ? ? ? ?if entry.is_arg:
> Cython/Compiler/Buffer.py: ? ? ? ? ? ? ? ?if entry.is_arg:
> Cython/Compiler/ExprNodes.py: ? ? ? ?return entry and (entry.is_local or
> entry.is_arg) and not entry.in_closure
> Cython/Compiler/FlowControl.py: ? ? ? ?return (entry.is_local or
> entry.is_pyclass_attr or entry.is_arg or
> Cython/Compiler/FlowControl.py: ? ? ? ?self.is_arg = False
> Cython/Compiler/FlowControl.py: ? ? ? ?self.is_arg = True
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?if assmt.is_arg:
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ?# TODO: starred args entries are
> not marked with is_arg flag
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?if assmt.is_arg:
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ? ? ?is_arg = True
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?is_arg = False
> Cython/Compiler/FlowControl.py: ? ? ? ? ? ?if is_arg:
> Cython/Compiler/Symtab.py: ? ?# is_arg ? ? ? ? ? boolean ? ?Is the arg of a
> method
> Cython/Compiler/Symtab.py: ? ?is_arg = 0
> Cython/Compiler/Symtab.py: ? ? ? ?entry.is_arg = 1
> """
>
> This doesn't look like it would be wrong (or even just unsafe) to consider
> the unset flag a bug and fix it. Basically, it's almost unused in the
> original sources, all places where the flag is being read currently are
> somewhat recent code that looks reasonable to me and that appear to assume
> that the flag is actually set.
>
> So, I'd say, if anyone wants to properly clean this up, please go ahead and
> do so, but please do it right on the master branch and send it through
> Jenkins.
>

Yeah, that would be really nice if all args including starred ones
will have is_arg attribute set.

-- 
vitja.

From vitja.makarov at gmail.com  Fri Aug 12 08:49:34 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Aug 2011 10:49:34 +0400
Subject: [Cython] Yet another Python to C compiler
Message-ID: <CAKGHGPSyQsjstrC=fEV5B5AQr3=2V_hrXGZVYt4JokYFsWo74Q@mail.gmail.com>

Hi!

Recently I've found one more Python to C compiler, that translates
python bytecode into C source.
And author says about 100% Python compatibility. The project is a
signle 800KB python file.

http://code.google.com/p/2c-python/

I was wondering when found that 2c beats Cython in some benchmarks.
For instance, it's about 2 times faster than Cython in pystone test
(pure python mode).

Think we should investigate performance differences and make cython faster.

-- 
vitja.

From vitja.makarov at gmail.com  Fri Aug 12 09:44:46 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Aug 2011 11:44:46 +0400
Subject: [Cython] Yet another Python to C compiler
In-Reply-To: <CAKGHGPSyQsjstrC=fEV5B5AQr3=2V_hrXGZVYt4JokYFsWo74Q@mail.gmail.com>
References: <CAKGHGPSyQsjstrC=fEV5B5AQr3=2V_hrXGZVYt4JokYFsWo74Q@mail.gmail.com>
Message-ID: <CAKGHGPSQAPAA8YoAr3QCeRo+1Q132-x+D-xV4DyoFpPveGzuDw@mail.gmail.com>

2011/8/12 Vitja Makarov <vitja.makarov at gmail.com>:
> Hi!
>
> Recently I've found one more Python to C compiler, that translates
> python bytecode into C source.
> And author says about 100% Python compatibility. The project is a
> signle 800KB python file.
>
> http://code.google.com/p/2c-python/
>
> I was wondering when found that 2c beats Cython in some benchmarks.
> For instance, it's about 2 times faster than Cython in pystone test
> (pure python mode).
>
> Think we should investigate performance differences and make cython faster.
>

Performance boost is because of --direct-call flag.
With --no-direct-call flag performance is about the same.

-- 
vitja.

From stefan_ml at behnel.de  Fri Aug 12 10:17:07 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Aug 2011 10:17:07 +0200
Subject: [Cython] Yet another Python to C compiler
In-Reply-To: <CAKGHGPSyQsjstrC=fEV5B5AQr3=2V_hrXGZVYt4JokYFsWo74Q@mail.gmail.com>
References: <CAKGHGPSyQsjstrC=fEV5B5AQr3=2V_hrXGZVYt4JokYFsWo74Q@mail.gmail.com>
Message-ID: <4E44E183.9040409@behnel.de>

Vitja Makarov, 12.08.2011 08:49:
> Recently I've found one more Python to C compiler, that translates
> python bytecode into C source.
> And author says about 100% Python compatibility.

That's clearly incorrect. From the code, it appears that at least the 
builtins are static, which means that it's *not* Python compatible (even 
less so than Cython currently is). I'm also not sure about the integer type 
handling - worth some investigation, but it looks somewhat sloppy.

I mean, it's well known that you can generate fast code by diverting from 
Python semantics. Shedskin excels in that. They mostly seem to be using the 
same tricks that Cython uses as well, namely static builtins, as well as 
optimistic optimisations for likely types based on usage patterns. And a 
huge amount of special casing, maybe even more than Cython currently applies.

They also seem to make excessive use of CPython internals and internal 
APIs. Not sure if that's a good idea. Specifically, they didn't care a bit 
about Python 3 compatibility, it seems.


> The project is a signle 800KB python file.

That's just plain sick.


> http://code.google.com/p/2c-python/
>
> I was wondering when found that 2c beats Cython in some benchmarks.
> For instance, it's about 2 times faster than Cython in pystone test

PyStone is known to be a particularly bad benchmark.

The other benchmark results are somewhat surprising and (IMHO) hint mostly 
at a lack of Python compatibility. Again, it's well known that you can make 
specific benchmarks fast by diverting from Python semantics in general.

For example, Cython runs richards.py ~70% faster, whereas they claim ~90%. 
Cython is ~50% faster on slowpickle, they claim ~80%. Not really that much 
of a difference actually, and easily achieved by tuning the language 
semantics to the code.


> Think we should investigate performance differences and make cython faster.

You could start by contacting the authors. From the project site, it 
appears that it's basically Russian(?)-only.

My guess is that they simply use more special casing and slightly better 
type inference than Cython currently does. Look at these, for example:

http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#15583

http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#15869

They infer some more return types of builtin methods and their compiler 
knows about some stdlib modules (such as math):

http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#578

Overriding external modules statically means diverting from Python 
semantics. Cython would want to require user interaction for this, e.g. an 
explicit external .pxd file.

Basically, I think that Cython could do a lot better with control flow 
driven type inference. Another thing is that it would be nice to extend the 
type system so that it knows about data types in Python containers.

What we should definitely do is to use Mark's fused types for 
optimisations, e.g. when default arguments hint at a specific input type, 
or even just when we find a function call inside the module with a specific 
combination of input types.

Also, I would expect that eventually optimising the CyFunction type would 
give us another bit of performance.

Stefan

From stefan_ml at behnel.de  Fri Aug 12 14:45:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Aug 2011 14:45:45 +0200
Subject: [Cython] [cython-users] Calling gil-requiring function not
 allowed without gil
In-Reply-To: <4E44CD52.5040504@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no>
Message-ID: <4E452079.9010204@behnel.de>

[second try in moving this discussion to cython-devel]

Dag Sverre Seljebotn, 12.08.2011 08:50:
> On 08/12/2011 06:44 AM, Robert Bradshaw wrote:
>> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 08/11/2011 02:13 PM, Stefan Behnel wrote:
>>>>
>>>> Dag Sverre Seljebotn, 11.08.2011 13:58:
>>>>>
>>>>> On 08/11/2011 01:43 PM, Ting Zhou wrote:
>>>>>>
>>>>>> here is my pyx file. When I compile it, it says "Calling gil-requiring
>>>>>> function not allowed without gil" at calling dabs(x[i]). Why my dabs
>>>>>> function is a gil-requiring function?
>>>>>>
>>>>>> ====================================
>>>>>> from cython.parallel import *
>>>>>> import numpy as np
>>>>>> cimport numpy as np
>>>>>> cimport cython
>>>>>>
>>>>>> cdef double dabs(double x):
>>>>>
>>>>> This should be
>>>>>
>>>>> cdef double dabs(double x) nogil:
>>>>
>>>> Note that Cython cannot infer this automatically. Even a trivial
>>>> function may require an exclusive global lock for some reason, and it's
>>>> common to use the GIL for that. So the programmer must be explicit here.
>>>
>>> Are you still against this mini-CEP?:
>>>
>>> with cython.global_lock():
>>>     ...
>>>
>>> Where global_lock() is GIL-requiring noop.
>>
>> Just reading this, it's not immediately obvious what this means. (I
>> thought at first this was different syntax for "with gil"...)
>
> True. "cython.synchronized"? The synchronized keyword in Java does not
> quite the same thing but almost.
>
>>> This
>>>
>>> a) Improves code readability vastly. Having a critical section take effect
>>> because of the *lack* of a keyword is just very odd to anyone who's not
>>> shoulder deep in CPython internals
>>
>> I'm not following you here. The only way to run into this is if you
>> have explicitly release it. Presumably you can learn both keywords at
>> the same time. Perhaps extern function could be nogil by default.
>
> I'll try to explain again. Consider code like this:
>
> cdef call_c():
>     magic_c_function(3)
>
> This code is perfectly fine even if magic_c_function is not reentrant --
> but that's hardly well documented! So it's conceivable that another
> programmer comes along (who doesn't know the C library well) and decides
> that "somebody has just been too lazy to add the nogil specifier" and slaps
> it on -- at which point you have a bug.
>
> This is more to the point in creating readable code IMO:
>
> cdef call_c():
>     with cython.synchronized():
>         magic_c_function(3)

I think I'm as confused as Robert here. Is that the GIL or some separate lock?

If the latter, where would that lock be stored? Module-wide? While there 
are certainly use cases for that, I think it's much more common to either 
a) use the GIL or b) use an explicit lock at some well defined point, e.g. 
at an object or class level.

I don't think we disagree when I say that locking should be explicit, but 
that includes the "thing" that keeps the lock during its lifetime.


> Also, consider your classical race conditions:
>
> cdef init():
>     global resource
>     if resource == NULL:
>         resource = malloc(sizeof(resource_t)) ...
>
> This is safe code.

Well, it's safe code as long as there is no Python code interaction in 
between. Calling back into the interpreter may trigger a thread switch.

Admittedly, this is not obvious and tends to introduce bugs. I learned that 
the hard way in lxml, actually twice.

If your "synchronised" directive refers to the GIL, then it would suffer 
from that problem as well. I think that's very undesirable for an explicit 
critical section statement.

Stefan

From d.s.seljebotn at astro.uio.no  Fri Aug 12 15:13:45 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Aug 2011 15:13:45 +0200
Subject: [Cython] [cython-users] Calling gil-requiring function not
 allowed without gil
In-Reply-To: <4E452079.9010204@behnel.de>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
Message-ID: <4E452709.6000404@astro.uio.no>

On 08/12/2011 02:45 PM, Stefan Behnel wrote:
 > [second try in moving this discussion to cython-devel]
 >
 > Dag Sverre Seljebotn, 12.08.2011 08:50:
 >> On 08/12/2011 06:44 AM, Robert Bradshaw wrote:
 >>> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn
 >>> <d.s.seljebotn at astro.uio.no> wrote:
 >>>> Are you still against this mini-CEP?:
 >>>>
 >>>> with cython.global_lock():
 >>>> ...
 >>>>
 >>>> Where global_lock() is GIL-requiring noop.
 >>>
 >>> Just reading this, it's not immediately obvious what this means. (I
 >>> thought at first this was different syntax for "with gil"...)
 >>
 >> True. "cython.synchronized"? The synchronized keyword in Java does not
 >> quite the same thing but almost.
 >>
 >>>> This
 >>>>
 >>>> a) Improves code readability vastly. Having a critical section take
 >>>> effect
 >>>> because of the *lack* of a keyword is just very odd to anyone 
who's not
 >>>> shoulder deep in CPython internals
 >>>
 >>> I'm not following you here. The only way to run into this is if you
 >>> have explicitly release it. Presumably you can learn both keywords at
 >>> the same time. Perhaps extern function could be nogil by default.
 >>
 >> I'll try to explain again. Consider code like this:
 >>
 >> cdef call_c():
 >> magic_c_function(3)
 >>
 >> This code is perfectly fine even if magic_c_function is not reentrant --
 >> but that's hardly well documented! So it's conceivable that another
 >> programmer comes along (who doesn't know the C library well) and decides
 >> that "somebody has just been too lazy to add the nogil specifier" and
 >> slaps
 >> it on -- at which point you have a bug.
 >>
 >> This is more to the point in creating readable code IMO:
 >>
 >> cdef call_c():
 >> with cython.synchronized():
 >> magic_c_function(3)
 >
 > I think I'm as confused as Robert here. Is that the GIL or some separate
 > lock?
 >
 > If the latter, where would that lock be stored? Module-wide? While there
 > are certainly use cases for that, I think it's much more common to
 > either a) use the GIL or b) use an explicit lock at some well defined
 > point, e.g. at an object or class level.

I intended it to be the GIL in current CPython. In GIL-less environments 
(whether Java, .NET or some future CPython) one would manuall ensure a 
fully global lock (inserting a _cython module in sys.path etc.).

But:

 > I don't think we disagree when I say that locking should be explicit,
 > but that includes the "thing" that keeps the lock during its lifetime.
 >
 >
 >> Also, consider your classical race conditions:
 >>
 >> cdef init():
 >> global resource
 >> if resource == NULL:
 >> resource = malloc(sizeof(resource_t)) ...
 >>
 >> This is safe code.
 >
 > Well, it's safe code as long as there is no Python code interaction in
 > between. Calling back into the interpreter may trigger a thread switch.
 >
 > Admittedly, this is not obvious and tends to introduce bugs. I learned
 > that the hard way in lxml, actually twice.
 >
 > If your "synchronised" directive refers to the GIL, then it would suffer
 > from that problem as well. I think that's very undesirable for an
 > explicit critical section statement.

This is a good point.

Hmm. This all started with your comment "Even a trivial function may 
require an exclusive global lock for some reason, and it's common to use 
the GIL for that. So the programmer must be explicit here."

And my problem is that it's too implicit -- when you /require/ the 
global lock, you /leave out/ the nogil modifier.

I just wanted to find some way about being explicit about what you 
already do. So it may be a question of naming. As you mention, using the 
GIL has drawbacks, but it performs better than a seperate lock when you 
know there won't be callbacks into Python...

Dag Sverre

From vitja.makarov at gmail.com  Sat Aug 13 06:41:49 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Aug 2011 08:41:49 +0400
Subject: [Cython] Vitja's CyFunction branch
In-Reply-To: <4E344FF4.3060000@behnel.de>
References: <4E33D0EA.9020504@behnel.de>
	<CAKGHGPRNrm4DkwDp15oUu40FeLxU-ciPoYrrOH58Cspr_eS8zA@mail.gmail.com>
	<CANg26EU2sE5e43huaPhjssC2_b_9A2x+YJysCWp1JrO2pMseKA@mail.gmail.com>
	<f5e55d33-5767-4fe9-8bed-0c029298f549@email.android.com>
	<CADiQ+QA-9K6BRRzgPv3Ma2BcgcT2sx_N0UsJA9BbBjX+NPFy+g@mail.gmail.com>
	<4E344FF4.3060000@behnel.de>
Message-ID: <CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>

2011/7/30 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 30.07.2011 18:49:
>>
>> The only reason I haven't pushed a release branch is that last
>> time I did that it kept getting the mainline development pulled into
>> it
>
> That was just an accident on my side when I wasn't aware of the new branch
> you had created. Won't happen again. Just open a new branch when you deem it
> appropriate.
>

0.15 is out. Is it time to merge CyFunction?

ps: I'm afraid of merge button

-- 
vitja.

From stefan_ml at behnel.de  Sat Aug 13 06:58:40 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Aug 2011 06:58:40 +0200
Subject: [Cython] Vitja's CyFunction branch
In-Reply-To: <CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>
References: <4E33D0EA.9020504@behnel.de>	<CAKGHGPRNrm4DkwDp15oUu40FeLxU-ciPoYrrOH58Cspr_eS8zA@mail.gmail.com>	<CANg26EU2sE5e43huaPhjssC2_b_9A2x+YJysCWp1JrO2pMseKA@mail.gmail.com>	<f5e55d33-5767-4fe9-8bed-0c029298f549@email.android.com>	<CADiQ+QA-9K6BRRzgPv3Ma2BcgcT2sx_N0UsJA9BbBjX+NPFy+g@mail.gmail.com>	<4E344FF4.3060000@behnel.de>
	<CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>
Message-ID: <4E460480.1040507@behnel.de>

Vitja Makarov, 13.08.2011 06:41:
> 2011/7/30 Stefan Behnel:
>> Robert Bradshaw, 30.07.2011 18:49:
>>>
>>> The only reason I haven't pushed a release branch is that last
>>> time I did that it kept getting the mainline development pulled into
>>> it
>>
>> That was just an accident on my side when I wasn't aware of the new branch
>> you had created. Won't happen again. Just open a new branch when you deem it
>> appropriate.
>
> 0.15 is out. Is it time to merge CyFunction?
>
> ps: I'm afraid of merge button

There were no objections to merging it, so I hit the button.

Stefan

From vitja.makarov at gmail.com  Sat Aug 13 07:40:11 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Aug 2011 09:40:11 +0400
Subject: [Cython] Vitja's CyFunction branch
In-Reply-To: <4E460480.1040507@behnel.de>
References: <4E33D0EA.9020504@behnel.de>
	<CAKGHGPRNrm4DkwDp15oUu40FeLxU-ciPoYrrOH58Cspr_eS8zA@mail.gmail.com>
	<CANg26EU2sE5e43huaPhjssC2_b_9A2x+YJysCWp1JrO2pMseKA@mail.gmail.com>
	<f5e55d33-5767-4fe9-8bed-0c029298f549@email.android.com>
	<CADiQ+QA-9K6BRRzgPv3Ma2BcgcT2sx_N0UsJA9BbBjX+NPFy+g@mail.gmail.com>
	<4E344FF4.3060000@behnel.de>
	<CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>
	<4E460480.1040507@behnel.de>
Message-ID: <CAKGHGPRJ8ntqzBCxuGJ71gy60Vr3iGBmD7sQAcvGt6rD-CzNdA@mail.gmail.com>

2011/8/13 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 13.08.2011 06:41:
>>
>> 2011/7/30 Stefan Behnel:
>>>
>>> Robert Bradshaw, 30.07.2011 18:49:
>>>>
>>>> The only reason I haven't pushed a release branch is that last
>>>> time I did that it kept getting the mainline development pulled into
>>>> it
>>>
>>> That was just an accident on my side when I wasn't aware of the new
>>> branch
>>> you had created. Won't happen again. Just open a new branch when you deem
>>> it
>>> appropriate.
>>
>> 0.15 is out. Is it time to merge CyFunction?
>>
>> ps: I'm afraid of merge button
>
> There were no objections to merging it, so I hit the button.
>

Thanks!

To be clear: we see increase in pyregr tests beacause of special hack
in runtests.py that enables binding for pyregr. And of course it's
disabled by default.


-- 
vitja.

From d.s.seljebotn at astro.uio.no  Sat Aug 13 13:09:25 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Aug 2011 13:09:25 +0200
Subject: [Cython] Cython, Go and static duck typing
Message-ID: <4E465B65.9020500@astro.uio.no>

I read up on Go's interface model last week and am very excited about 
it. Perhaps this was old news to a lot of you, but it was new to me.

This post is mostly just to make sure everybody is aware of prior art 
here in case it is a good fit for Cython down the road. I hope I'm not 
starting a long wasteful thread, let's keep it FYI, and if it's 
interesting I can try to learn a bit more of the technical details and 
write up a CEP prior to next summer's GSoC or something...

The idea of polymorphism in Go might be phrased in the Cython language as:

cdef interface Walker:
     int walk(float)

cdef class Foo: # note: Does not inherit from Walker!
     cdef int walk(self, float length):
         ....


def f(Walker obj):
     obj.walk(2.3)

f(Foo())

This is duck typing, in the sense that Foo has no knowledge of the 
Walker interface (at module compilation time) -- it just satisfies it by 
implementing the same interface. The point of casting/conversion has all 
the information needed to build the vtable (somehow...). This of course 
uses a little more memory, but it is still bounded by the number of code 
lines you could write...

This has a lot more flexibility (libraries don't have to agree on which 
one should have the "ultimate superclass", they can just agree on method 
names), and it just fits my brain and workflow much better as well: I'd 
use profiling to pinpoint the spots where I really need a vtable instead 
of dict lookups, and then it is more natural to introduce the interface 
in the caller, rather than to go back into the callee to introduce a 
type hierarchy (which might be more geared towards optimizing calls than 
towards what type relation is "natural").

Implementing this in Cython is likely going to be tougher than in Go if 
we want to support <Walker>obj, where obj has type "object", and not 
only <Walker><Foo>obj. But I'd think it is not impossible, just somewhat 
more costly on assignment...but that is often amortized over multiple calls.

Dag Sverre

From stefan_ml at behnel.de  Sat Aug 13 14:28:56 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Aug 2011 14:28:56 +0200
Subject: [Cython] Cython, Go and static duck typing
In-Reply-To: <4E465B65.9020500@astro.uio.no>
References: <4E465B65.9020500@astro.uio.no>
Message-ID: <4E466E08.3020809@behnel.de>

Dag Sverre Seljebotn, 13.08.2011 13:09:
> I read up on Go's interface model last week and am very excited about it.
> Perhaps this was old news to a lot of you, but it was new to me.
>
> This post is mostly just to make sure everybody is aware of prior art here
> in case it is a good fit for Cython down the road. I hope I'm not starting
> a long wasteful thread, let's keep it FYI, and if it's interesting I can
> try to learn a bit more of the technical details and write up a CEP prior
> to next summer's GSoC or something...
>
> The idea of polymorphism in Go might be phrased in the Cython language as:
>
> cdef interface Walker:
>     int walk(float)
>
> cdef class Foo: # note: Does not inherit from Walker!
>     cdef int walk(self, float length):
>         ....
>
>
> def f(Walker obj):
>     obj.walk(2.3)
>
> f(Foo())
>
> This is duck typing, in the sense that Foo has no knowledge of the Walker
> interface (at module compilation time) -- it just satisfies it by
> implementing the same interface. The point of casting/conversion has all
> the information needed to build the vtable (somehow...). This of course
> uses a little more memory, but it is still bounded by the number of code
> lines you could write...
>
> This has a lot more flexibility (libraries don't have to agree on which one
> should have the "ultimate superclass", they can just agree on method
> names), and it just fits my brain and workflow much better as well: I'd use
> profiling to pinpoint the spots where I really need a vtable instead of
> dict lookups, and then it is more natural to introduce the interface in the
> caller, rather than to go back into the callee to introduce a type
> hierarchy (which might be more geared towards optimizing calls than towards
> what type relation is "natural").
>
> Implementing this in Cython is likely going to be tougher than in Go if we
> want to support <Walker>obj, where obj has type "object", and not only
> <Walker><Foo>obj. But I'd think it is not impossible, just somewhat more
> costly on assignment...but that is often amortized over multiple calls.

I like that. However, I also agree that it will be tricky to implement in 
C. Think of this case:

cdef class X:
     cdef int walk(self, float length):
         ...

cdef class Y:
     pass

cdef class Z(Y):
     cdef int walk(self, float length):

Here, the method is in two different places of the vtable in both cases, so 
the necessary access code in C would need to be different.

Go has the advantage of using static typing, so the type of the values that 
is being passed into the interface variable is always known. It may not be 
known in Cython or Python.

Stefan

From d.s.seljebotn at astro.uio.no  Sat Aug 13 16:25:39 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Aug 2011 16:25:39 +0200
Subject: [Cython] Cython, Go and static duck typing
In-Reply-To: <4E466E08.3020809@behnel.de>
References: <4E465B65.9020500@astro.uio.no> <4E466E08.3020809@behnel.de>
Message-ID: <4E468963.8040303@astro.uio.no>

On 08/13/2011 02:28 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 13.08.2011 13:09:
>> I read up on Go's interface model last week and am very excited about it.
>> Perhaps this was old news to a lot of you, but it was new to me.
>>
>> This post is mostly just to make sure everybody is aware of prior art
>> here
>> in case it is a good fit for Cython down the road. I hope I'm not
>> starting
>> a long wasteful thread, let's keep it FYI, and if it's interesting I can
>> try to learn a bit more of the technical details and write up a CEP prior
>> to next summer's GSoC or something...
>>
>> The idea of polymorphism in Go might be phrased in the Cython language
>> as:
>>
>> cdef interface Walker:
>> int walk(float)
>>
>> cdef class Foo: # note: Does not inherit from Walker!
>> cdef int walk(self, float length):
>> ....
>>
>>
>> def f(Walker obj):
>> obj.walk(2.3)
>>
>> f(Foo())
>>
>> This is duck typing, in the sense that Foo has no knowledge of the Walker
>> interface (at module compilation time) -- it just satisfies it by
>> implementing the same interface. The point of casting/conversion has all
>> the information needed to build the vtable (somehow...). This of course
>> uses a little more memory, but it is still bounded by the number of code
>> lines you could write...
>>
>> This has a lot more flexibility (libraries don't have to agree on
>> which one
>> should have the "ultimate superclass", they can just agree on method
>> names), and it just fits my brain and workflow much better as well:
>> I'd use
>> profiling to pinpoint the spots where I really need a vtable instead of
>> dict lookups, and then it is more natural to introduce the interface
>> in the
>> caller, rather than to go back into the callee to introduce a type
>> hierarchy (which might be more geared towards optimizing calls than
>> towards
>> what type relation is "natural").
>>
>> Implementing this in Cython is likely going to be tougher than in Go
>> if we
>> want to support <Walker>obj, where obj has type "object", and not only
>> <Walker><Foo>obj. But I'd think it is not impossible, just somewhat more
>> costly on assignment...but that is often amortized over multiple calls.
>
> I like that. However, I also agree that it will be tricky to implement
> in C. Think of this case:
>
> cdef class X:
> cdef int walk(self, float length):
> ...
>
> cdef class Y:
> pass
>
> cdef class Z(Y):
> cdef int walk(self, float length):
>
> Here, the method is in two different places of the vtable in both cases,
> so the necessary access code in C would need to be different.

Yup, vtables must be generated on "cast sites" (but see below), and then 
I suppose passing around interface references would pass around two 
pointers, a vtable pointer and the "self" object.

> Go has the advantage of using static typing, so the type of the values
> that is being passed into the interface variable is always known. It may
> not be known in Cython or Python.

True, but I don't think this is a serious obstacle. Here's one scheme:

a) Each interface is given an unique ID. By resolving this at module 
load time it should be sufficient with 32-bit IDs.

b) Keep a hash table for each cdef class, which can be pointed to by the 
object. It'd be our own custom implementation designed for looking up by 
32-bit ints, so no string compares etc.

c) If there's a miss (which happens almost never), one needs to generate 
a vtable from some RTTI of the cdef class. This vtable stays allocated 
for the duration of the process, and is probably used by "cast sites" as 
well (at module load time).

It doesn't have all the details but I think it shows that it should be 
fast enough even without fully static typing.

Dag Sverre

From robertwb at math.washington.edu  Sun Aug 14 07:38:12 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 13 Aug 2011 22:38:12 -0700
Subject: [Cython] Vitja's CyFunction branch
In-Reply-To: <CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>
References: <4E33D0EA.9020504@behnel.de>
	<CAKGHGPRNrm4DkwDp15oUu40FeLxU-ciPoYrrOH58Cspr_eS8zA@mail.gmail.com>
	<CANg26EU2sE5e43huaPhjssC2_b_9A2x+YJysCWp1JrO2pMseKA@mail.gmail.com>
	<f5e55d33-5767-4fe9-8bed-0c029298f549@email.android.com>
	<CADiQ+QA-9K6BRRzgPv3Ma2BcgcT2sx_N0UsJA9BbBjX+NPFy+g@mail.gmail.com>
	<4E344FF4.3060000@behnel.de>
	<CAKGHGPRK8QtgGZ_J59FXx+cPxXf-L6=hJwgDWci7Cwnm5bEovQ@mail.gmail.com>
Message-ID: <CADiQ+QBzRTJqtYvCmFWd1Z5voR1HEJ7OHf8T0-+0t1BPeTfcwA@mail.gmail.com>

On Fri, Aug 12, 2011 at 9:41 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/7/30 Stefan Behnel <stefan_ml at behnel.de>:
>> Robert Bradshaw, 30.07.2011 18:49:
>>>
>>> The only reason I haven't pushed a release branch is that last
>>> time I did that it kept getting the mainline development pulled into
>>> it
>>
>> That was just an accident on my side when I wasn't aware of the new branch
>> you had created. Won't happen again. Just open a new branch when you deem it
>> appropriate.
>>
>
> 0.15 is out. Is it time to merge CyFunction?

+1. (I see you already did.)

- Robert

From robertwb at math.washington.edu  Sun Aug 14 08:18:30 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 13 Aug 2011 23:18:30 -0700
Subject: [Cython] Cython, Go and static duck typing
In-Reply-To: <4E465B65.9020500@astro.uio.no>
References: <4E465B65.9020500@astro.uio.no>
Message-ID: <CADiQ+QAT9L5JY5erzFacfyiv94gasFP+Fb5BFFeEtODapS7Jyg@mail.gmail.com>

On Sat, Aug 13, 2011 at 4:09 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> I read up on Go's interface model last week and am very excited about it.
> Perhaps this was old news to a lot of you, but it was new to me.
>
> This post is mostly just to make sure everybody is aware of prior art here
> in case it is a good fit for Cython down the road. I hope I'm not starting a
> long wasteful thread, let's keep it FYI, and if it's interesting I can try
> to learn a bit more of the technical details and write up a CEP prior to
> next summer's GSoC or something...
>
> The idea of polymorphism in Go might be phrased in the Cython language as:
>
> cdef interface Walker:
> ? ?int walk(float)
>
> cdef class Foo: # note: Does not inherit from Walker!
> ? ?cdef int walk(self, float length):
> ? ? ? ?....
>
>
> def f(Walker obj):
> ? ?obj.walk(2.3)
>
> f(Foo())
>
> This is duck typing, in the sense that Foo has no knowledge of the Walker
> interface (at module compilation time) -- it just satisfies it by
> implementing the same interface. The point of casting/conversion has all the
> information needed to build the vtable (somehow...). This of course uses a
> little more memory, but it is still bounded by the number of code lines you
> could write...
>
> This has a lot more flexibility (libraries don't have to agree on which one
> should have the "ultimate superclass", they can just agree on method names),
> and it just fits my brain and workflow much better as well: I'd use
> profiling to pinpoint the spots where I really need a vtable instead of dict
> lookups, and then it is more natural to introduce the interface in the
> caller, rather than to go back into the callee to introduce a type hierarchy
> (which might be more geared towards optimizing calls than towards what type
> relation is "natural").

These are exactly the kinds of thoughts I had when I first saw Go
interfaces, including wanting to add this idea to Cython :). In order
to take advantage of static typing, could there/should there be an
error/warning for trying to invoke a method not offered by the
interface? (Or would the "yellowness" in cython -a be sufficient for
the careful user.)

> Implementing this in Cython is likely going to be tougher than in Go if we
> want to support <Walker>obj, where obj has type "object", and not only
> <Walker><Foo>obj. But I'd think it is not impossible, just somewhat more
> costly on assignment...but that is often amortized over multiple calls.

I was thinking about caching this data to the class object (say, in
its dict) and grabbing/creating it on assignment/casting. A custom
implementation as described above could perhaps be faster.

- Robert

From vitja.makarov at gmail.com  Sun Aug 14 16:57:25 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 14 Aug 2011 18:57:25 +0400
Subject: [Cython] Cython bug ? - Entry.is_arg flag not set for all
	arguments
In-Reply-To: <CAKGHGPSLaP59SHkkdqrmg=khHJxwRpxkaHDqsyZF65nwDu54eg@mail.gmail.com>
References: <20110812011901.GA9596@ubuntu> <4E44A2EA.8010700@behnel.de>
	<CAKGHGPSLaP59SHkkdqrmg=khHJxwRpxkaHDqsyZF65nwDu54eg@mail.gmail.com>
Message-ID: <CAKGHGPQVSE8P9zGHiHTHPs8az2WBBpkH_8cphqqHTV6OcvQ=hg@mail.gmail.com>

2011/8/12 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/12 Stefan Behnel <stefan_ml at behnel.de>:
>> [fixed subject]
>>
>> Romain Guillebert, 12.08.2011 03:19:
>>>
>>> I tried to compiled Demos/primes.pyx using the ctypes backend and I
>>> think I've found a bug :
>>> The Entry for the kmax parameter does not set is_arg to 1. However if I
>>> turn the def into a cdef, it works fine.
>>>
>>> Is this a bug or a known hack ?
>>
>> Vitja already brought this up, too:
>>
>> http://article.gmane.org/gmane.comp.python.cython.devel/12385
>>
>> A quick grep on the sources gives me this:
>>
>> """
>> Cython/Compiler/Buffer.py: ? ? ? ? ? ?if entry.is_arg:
>> Cython/Compiler/Buffer.py: ? ? ? ? ? ? ? ?if entry.is_arg:
>> Cython/Compiler/ExprNodes.py: ? ? ? ?return entry and (entry.is_local or
>> entry.is_arg) and not entry.in_closure
>> Cython/Compiler/FlowControl.py: ? ? ? ?return (entry.is_local or
>> entry.is_pyclass_attr or entry.is_arg or
>> Cython/Compiler/FlowControl.py: ? ? ? ?self.is_arg = False
>> Cython/Compiler/FlowControl.py: ? ? ? ?self.is_arg = True
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?if assmt.is_arg:
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ?# TODO: starred args entries are
>> not marked with is_arg flag
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?if assmt.is_arg:
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ? ? ?is_arg = True
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ? ? ?is_arg = False
>> Cython/Compiler/FlowControl.py: ? ? ? ? ? ?if is_arg:
>> Cython/Compiler/Symtab.py: ? ?# is_arg ? ? ? ? ? boolean ? ?Is the arg of a
>> method
>> Cython/Compiler/Symtab.py: ? ?is_arg = 0
>> Cython/Compiler/Symtab.py: ? ? ? ?entry.is_arg = 1
>> """
>>
>> This doesn't look like it would be wrong (or even just unsafe) to consider
>> the unset flag a bug and fix it. Basically, it's almost unused in the
>> original sources, all places where the flag is being read currently are
>> somewhat recent code that looks reasonable to me and that appear to assume
>> that the flag is actually set.
>>
>> So, I'd say, if anyone wants to properly clean this up, please go ahead and
>> do so, but please do it right on the master branch and send it through
>> Jenkins.
>>
>
> Yeah, that would be really nice if all args including starred ones
> will have is_arg attribute set.
>

I've fixed the issue
https://github.com/vitek/cython/commits/_is_arg

It passes all tests if everything is ok I'll push that to upstream

-- 
vitja.

From stefan_ml at behnel.de  Sun Aug 14 17:02:19 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Aug 2011 17:02:19 +0200
Subject: [Cython] Problems with decorated methods in cdef classes
Message-ID: <4E47E37B.5010706@behnel.de>

Hi,

I've taken another stab at #593 and changed the way decorators are 
currently evaluated.

http://trac.cython.org/cython_trac/ticket/593

https://github.com/cython/cython/commit/c40ff48f84b5e5841e4e2d2c6dcce3e6494e4c25

We previously had

     @deco
     def func(): pass

turn into this:

     def func(): pass
     func = deco(func)

Note how this binds the name more than once and even looks it up in 
between, which is problematic in class scopes and some other special cases. 
For example, this doesn't work:

class Foo(object):
     @property
     def x(self):
         ...
     @x.setter
     def x(self, value):
         ...

because "x.setter" is looked up *after* binding the second function "x" to 
its method name, thus overwriting the initial property.

The correct way to do it is to create the function object in a temp, pass 
it through the decorator call chain, and then assign whatever result this 
produces to the method name. This works nicely, but it triggered a crash in 
problematic code of another test case, namely "closure_decorators_T478". 
That test does this:

def print_args(func):
     def f(*args, **kwds):
         print "args", args, "kwds", kwds
         return func(*args, **kwds)
     return f

cdef class Num:
     @print_args
     def is_prime(self, bint print_factors=False):
         ...

Now, the problem is that Cython considers "is_prime" to be a method of a 
cdef class, although it actually is not. It's only an arbitrary function 
that happens to be defined inside of a cdef class body and that happens to 
be *called* by a method, namely "f". It now crashes for me because the 
"self" argument is not being passed into is_prime() as a C method argument 
when called by the wrapper function - and that's correct, because it's not 
a method call but a regular function call at that point.

The correct way to fix this is to turn all decorated methods in cdef 
classes into plain functions. However, this has huge drawbacks, especially 
that the first argument ('self') can no longer be typed as the surrounding 
extension type. But, after all, you could do this:

def swap_args(func):
     def f(*args):
         return func(*args[::-1])
     return f

cdef class Num:
     @swap_args
     def is_prime(arg, self):
         ...

I'm not sure what to make of this. Does it make sense to go this route? Or 
does anyone see a way to make this "mostly" work, e.g. by somehow 
restricting cdef classes and their methods? Or should we just add runtime 
checks to prevent bad behaviour of decorators?

Stefan


From stefan_ml at behnel.de  Sun Aug 14 17:37:39 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Aug 2011 17:37:39 +0200
Subject: [Cython] Cython bug ? - Entry.is_arg flag not set for
	all	arguments
In-Reply-To: <CAKGHGPQVSE8P9zGHiHTHPs8az2WBBpkH_8cphqqHTV6OcvQ=hg@mail.gmail.com>
References: <20110812011901.GA9596@ubuntu>
	<4E44A2EA.8010700@behnel.de>	<CAKGHGPSLaP59SHkkdqrmg=khHJxwRpxkaHDqsyZF65nwDu54eg@mail.gmail.com>
	<CAKGHGPQVSE8P9zGHiHTHPs8az2WBBpkH_8cphqqHTV6OcvQ=hg@mail.gmail.com>
Message-ID: <4E47EBC3.3080501@behnel.de>

Vitja Makarov, 14.08.2011 16:57:
>> Yeah, that would be really nice if all args including starred ones
>> will have is_arg attribute set.
>
> I've fixed the issue
> https://github.com/vitek/cython/commits/_is_arg
>
> It passes all tests if everything is ok I'll push that to upstream

Fine with me.

Stefan

From vitja.makarov at gmail.com  Sun Aug 14 17:43:43 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 14 Aug 2011 19:43:43 +0400
Subject: [Cython] Cython bug ? - Entry.is_arg flag not set for all
	arguments
In-Reply-To: <4E47EBC3.3080501@behnel.de>
References: <20110812011901.GA9596@ubuntu> <4E44A2EA.8010700@behnel.de>
	<CAKGHGPSLaP59SHkkdqrmg=khHJxwRpxkaHDqsyZF65nwDu54eg@mail.gmail.com>
	<CAKGHGPQVSE8P9zGHiHTHPs8az2WBBpkH_8cphqqHTV6OcvQ=hg@mail.gmail.com>
	<4E47EBC3.3080501@behnel.de>
Message-ID: <CAKGHGPRu2yysNrON6UYSLY5v1rsR2sJ7g6k6oRFFTDzwPYjrSA@mail.gmail.com>

2011/8/14 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 14.08.2011 16:57:
>>>
>>> Yeah, that would be really nice if all args including starred ones
>>> will have is_arg attribute set.
>>
>> I've fixed the issue
>> https://github.com/vitek/cython/commits/_is_arg
>>
>> It passes all tests if everything is ok I'll push that to upstream
>
> Fine with me.
>

Ok, pushed.


-- 
vitja.

From vitja.makarov at gmail.com  Sun Aug 14 18:55:31 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 14 Aug 2011 20:55:31 +0400
Subject: [Cython] Problems with decorated methods in cdef classes
In-Reply-To: <4E47E37B.5010706@behnel.de>
References: <4E47E37B.5010706@behnel.de>
Message-ID: <CAKGHGPTUFqzwyLiepg=4X0uwPHmKVenp1Oz1EyhgdM+E9bm_7A@mail.gmail.com>

2011/8/14 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> I've taken another stab at #593 and changed the way decorators are currently
> evaluated.
>
> http://trac.cython.org/cython_trac/ticket/593
>
> https://github.com/cython/cython/commit/c40ff48f84b5e5841e4e2d2c6dcce3e6494e4c25
>
> We previously had
>
> ? ?@deco
> ? ?def func(): pass
>
> turn into this:
>
> ? ?def func(): pass
> ? ?func = deco(func)
>
> Note how this binds the name more than once and even looks it up in between,
> which is problematic in class scopes and some other special cases. For
> example, this doesn't work:
>
> class Foo(object):
> ? ?@property
> ? ?def x(self):
> ? ? ? ?...
> ? ?@x.setter
> ? ?def x(self, value):
> ? ? ? ?...
>
> because "x.setter" is looked up *after* binding the second function "x" to
> its method name, thus overwriting the initial property.
>
> The correct way to do it is to create the function object in a temp, pass it
> through the decorator call chain, and then assign whatever result this
> produces to the method name.


I'd prefer removal of assignment synthesis from DefNode and
transformation that injects assignment node.
I hope that could make code cleaner.


> This works nicely, but it triggered a crash in
> problematic code of another test case, namely "closure_decorators_T478".
> That test does this:
>
> def print_args(func):
> ? ?def f(*args, **kwds):
> ? ? ? ?print "args", args, "kwds", kwds
> ? ? ? ?return func(*args, **kwds)
> ? ?return f
>
> cdef class Num:
> ? ?@print_args
> ? ?def is_prime(self, bint print_factors=False):
> ? ? ? ?...
>
> Now, the problem is that Cython considers "is_prime" to be a method of a
> cdef class, although it actually is not. It's only an arbitrary function
> that happens to be defined inside of a cdef class body and that happens to
> be *called* by a method, namely "f". It now crashes for me because the
> "self" argument is not being passed into is_prime() as a C method argument
> when called by the wrapper function - and that's correct, because it's not a
> method call but a regular function call at that point.
>
> The correct way to fix this is to turn all decorated methods in cdef classes
> into plain functions. However, this has huge drawbacks, especially that the
> first argument ('self') can no longer be typed as the surrounding extension
> type. But, after all, you could do this:
>
> def swap_args(func):
> ? ?def f(*args):
> ? ? ? ?return func(*args[::-1])
> ? ?return f
>
> cdef class Num:
> ? ?@swap_args
> ? ?def is_prime(arg, self):
> ? ? ? ?...
>
> I'm not sure what to make of this. Does it make sense to go this route? Or
> does anyone see a way to make this "mostly" work, e.g. by somehow
> restricting cdef classes and their methods? Or should we just add runtime
> checks to prevent bad behaviour of decorators?
>

Future CyFunction optimizaition could help here. CyFunction should
accept the following two signatures:

 - self is passed as second (or first?) C-function parameter
 - self is passed inside tuple args

-- 
vitja.

From stefan_ml at behnel.de  Sun Aug 14 19:15:27 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Aug 2011 19:15:27 +0200
Subject: [Cython] Problems with decorated methods in cdef classes
In-Reply-To: <CAKGHGPTUFqzwyLiepg=4X0uwPHmKVenp1Oz1EyhgdM+E9bm_7A@mail.gmail.com>
References: <4E47E37B.5010706@behnel.de>
	<CAKGHGPTUFqzwyLiepg=4X0uwPHmKVenp1Oz1EyhgdM+E9bm_7A@mail.gmail.com>
Message-ID: <4E4802AF.4020108@behnel.de>

Vitja Makarov, 14.08.2011 18:55:
> 2011/8/14 Stefan Behnel<stefan_ml at behnel.de>:
>> Hi,
>>
>> I've taken another stab at #593 and changed the way decorators are currently
>> evaluated.
>>
>> http://trac.cython.org/cython_trac/ticket/593
>>
>> https://github.com/cython/cython/commit/c40ff48f84b5e5841e4e2d2c6dcce3e6494e4c25
>>
>> We previously had
>>
>>     @deco
>>     def func(): pass
>>
>> turn into this:
>>
>>     def func(): pass
>>     func = deco(func)
>>
>> Note how this binds the name more than once and even looks it up in between,
>> which is problematic in class scopes and some other special cases. For
>> example, this doesn't work:
>>
>> class Foo(object):
>>     @property
>>     def x(self):
>>         ...
>>     @x.setter
>>     def x(self, value):
>>         ...
>>
>> because "x.setter" is looked up *after* binding the second function "x" to
>> its method name, thus overwriting the initial property.
>>
>> The correct way to do it is to create the function object in a temp, pass it
>> through the decorator call chain, and then assign whatever result this
>> produces to the method name.
>
> I'd prefer removal of assignment synthesis from DefNode and
> transformation that injects assignment node.
> I hope that could make code cleaner.

Absolutely.


>> This works nicely, but it triggered a crash in
>> problematic code of another test case, namely "closure_decorators_T478".
>> That test does this:
>>
>> def print_args(func):
>>     def f(*args, **kwds):
>>         print "args", args, "kwds", kwds
>>         return func(*args, **kwds)
>>     return f
>>
>> cdef class Num:
>>     @print_args
>>     def is_prime(self, bint print_factors=False):
>>         ...
>>
>> Now, the problem is that Cython considers "is_prime" to be a method of a
>> cdef class, although it actually is not. It's only an arbitrary function
>> that happens to be defined inside of a cdef class body and that happens to
>> be *called* by a method, namely "f". It now crashes for me because the
>> "self" argument is not being passed into is_prime() as a C method argument
>> when called by the wrapper function - and that's correct, because it's not a
>> method call but a regular function call at that point.
>>
>> The correct way to fix this is to turn all decorated methods in cdef classes
>> into plain functions. However, this has huge drawbacks, especially that the
>> first argument ('self') can no longer be typed as the surrounding extension
>> type. But, after all, you could do this:
>>
>> def swap_args(func):
>>     def f(*args):
>>         return func(*args[::-1])
>>     return f
>>
>> cdef class Num:
>>     @swap_args
>>     def is_prime(arg, self):
>>         ...
>>
>> I'm not sure what to make of this. Does it make sense to go this route? Or
>> does anyone see a way to make this "mostly" work, e.g. by somehow
>> restricting cdef classes and their methods? Or should we just add runtime
>> checks to prevent bad behaviour of decorators?
>
> Future CyFunction optimizaition could help here. CyFunction should
> accept the following two signatures:
>
>   - self is passed as second (or first?) C-function parameter
>   - self is passed inside tuple args

Yes, it would simply take the 'self' argument from the Python arguments, 
check its type and raise an exception if it doesn't match the expected 
type. That's what I meant with "runtime checks". This does sound like a 
reasonable compromise between static extension type semantics and general 
Python semantics. After all, if users want general functions, they can 
always define them outside of the class scope.

There are other issues with decorators in cdef classes, though. For 
example, this

cdef class T:
     @staticmethod
     @some_deco
     def func(arg): pass

is supposed to be different from this:

cdef class T:
     @some_deco
     @staticmethod
     def func(arg): pass

But currently, staticmethod and classmethod change func() directly.

Stefan

From vitja.makarov at gmail.com  Mon Aug 15 07:45:52 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 09:45:52 +0400
Subject: [Cython] Class scope lookup order
Message-ID: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>

When creating python-class dict it seems that CPython first looks at
dict then at globals:

A = 1
class X:
    A = A

def y():
    A = 3
    class Y:
        A = A
    return Y
Y = y()

print(X.A, Y.A)

Will print: 1, 1
And not: 1, 3

I didn't find documentation for this but if I'm correct that should be
easy to fix issue in Cython.

Now for this code cython reports compile-time error: local variable
referenced before assignment


-- 
vitja.

From vitja.makarov at gmail.com  Mon Aug 15 07:49:54 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 09:49:54 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
Message-ID: <CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>

2011/8/15 Vitja Makarov <vitja.makarov at gmail.com>:
> When creating python-class dict it seems that CPython first looks at
> dict then at globals:
>
> A = 1
> class X:
> ? ?A = A
>
> def y():
> ? ?A = 3
> ? ?class Y:
> ? ? ? ?A = A
> ? ?return Y
> Y = y()
>
> print(X.A, Y.A)
>
> Will print: 1, 1
> And not: 1, 3
>
> I didn't find documentation for this but if I'm correct that should be
> easy to fix issue in Cython.
>
> Now for this code cython reports compile-time error: local variable
> referenced before assignment
>
>

One more interesting example:

A = 1
def foo(a):
    A = a
    class X:
        a = A
        A = A
    class Y:
        a = A
    return X, Y
X, Y = foo(666)
print X.a, X.A, Y.a

-- 
vitja.

From stefan_ml at behnel.de  Mon Aug 15 11:34:25 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 11:34:25 +0200
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
Message-ID: <4E48E821.6070601@behnel.de>

Vitja Makarov, 15.08.2011 07:49:
> 2011/8/15 Vitja Makarov:
>> When creating python-class dict it seems that CPython first looks at
>> dict then at globals:
>>
>> A = 1
>> class X:
>>     A = A
>>
>> def y():
>>     A = 3
>>     class Y:
>>         A = A
>>     return Y
>> Y = y()
>>
>> print(X.A, Y.A)
>>
>> Will print: 1, 1
>> And not: 1, 3
>>
>> I didn't find documentation for this but if I'm correct that should be
>> easy to fix issue in Cython.
>>
>> Now for this code cython reports compile-time error: local variable
>> referenced before assignment
>
> One more interesting example:
>
> A = 1
> def foo(a):
>      A = a
>      class X:
>          a = A
>          A = A
>      class Y:
>          a = A
>      return X, Y
> X, Y = foo(666)
> print X.a, X.A, Y.a

This prints "1 1 666". However, at least to me, the most surprising thing 
is this:

   >>> A = 1
   >>> def foo(x):
   ...     A = x
   ...     class X:
   ...         a = A
   ...     return X
   ...
   >>> foo(2).a
   2
   >>> def foo(x):
   ...     A = x
   ...     class X:
   ...         A = A
   ...     return X
   ...
   >>> foo(2).A
   1

That looks pretty sick, but I would guess that the 'reasoning' behind this 
is that the second case contains an assignment to A inside of the class 
namespace, and assignments make a variable local to a scope, in this case, 
the function scope. Therefore, the A on the rhs is looked up in that scope 
as well. However, this is just a hand waving guess. I just asked on c.l.py, 
we may get a pointer to relevant documentation that way.

Also see this bug:

http://trac.cython.org/cython_trac/ticket/671

Stefan

From markflorisson88 at gmail.com  Mon Aug 15 11:42:20 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 15 Aug 2011 11:42:20 +0200
Subject: [Cython] [cython-users] Re: Segfault in PyThread_release_lock
In-Reply-To: <878vqved66.fsf@vostro.rath.org>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
Message-ID: <CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>

On 15 August 2011 02:22, Nikolaus Rath <Nikolaus at rath.org> wrote:
> mark florisson <markflorisson88 at gmail.com> writes:
>> On 14 August 2011 17:50, Nikolaus Rath <Nikolaus at rath.org> wrote:
>>> mark florisson <markflorisson88 at gmail.com> writes:
>>>> On 12 August 2011 20:23, Nikolaus Rath <Nikolaus at rath.org> wrote:
>>>>> Hello,
>>>>>
>>>>> The following segfault is completely incomprehensible to me:
>>>>>
>>>>> (gdb) bt full
>>>>> #0 ?sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
>>>>> No locals.
>>>>> #1 ?0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>>>>> ? ? ? ?thelock = 0x0
>>>>> ? ? ? ?status = 0
>>>>> ? ? ? ?error = 0
>>>>> #2 ?0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
>>>>> No locals.
>>>>> #3 ?0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>>>>> ? ? ? ?tstate = 0x925da0
>>>>> #4 ?0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>>>>> ? ?at ../Python/pystate.c:651
>>>>> ? ? ? ?tcur = 0x925da0
>>>>> ? ? ? ?__PRETTY_FUNCTION__ = "PyGILState_Release"
>>>>> [...]
>>>>> Why am I getting an error so deep in the python internals?
>>>>
>>>> The error is deep because Python needs to do certain things. So what
>>>> it's doing here is destructing your thread state, which means that was
>>>> the last call to PyGILState_Ensure(). This means your function was
>>>> called from another thread than a Python thread. It looks to me like
>>>> the GIL is not initialized (__pyx_gilstate_save =
>>>> PyGILState_UNLOCKED). Are you sure you have called
>>>> PyEval_EvalInitThreads()?
>>>
>>> No, I have not. But isn't Cython supposed to do this automatically when
>>> entering the function if I define the function as "with gil"?
>>
>> No, it acquires the GIL, but it does not attempt to initialize it.
>> Seeing how many people have issues with it, I'm thinking that perhaps
>> we should if the module contains with gil functions, or with gil
>> blocks in nogil functions. If you're using cython.parallel and with
>> gil blocks, the GIL will be initialized at module-load time. Note that
>> initializing the GIL slows down serially executed Python code, so
>> that's why Cython doesn't initialize it unconditionally. It will have
>> to be called at module-load time, as it should be called in the main
>> python thread (or from any other python thread started by the
>> thread(ing) module).
>
> Oh, wow. That's a tricky one. And it explains why I have the segfault
> only in a reduced testcase, but not in the full blown application (the
> application probably creates a Python thread before calling into Cython
> code). I banged my head against the wall for quite some time already
> because of that.
>
> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
> to figure that out myself.
>
> I wouldn't mind if Cython inserted the call automatically, but a big
> warning in
> http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
> (under the "acquiring the GIL") might already be enough.
>
>
> Best,
>
> ? -Nikolaus
>
> --
> ??Time flies like an arrow, fruit flies like a Banana.?
>
> ?PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 ?02CF A9AD B7F8 AE4E 425C
>

Definitely, I'll do that one of these days. I do think it's rather
common to have with gil functions called as a callback from another
(non-python) thread. I'm not so sure about with gil blocks in nogil
functions.

@Cython-dev: Do we merely want to update the docs, or do we want to
initialize the GIL for either case, or only for the with gil
functions? I'm not entirely sure what the overhead is for
single-threaded code, but I'd say we need to initialize it for both
cases.

From stefan_ml at behnel.de  Mon Aug 15 11:47:42 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 11:47:42 +0200
Subject: [Cython] Class scope lookup order
In-Reply-To: <4E48E821.6070601@behnel.de>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de>
Message-ID: <4E48EB3E.40009@behnel.de>

Stefan Behnel, 15.08.2011 11:34:
> at least to me, the most surprising thing is this:
>
>  >>> A = 1
>  >>> def foo(x):
>  ...   A = x
>  ...   class X:
>  ...     a = A
>  ...   return X
>  ...
>  >>> foo(2).a
>  2
>  >>> def foo(x):
>  ...   A = x
>  ...   class X:
>  ...     A = A
>  ...   return X
>  ...
>  >>> foo(2).A
>  1
>
> That looks pretty sick, but I would guess that the 'reasoning' behind this
> is that the second case contains an assignment to A inside of the class
> namespace, and assignments make a variable local to a scope, in this case,
> the function scope.

Argh, sorry, I interpreted the results the wrong way around. No, I have no 
idea for a serious explanation. Let's see what c.l.py brings up.

Stefan

From d.s.seljebotn at astro.uio.no  Mon Aug 15 11:54:10 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Aug 2011 11:54:10 +0200
Subject: [Cython] [cython-users] Re: Segfault in PyThread_release_lock
In-Reply-To: <CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>	<87ipq0ouvc.fsf@vostro.rath.org>	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
Message-ID: <4E48ECC2.3000806@astro.uio.no>

On 08/15/2011 11:42 AM, mark florisson wrote:
> On 15 August 2011 02:22, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>> mark florisson<markflorisson88 at gmail.com>  writes:
>>> On 14 August 2011 17:50, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>>>> mark florisson<markflorisson88 at gmail.com>  writes:
>>>>> On 12 August 2011 20:23, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>>>>>> Hello,
>>>>>>
>>>>>> The following segfault is completely incomprehensible to me:
>>>>>>
>>>>>> (gdb) bt full
>>>>>> #0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
>>>>>> No locals.
>>>>>> #1  0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>>>>>>         thelock = 0x0
>>>>>>         status = 0
>>>>>>         error = 0
>>>>>> #2  0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
>>>>>> No locals.
>>>>>> #3  0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>>>>>>         tstate = 0x925da0
>>>>>> #4  0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>>>>>>     at ../Python/pystate.c:651
>>>>>>         tcur = 0x925da0
>>>>>>         __PRETTY_FUNCTION__ = "PyGILState_Release"
>>>>>> [...]
>>>>>> Why am I getting an error so deep in the python internals?
>>>>>
>>>>> The error is deep because Python needs to do certain things. So what
>>>>> it's doing here is destructing your thread state, which means that was
>>>>> the last call to PyGILState_Ensure(). This means your function was
>>>>> called from another thread than a Python thread. It looks to me like
>>>>> the GIL is not initialized (__pyx_gilstate_save =
>>>>> PyGILState_UNLOCKED). Are you sure you have called
>>>>> PyEval_EvalInitThreads()?
>>>>
>>>> No, I have not. But isn't Cython supposed to do this automatically when
>>>> entering the function if I define the function as "with gil"?
>>>
>>> No, it acquires the GIL, but it does not attempt to initialize it.
>>> Seeing how many people have issues with it, I'm thinking that perhaps
>>> we should if the module contains with gil functions, or with gil
>>> blocks in nogil functions. If you're using cython.parallel and with
>>> gil blocks, the GIL will be initialized at module-load time. Note that
>>> initializing the GIL slows down serially executed Python code, so
>>> that's why Cython doesn't initialize it unconditionally. It will have
>>> to be called at module-load time, as it should be called in the main
>>> python thread (or from any other python thread started by the
>>> thread(ing) module).
>>
>> Oh, wow. That's a tricky one. And it explains why I have the segfault
>> only in a reduced testcase, but not in the full blown application (the
>> application probably creates a Python thread before calling into Cython
>> code). I banged my head against the wall for quite some time already
>> because of that.
>>
>> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
>> to figure that out myself.
>>
>> I wouldn't mind if Cython inserted the call automatically, but a big
>> warning in
>> http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
>> (under the "acquiring the GIL") might already be enough.
>>
>>
>> Best,
>>
>>    -Nikolaus
>>
>> --
>>   ?Time flies like an arrow, fruit flies like a Banana.?
>>
>>   PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
>>
>
> Definitely, I'll do that one of these days. I do think it's rather
> common to have with gil functions called as a callback from another
> (non-python) thread. I'm not so sure about with gil blocks in nogil
> functions.
>
> @Cython-dev: Do we merely want to update the docs, or do we want to
> initialize the GIL for either case, or only for the with gil
> functions? I'm not entirely sure what the overhead is for
> single-threaded code, but I'd say we need to initialize it for both
> cases.

I'm not sure if I like magic related to having used "with gil" on a 
function signature -- it's a bit too magic and obscure.

I think we should have another directive for this as well as the #ifdef 
macro (that is, the directive controls the default value of the C 
macro), and document that a bit better.

As for the default value of that, here's two proposals:

  Alt. 1) Always initialize the GIL by default.

  Alt. 2) Issue a warning whenever a with-gil function is present unless 
the directive is explicitly set (one way or the other). Perhaps use the 
same warning if "with nogil" is used as well.

I'm in favour of 1), since it would save a lot of frustration among 
users, since alt 2) is a rather minor case only costing a little bit of 
performance (?), and since in many of the cases where the GIL should not 
be initialized, the multithreading support is somewhat likely to turned 
off at the CPython level anyway during compilation (?).

Dag Sverre

From vitja.makarov at gmail.com  Mon Aug 15 11:55:38 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 13:55:38 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <4E48EB3E.40009@behnel.de>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de> <4E48EB3E.40009@behnel.de>
Message-ID: <CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>

2011/8/15 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 15.08.2011 11:34:
>>
>> at least to me, the most surprising thing is this:
>>
>> ?>>> A = 1
>> ?>>> def foo(x):
>> ?... ? A = x
>> ?... ? class X:
>> ?... ? ? a = A
>> ?... ? return X
>> ?...
>> ?>>> foo(2).a
>> ?2
>> ?>>> def foo(x):
>> ?... ? A = x
>> ?... ? class X:
>> ?... ? ? A = A
>> ?... ? return X
>> ?...
>> ?>>> foo(2).A
>> ?1
>>
>> That looks pretty sick, but I would guess that the 'reasoning' behind this
>> is that the second case contains an assignment to A inside of the class
>> namespace, and assignments make a variable local to a scope, in this case,
>> the function scope.
>
> Argh, sorry, I interpreted the results the wrong way around. No, I have no
> idea for a serious explanation. Let's see what c.l.py brings up.
>

I've tried emulate this behaviour in cython here:

https://github.com/vitek/cython/commit/26ec702fd93fb6dc2109a501b52130e5f775c793


Actually I think this is made to support assignments like A = A in class scope.

-- 
vitja.

From markflorisson88 at gmail.com  Mon Aug 15 12:35:17 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 15 Aug 2011 12:35:17 +0200
Subject: [Cython] [cython-users] Segfault in PyThread_release_lock
In-Reply-To: <4E48ECC2.3000806@astro.uio.no>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no>
Message-ID: <CANg26EU4ufkiDTc31xBoKT1p6w4Z5csPZ7Wbp6ju6VWi6_2YSA@mail.gmail.com>

On Monday, 15 August 2011, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/15/2011 11:42 AM, mark florisson wrote:
>
> On 15 August 2011 02:22, Nikolaus Rath<Nikolaus at rath.org> ?wrote:
>
> mark florisson<markflorisson88 at gmail.com> ?writes:
>
> On 14 August 2011 17:50, Nikolaus Rath<Nikolaus at rath.org> ?wrote:
>
> mark florisson<markflorisson88 at gmail.com> ?writes:
>
> On 12 August 2011 20:23, Nikolaus Rath<Nikolaus at rath.org> ?wrote:
>
> Hello,
>
> The following segfault is completely incomprehensible to me:
>
> (gdb) bt full
> #0 ?sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
> No locals.
> #1 ?0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>  ? ? ? ?thelock = 0x0
>  ? ? ? ?status = 0
>  ? ? ? ?error = 0
> #2 ?0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
> No locals.
> #3 ?0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>  ? ? ? ?tstate = 0x925da0
> #4 ?0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>  ? ?at ../Python/pystate.c:651
>  ? ? ? ?tcur = 0x925da0
>  ? ? ? ?__PRETTY_FUNCTION__ = "PyGILState_Release"
> [...]
> Why am I getting an error so deep in the python internals?
>
>
> The error is deep because Python needs to do certain things. So what
> it's doing here is destructing your thread state, which means that was
> the last call to PyGILState_Ensure(). This means your function was
> called from another thread than a Python thread. It looks to me like
> the GIL is not initialized (__pyx_gilstate_save =
> PyGILState_UNLOCKED). Are you sure you have called
> PyEval_EvalInitThreads()?
>
>
> No, I have not. But isn't Cython supposed to do this automatically when
> entering the function if I define the function as "with gil"?
>
>
> No, it acquires the GIL, but it does not attempt to initialize it.
> Seeing how many people have issues with it, I'm thinking that perhaps
> we should if the module contains with gil functions, or with gil
> blocks in nogil functions. If you're using cython.parallel and with
> gil blocks, the GIL will be initialized at module-load time. Note that
> initializing the GIL slows down serially executed Python code, so
> that's why Cython doesn't initialize it unconditionally. It will have
> to be called at module-load time, as it should be called in the main
> python thread (or from any other python thread started by the
> thread(ing) module).
>
>
> Oh, wow. That's a tricky one. And it explains why I have the segfault
> only in a reduced testcase, but not in the full blown application (the
> application probably creates a Python thread before calling into Cython
> code). I banged my head against the wall for quite some time already
> because of that.
>
> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
> to figure that out myself.
>
> I wouldn't mind if Cython inserted the call automatically, but a big
> warning in
> http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
> (under the "acquiring the GIL") might already be enough.
>
>
> Best,
>
>  ? -Nikolaus
>
> --
>  ??Time flies like an arrow, fruit flies like a Banana.?
>
>  ?PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 ?02CF A9AD B7F8 AE4E 425C
>
>
>
> Definitely, I'll do that one of these days. I do think it's rather
> common to have with gil functions called as a callback from another
> (non-python) thread. I'm not so sure about with gil blocks in nogil
> functions.
>
> @Cython-dev: Do we merely want to update the docs, or do we want to
> initialize the GIL for either case, or only for the with gil
> functions? I'm not entirely sure what the overhead is for
> single-threaded code, but I'd say we need to initialize it for both
> cases.
>
>
> I'm not sure if I like magic related to having used "with gil" on a function signature -- it's a bit too magic and obscure.

I don't know, you'll be 100% safe and there's not any really big
drawback (if used for either case).
> I think we should have another directive for this as well as the #ifdef macro (that is, the directive controls the default value of the C macro), and document that a bit better.
>
> As for the default value of that, here's two proposals:
>
> ?Alt. 1) Always initialize the GIL by default.
>
> ?Alt. 2) Issue a warning whenever a with-gil function is present unless the directive is explicitly set (one way or the other). Perhaps use the same warning if "with nogil" is used as well.

'with nogil' is not a problem, releasing and acquiring in the main
thread is safe.

> I'm in favour of 1), since it would save a lot of frustration among users, since alt 2) is a rather minor case only costing a little bit of performance (?), and since in many of the cases where the GIL should not be initialized, the multithreading support is somewhat likely to turned off at the CPython level anyway during compilation (?).
I don't think a lot of users compile without threading support. But
you don't need to initialize the gil if you don't use with gil blocks
in nogil functions or with gil functions, so why always initialize it?
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Mon Aug 15 12:39:48 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Aug 2011 12:39:48 +0200
Subject: [Cython] [cython-users] Segfault in PyThread_release_lock
In-Reply-To: <CANg26EU4ufkiDTc31xBoKT1p6w4Z5csPZ7Wbp6ju6VWi6_2YSA@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>	<87ipq0ouvc.fsf@vostro.rath.org>	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>	<878vqved66.fsf@vostro.rath.org>	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>	<4E48ECC2.3000806@astro.uio.no>
	<CANg26EU4ufkiDTc31xBoKT1p6w4Z5csPZ7Wbp6ju6VWi6_2YSA@mail.gmail.com>
Message-ID: <4E48F774.9080505@astro.uio.no>

On 08/15/2011 12:35 PM, mark florisson wrote:
> On Monday, 15 August 2011, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 08/15/2011 11:42 AM, mark florisson wrote:
>>
>> On 15 August 2011 02:22, Nikolaus Rath<Nikolaus at rath.org>    wrote:
>>
>> mark florisson<markflorisson88 at gmail.com>    writes:
>>
>> On 14 August 2011 17:50, Nikolaus Rath<Nikolaus at rath.org>    wrote:
>>
>> mark florisson<markflorisson88 at gmail.com>    writes:
>>
>> On 12 August 2011 20:23, Nikolaus Rath<Nikolaus at rath.org>    wrote:
>>
>> Hello,
>>
>> The following segfault is completely incomprehensible to me:
>>
>> (gdb) bt full
>> #0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
>> No locals.
>> #1  0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>>          thelock = 0x0
>>          status = 0
>>          error = 0
>> #2  0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
>> No locals.
>> #3  0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>>          tstate = 0x925da0
>> #4  0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>>      at ../Python/pystate.c:651
>>          tcur = 0x925da0
>>          __PRETTY_FUNCTION__ = "PyGILState_Release"
>> [...]
>> Why am I getting an error so deep in the python internals?
>>
>>
>> The error is deep because Python needs to do certain things. So what
>> it's doing here is destructing your thread state, which means that was
>> the last call to PyGILState_Ensure(). This means your function was
>> called from another thread than a Python thread. It looks to me like
>> the GIL is not initialized (__pyx_gilstate_save =
>> PyGILState_UNLOCKED). Are you sure you have called
>> PyEval_EvalInitThreads()?
>>
>>
>> No, I have not. But isn't Cython supposed to do this automatically when
>> entering the function if I define the function as "with gil"?
>>
>>
>> No, it acquires the GIL, but it does not attempt to initialize it.
>> Seeing how many people have issues with it, I'm thinking that perhaps
>> we should if the module contains with gil functions, or with gil
>> blocks in nogil functions. If you're using cython.parallel and with
>> gil blocks, the GIL will be initialized at module-load time. Note that
>> initializing the GIL slows down serially executed Python code, so
>> that's why Cython doesn't initialize it unconditionally. It will have
>> to be called at module-load time, as it should be called in the main
>> python thread (or from any other python thread started by the
>> thread(ing) module).
>>
>>
>> Oh, wow. That's a tricky one. And it explains why I have the segfault
>> only in a reduced testcase, but not in the full blown application (the
>> application probably creates a Python thread before calling into Cython
>> code). I banged my head against the wall for quite some time already
>> because of that.
>>
>> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
>> to figure that out myself.
>>
>> I wouldn't mind if Cython inserted the call automatically, but a big
>> warning in
>> http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
>> (under the "acquiring the GIL") might already be enough.
>>
>>
>> Best,
>>
>>     -Nikolaus
>>
>> --
>>    ?Time flies like an arrow, fruit flies like a Banana.?
>>
>>    PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
>>
>>
>>
>> Definitely, I'll do that one of these days. I do think it's rather
>> common to have with gil functions called as a callback from another
>> (non-python) thread. I'm not so sure about with gil blocks in nogil
>> functions.
>>
>> @Cython-dev: Do we merely want to update the docs, or do we want to
>> initialize the GIL for either case, or only for the with gil
>> functions? I'm not entirely sure what the overhead is for
>> single-threaded code, but I'd say we need to initialize it for both
>> cases.
>>
>>
>> I'm not sure if I like magic related to having used "with gil" on a function signature -- it's a bit too magic and obscure.
>
> I don't know, you'll be 100% safe and there's not any really big
> drawback (if used for either case).

I'm sorry, I didn't think through this enough. Yes, if you're absolutely 
sure that we'll never need the GIL to be initialized if there's no "with 
gil" (one way or the other), then I support your proposal.

(I suppose that if you're calling external C code which does acquire the 
GIL internally, you're outside of Cython's realm of protection :-) )

Dag Sverre



>> I think we should have another directive for this as well as the #ifdef macro (that is, the directive controls the default value of the C macro), and document that a bit better.
>>
>> As for the default value of that, here's two proposals:
>>
>>   Alt. 1) Always initialize the GIL by default.
>>
>>   Alt. 2) Issue a warning whenever a with-gil function is present unless the directive is explicitly set (one way or the other). Perhaps use the same warning if "with nogil" is used as well.
>
> 'with nogil' is not a problem, releasing and acquiring in the main
> thread is safe.
>
>> I'm in favour of 1), since it would save a lot of frustration among users, since alt 2) is a rather minor case only costing a little bit of performance (?), and since in many of the cases where the GIL should not be initialized, the multithreading support is somewhat likely to turned off at the CPython level anyway during compilation (?).
> I don't think a lot of users compile without threading support. But
> you don't need to initialize the gil if you don't use with gil blocks
> in nogil functions or with gil functions, so why always initialize it?


From stefan_ml at behnel.de  Mon Aug 15 12:42:14 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 12:42:14 +0200
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>	<4E48E821.6070601@behnel.de>
	<4E48EB3E.40009@behnel.de>
	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
Message-ID: <4E48F806.8000708@behnel.de>

Vitja Makarov, 15.08.2011 11:55:
> 2011/8/15 Stefan Behnel:
>> Stefan Behnel, 15.08.2011 11:34:
>>>
>>> at least to me, the most surprising thing is this:
>>>
>>>   >>>  A = 1
>>>   >>>  def foo(x):
>>>   ...   A = x
>>>   ...   class X:
>>>   ...     a = A
>>>   ...   return X
>>>   ...
>>>   >>>  foo(2).a
>>>   2
>>>   >>>  def foo(x):
>>>   ...   A = x
>>>   ...   class X:
>>>   ...     A = A
>>>   ...   return X
>>>   ...
>>>   >>>  foo(2).A
>>>   1
>>>
>>> That looks pretty sick, but I would guess that the 'reasoning' behind this
>>> is that the second case contains an assignment to A inside of the class
>>> namespace, and assignments make a variable local to a scope, in this case,
>>> the function scope.
>>
>> Argh, sorry, I interpreted the results the wrong way around.

... or rather used the right reasoning to draw the wrong conclusion. ;)

Ok, so there's an assignment in the second case, which makes A local to the 
class scope, thus overriding the definition in the function scope. Now, 
when it's being looked up at runtime, it's not in the class locals, so the 
lookup falls through to the globals. In the first case, it's not defined in 
the class scope at all, so it's taken from the surrounding function scope 
directly (and statically).


> I've tried emulate this behaviour in cython here:
>
> https://github.com/vitek/cython/commit/26ec702fd93fb6dc2109a501b52130e5f775c793

My feeling is that it should rather be a matter of static analysis to 
figure out which is meant, although I think your change mimics what CPython 
does. I guess this would fix #671, right?


> Actually I think this is made to support assignments like A = A in class scope.

This behaviour wouldn't be strictly required for that goal. My guess is 
that it was just simpler to implement in CPython, because it has less 
special cases. Having to look up a name in a series of outer scopes in the 
case that it happens not to be defined in a class scope yet (although it is 
known to get defined therein at some point), or even remembering the right 
scope where it needs to get looked up, is trickier than just falling 
through to the globals.

Stefan

From vitja.makarov at gmail.com  Mon Aug 15 13:03:02 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 15:03:02 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <4E48F806.8000708@behnel.de>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de> <4E48EB3E.40009@behnel.de>
	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
	<4E48F806.8000708@behnel.de>
Message-ID: <CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>

2011/8/15 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 15.08.2011 11:55:
>>
>> 2011/8/15 Stefan Behnel:
>>>
>>> Stefan Behnel, 15.08.2011 11:34:
>>>>
>>>> at least to me, the most surprising thing is this:
>>>>
>>>> ?>>> ?A = 1
>>>> ?>>> ?def foo(x):
>>>> ?... ? A = x
>>>> ?... ? class X:
>>>> ?... ? ? a = A
>>>> ?... ? return X
>>>> ?...
>>>> ?>>> ?foo(2).a
>>>> ?2
>>>> ?>>> ?def foo(x):
>>>> ?... ? A = x
>>>> ?... ? class X:
>>>> ?... ? ? A = A
>>>> ?... ? return X
>>>> ?...
>>>> ?>>> ?foo(2).A
>>>> ?1
>>>>
>>>> That looks pretty sick, but I would guess that the 'reasoning' behind
>>>> this
>>>> is that the second case contains an assignment to A inside of the class
>>>> namespace, and assignments make a variable local to a scope, in this
>>>> case,
>>>> the function scope.
>>>
>>> Argh, sorry, I interpreted the results the wrong way around.
>
> ... or rather used the right reasoning to draw the wrong conclusion. ;)
>
> Ok, so there's an assignment in the second case, which makes A local to the
> class scope, thus overriding the definition in the function scope. Now, when
> it's being looked up at runtime, it's not in the class locals, so the lookup
> falls through to the globals. In the first case, it's not defined in the
> class scope at all, so it's taken from the surrounding function scope
> directly (and statically).
>
>
>> I've tried emulate this behaviour in cython here:
>>
>>
>> https://github.com/vitek/cython/commit/26ec702fd93fb6dc2109a501b52130e5f775c793
>
> My feeling is that it should rather be a matter of static analysis to figure
> out which is meant, although I think your change mimics what CPython does. I
> guess this would fix #671, right?
>

Yes. And I use CF info in my commit.

>
>> Actually I think this is made to support assignments like A = A in class
>> scope.
>
> This behaviour wouldn't be strictly required for that goal. My guess is that
> it was just simpler to implement in CPython, because it has less special
> cases. Having to look up a name in a series of outer scopes in the case that
> it happens not to be defined in a class scope yet (although it is known to
> get defined therein at some point), or even remembering the right scope
> where it needs to get looked up, is trickier than just falling through to
> the globals.
>

It's documented here:

http://docs.python.org/reference/executionmodel.html#interaction-with-dynamic-features

"""
Names may be resolved in the local and global namespaces of the
caller. Free variables are not resolved in the nearest enclosing
namespace, but in the global namespace.
"""



-- 
vitja.

From vitja.makarov at gmail.com  Mon Aug 15 19:11:22 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 21:11:22 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de> <4E48EB3E.40009@behnel.de>
	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
	<4E48F806.8000708@behnel.de>
	<CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>
Message-ID: <CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>

2011/8/15 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/15 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 15.08.2011 11:55:
>>>
>>> 2011/8/15 Stefan Behnel:
>>>>
>>>> Stefan Behnel, 15.08.2011 11:34:
>>>>>
>>>>> at least to me, the most surprising thing is this:
>>>>>
>>>>> ?>>> ?A = 1
>>>>> ?>>> ?def foo(x):
>>>>> ?... ? A = x
>>>>> ?... ? class X:
>>>>> ?... ? ? a = A
>>>>> ?... ? return X
>>>>> ?...
>>>>> ?>>> ?foo(2).a
>>>>> ?2
>>>>> ?>>> ?def foo(x):
>>>>> ?... ? A = x
>>>>> ?... ? class X:
>>>>> ?... ? ? A = A
>>>>> ?... ? return X
>>>>> ?...
>>>>> ?>>> ?foo(2).A
>>>>> ?1
>>>>>
>>>>> That looks pretty sick, but I would guess that the 'reasoning' behind
>>>>> this
>>>>> is that the second case contains an assignment to A inside of the class
>>>>> namespace, and assignments make a variable local to a scope, in this
>>>>> case,
>>>>> the function scope.
>>>>
>>>> Argh, sorry, I interpreted the results the wrong way around.
>>
>> ... or rather used the right reasoning to draw the wrong conclusion. ;)
>>
>> Ok, so there's an assignment in the second case, which makes A local to the
>> class scope, thus overriding the definition in the function scope. Now, when
>> it's being looked up at runtime, it's not in the class locals, so the lookup
>> falls through to the globals. In the first case, it's not defined in the
>> class scope at all, so it's taken from the surrounding function scope
>> directly (and statically).
>>
>>
>>> I've tried emulate this behaviour in cython here:
>>>
>>>
>>> https://github.com/vitek/cython/commit/26ec702fd93fb6dc2109a501b52130e5f775c793
>>
>> My feeling is that it should rather be a matter of static analysis to figure
>> out which is meant, although I think your change mimics what CPython does. I
>> guess this would fix #671, right?
>>
>
> Yes. And I use CF info in my commit.
>
>>
>>> Actually I think this is made to support assignments like A = A in class
>>> scope.
>>
>> This behaviour wouldn't be strictly required for that goal. My guess is that
>> it was just simpler to implement in CPython, because it has less special
>> cases. Having to look up a name in a series of outer scopes in the case that
>> it happens not to be defined in a class scope yet (although it is known to
>> get defined therein at some point), or even remembering the right scope
>> where it needs to get looked up, is trickier than just falling through to
>> the globals.
>>
>
> It's documented here:
>
> http://docs.python.org/reference/executionmodel.html#interaction-with-dynamic-features
>
> """
> Names may be resolved in the local and global namespaces of the
> caller. Free variables are not resolved in the nearest enclosing
> namespace, but in the global namespace.
> """
>
>

I think I'm done with the issue and if you don't mind I'll push that.

https://github.com/vitek/cython/commit/b4a0493efbae68b30d54f0c58311c71465d57d43

-- 
vitja.

From stefan_ml at behnel.de  Mon Aug 15 19:20:00 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 19:20:00 +0200
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>	<4E48E821.6070601@behnel.de>
	<4E48EB3E.40009@behnel.de>	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>	<4E48F806.8000708@behnel.de>	<CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>
	<CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>
Message-ID: <4E495540.7050405@behnel.de>

Vitja Makarov, 15.08.2011 19:11:
> I think I'm done with the issue and if you don't mind I'll push that.
>
> https://github.com/vitek/cython/commit/b4a0493efbae68b30d54f0c58311c71465d57d43

Could you please use a .py file for the test instead of a .pyx file? It's 
worth comparing the result to CPython automatically. Also, please let the 
test classes inherit from object to make them new style classes in Py2.

Apart from that, the implementation looks good.

Stefan

From vitja.makarov at gmail.com  Mon Aug 15 19:56:42 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 21:56:42 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <4E495540.7050405@behnel.de>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de> <4E48EB3E.40009@behnel.de>
	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
	<4E48F806.8000708@behnel.de>
	<CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>
	<CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>
	<4E495540.7050405@behnel.de>
Message-ID: <CAKGHGPQFuhXzKKTxK=od4NohGXEGvNCfKg_+aFV=-ngPZJdk-Q@mail.gmail.com>

2011/8/15 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 15.08.2011 19:11:
>>
>> I think I'm done with the issue and if you don't mind I'll push that.
>>
>>
>> https://github.com/vitek/cython/commit/b4a0493efbae68b30d54f0c58311c71465d57d43
>
> Could you please use a .py file for the test instead of a .pyx file? It's
> worth comparing the result to CPython automatically. Also, please let the
> test classes inherit from object to make them new style classes in Py2.
>
> Apart from that, the implementation looks good.
>

Here it is:

https://github.com/vitek/cython/commit/6701884a37b982f78f9af423857b0c47c2d76867

CPython NameError is more verbose ''name '%s' is not defined" so I
used ellipsis. Probably we should change name error message someday.

What the difference between old-style and new-style classes for this tests?

-- 
vitja.

From stefan_ml at behnel.de  Mon Aug 15 20:17:37 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 20:17:37 +0200
Subject: [Cython] Class scope lookup order
In-Reply-To: <CAKGHGPQFuhXzKKTxK=od4NohGXEGvNCfKg_+aFV=-ngPZJdk-Q@mail.gmail.com>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>	<4E48E821.6070601@behnel.de>
	<4E48EB3E.40009@behnel.de>	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>	<4E48F806.8000708@behnel.de>	<CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>	<CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>	<4E495540.7050405@behnel.de>
	<CAKGHGPQFuhXzKKTxK=od4NohGXEGvNCfKg_+aFV=-ngPZJdk-Q@mail.gmail.com>
Message-ID: <4E4962C1.30405@behnel.de>

Vitja Makarov, 15.08.2011 19:56:
> 2011/8/15 Stefan Behnel:
>> Vitja Makarov, 15.08.2011 19:11:
>>>
>>> I think I'm done with the issue and if you don't mind I'll push that.
>>>
>>> https://github.com/vitek/cython/commit/b4a0493efbae68b30d54f0c58311c71465d57d43
>>
>> Could you please use a .py file for the test instead of a .pyx file? It's
>> worth comparing the result to CPython automatically. Also, please let the
>> test classes inherit from object to make them new style classes in Py2.
>>
>> Apart from that, the implementation looks good.
>
> Here it is:
>
> https://github.com/vitek/cython/commit/6701884a37b982f78f9af423857b0c47c2d76867

Thanks.


> CPython NameError is more verbose ''name '%s' is not defined"  so I
> used ellipsis.

Note that you can also use something like

    NameError:... 'B' ...


> What the difference between old-style and new-style classes for this tests?

None. That's why I think we should not use old-style classes.

Stefan

From vitja.makarov at gmail.com  Mon Aug 15 20:29:42 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Aug 2011 22:29:42 +0400
Subject: [Cython] Class scope lookup order
In-Reply-To: <4E4962C1.30405@behnel.de>
References: <CAKGHGPTO6M1-vbfdDgwAvLy+_4JGr40GK9J=JGm9iHNp6P5MDA@mail.gmail.com>
	<CAKGHGPQ7szpHX=Ep3GD93gJ6qe4dGd=2ZiZQwCu_TR5mEW6ADA@mail.gmail.com>
	<4E48E821.6070601@behnel.de> <4E48EB3E.40009@behnel.de>
	<CAKGHGPSDZMOwdqwhVS7QxairSvXBy7xqSDO_dQiLFvx9oSiajw@mail.gmail.com>
	<4E48F806.8000708@behnel.de>
	<CAKGHGPQE9zLtS6iAZHv0Ci3NhZd56Pyoatx-j8wFAeAuH+GvBw@mail.gmail.com>
	<CAKGHGPTe_okA0_-V1SOqbJkJy0SC=FUQFMpd0Q2kj=07k4s8qA@mail.gmail.com>
	<4E495540.7050405@behnel.de>
	<CAKGHGPQFuhXzKKTxK=od4NohGXEGvNCfKg_+aFV=-ngPZJdk-Q@mail.gmail.com>
	<4E4962C1.30405@behnel.de>
Message-ID: <CAKGHGPQF9CP3gOVUskgJastbfM0tN-w0qwHKSuROB9uFTmPQzA@mail.gmail.com>

2011/8/15 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 15.08.2011 19:56:
>>
>> 2011/8/15 Stefan Behnel:
>>>
>>> Vitja Makarov, 15.08.2011 19:11:
>>>>
>>>> I think I'm done with the issue and if you don't mind I'll push that.
>>>>
>>>>
>>>> https://github.com/vitek/cython/commit/b4a0493efbae68b30d54f0c58311c71465d57d43
>>>
>>> Could you please use a .py file for the test instead of a .pyx file? It's
>>> worth comparing the result to CPython automatically. Also, please let the
>>> test classes inherit from object to make them new style classes in Py2.
>>>
>>> Apart from that, the implementation looks good.
>>
>> Here it is:
>>
>>
>> https://github.com/vitek/cython/commit/6701884a37b982f78f9af423857b0c47c2d76867
>
> Thanks.
>
>
>> CPython NameError is more verbose ''name '%s' is not defined" ?so I
>> used ellipsis.
>
> Note that you can also use something like
>
> ? NameError:... 'B' ...
>

That should be ...B... look ugly to me but I'll better do that

>
>> What the difference between old-style and new-style classes for this
>> tests?
>
> None. That's why I think we should not use old-style classes.
>

Ok.


-- 
vitja.

From stefan_ml at behnel.de  Mon Aug 15 22:19:30 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Aug 2011 22:19:30 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <4E48ECC2.3000806@astro.uio.no>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>	<87ipq0ouvc.fsf@vostro.rath.org>	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>	<878vqved66.fsf@vostro.rath.org>	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no>
Message-ID: <4E497F52.3040807@behnel.de>

Dag Sverre Seljebotn, 15.08.2011 11:54:
> On 08/15/2011 11:42 AM, mark florisson wrote:
>> @Cython-dev: Do we merely want to update the docs, or do we want to
>> initialize the GIL for either case, or only for the with gil
>> functions? I'm not entirely sure what the overhead is for
>> single-threaded code, but I'd say we need to initialize it for both
>> cases.
>
> I'm not sure if I like magic related to having used "with gil" on a
> function signature -- it's a bit too magic and obscure.

I don't really like this either. The reason why the GIL is not being 
initialised by default is that it has a runtime impact. Just because code 
is aware of threading does not mean it is going to be used with threads. 
Especially library code (which is a common place for using Cython) 
shouldn't impose such an impact on the rest of the runtime.

Stefan

From d.s.seljebotn at astro.uio.no  Mon Aug 15 23:33:44 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Aug 2011 23:33:44 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <4E497F52.3040807@behnel.de>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
Message-ID: <a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>

Would it be horribly expensive to generate a better runtime error, or even initialize the gil on demand? If the gil is not initialized, get the thread ID of the thread calling the callback and check against the thread entering at module initialization time...I don't know whether pythread.h has a cheap way of achieving this.
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Stefan Behnel <stefan_ml at behnel.de> wrote:

Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do we want to >> initialize the GIL for either case, or only for the with gil >> functions? I'm not entirely sure what the overhead is for >> single-threaded code, but I'd say we need to initialize it for both >> cases. > > I'm not sure if I like magic related to having used "with gil" on a > function signature -- it's a bit too magic and obscure. I don't really like this either. The reason why the GIL is not being initialised by default is that it has a runtime impact. Just because code is aware of threading does not mean it is going to be used with threads. Especially library code (which is a common place for using Cython) shouldn't impose such an impact on the rest of the runtime. Stefan_____________________________________________
cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110815/8dd9f9d5/attachment.html>

From robertwb at math.washington.edu  Tue Aug 16 07:52:33 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 15 Aug 2011 22:52:33 -0700
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
Message-ID: <CADiQ+QBOeqxWskvinJu0nRCv6daKr9nTZdR7Wy+A=6_n85dn6Q@mail.gmail.com>

On Mon, Aug 15, 2011 at 2:33 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> Would it be horribly expensive to generate a better runtime error, or even
> initialize the gil on demand? If the gil is not initialized, get the thread
> ID of the thread calling the callback and check against the thread entering
> at module initialization time...I don't know whether pythread.h has a cheap
> way of achieving this.

This seems reasonable. In particular, it wouldn't have an impact on
code not referencing the GIL. This wouldn't solve the issue of making
sure the threads have been registered with the interpreter though,
would it?

> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>
>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>> we want to >> initialize the GIL for either case, or only for the with gil
>> >> functions? I'm not entirely sure what the overhead is for >>
>> single-threaded code, but I'd say we need to initialize it for both >>
>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>> a > function signature -- it's a bit too magic and obscure. I don't really
>> like this either. The reason why the GIL is not being initialised by default
>> is that it has a runtime impact. Just because code is aware of threading
>> does not mean it is going to be used with threads. Especially library code
>> (which is a common place for using Cython) shouldn't impose such an impact
>> on the rest of the runtime. Stefan
>> ________________________________
>> cython-devel mailing list cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From markflorisson88 at gmail.com  Tue Aug 16 11:33:02 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 16 Aug 2011 11:33:02 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <CADiQ+QBOeqxWskvinJu0nRCv6daKr9nTZdR7Wy+A=6_n85dn6Q@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
	<CADiQ+QBOeqxWskvinJu0nRCv6daKr9nTZdR7Wy+A=6_n85dn6Q@mail.gmail.com>
Message-ID: <CANg26EVbLZ5b5h+b+WN1qRoJfp4+Psyi_O6a8urSCeB686Fv6A@mail.gmail.com>

On 16 August 2011 07:52, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Mon, Aug 15, 2011 at 2:33 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> Would it be horribly expensive to generate a better runtime error, or even
>> initialize the gil on demand? If the gil is not initialized, get the thread
>> ID of the thread calling the callback and check against the thread entering
>> at module initialization time...I don't know whether pythread.h has a cheap
>> way of achieving this.
>
> This seems reasonable. In particular, it wouldn't have an impact on
> code not referencing the GIL. This wouldn't solve the issue of making
> sure the threads have been registered with the interpreter though,
> would it?

It is not possible to do it on demand, this has to happen in the main
thread, as the main thread needs to acquire the GIL (and release it
after so many ticks). As for the runtime impact, I'm not convinced
it's so big, I think we should test this. What we could do is abort
with a useful message if the GIL is not initialized in a non-main
thread, perhaps with a link to some documentation shedding some light
on the matter.

>> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>
>>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>>> we want to >> initialize the GIL for either case, or only for the with gil
>>> >> functions? I'm not entirely sure what the overhead is for >>
>>> single-threaded code, but I'd say we need to initialize it for both >>
>>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>>> a > function signature -- it's a bit too magic and obscure. I don't really
>>> like this either. The reason why the GIL is not being initialised by default
>>> is that it has a runtime impact. Just because code is aware of threading
>>> does not mean it is going to be used with threads. Especially library code
>>> (which is a common place for using Cython) shouldn't impose such an impact
>>> on the rest of the runtime. Stefan
>>> ________________________________
>>> cython-devel mailing list cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Aug 16 11:39:33 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 16 Aug 2011 11:39:33 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
Message-ID: <CANg26EXoQK48PtvP3G67PN9Uep6D3ynMqg0VQyHE=QTcdu5kgg@mail.gmail.com>

On 15 August 2011 23:33, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> Would it be horribly expensive to generate a better runtime error, or even
> initialize the gil on demand? If the gil is not initialized, get the thread
> ID of the thread calling the callback and check against the thread entering
> at module initialization time...I don't know whether pythread.h has a cheap
> way of achieving this.

I'm not sure how expensive PyThread_get_thread_ident(), for pthreads
it calls pthread_self() is. So it's either such a call for every with
gil, or initializing the GIL... I'll try to do some tests later today
and report back.

> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>
>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>> we want to >> initialize the GIL for either case, or only for the with gil
>> >> functions? I'm not entirely sure what the overhead is for >>
>> single-threaded code, but I'd say we need to initialize it for both >>
>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>> a > function signature -- it's a bit too magic and obscure. I don't really
>> like this either. The reason why the GIL is not being initialised by default
>> is that it has a runtime impact. Just because code is aware of threading
>> does not mean it is going to be used with threads. Especially library code
>> (which is a common place for using Cython) shouldn't impose such an impact
>> on the rest of the runtime. Stefan
>> ________________________________
>> cython-devel mailing list cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From markflorisson88 at gmail.com  Tue Aug 16 12:49:38 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 16 Aug 2011 12:49:38 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <CANg26EXoQK48PtvP3G67PN9Uep6D3ynMqg0VQyHE=QTcdu5kgg@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
	<CANg26EXoQK48PtvP3G67PN9Uep6D3ynMqg0VQyHE=QTcdu5kgg@mail.gmail.com>
Message-ID: <CANg26EUc4NqZ_C9jmiuD_id=E=tMrRuGYxJAkMSzVZLHLFQcSA@mail.gmail.com>

On 16 August 2011 11:39, mark florisson <markflorisson88 at gmail.com> wrote:
> On 15 August 2011 23:33, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> Would it be horribly expensive to generate a better runtime error, or even
>> initialize the gil on demand? If the gil is not initialized, get the thread
>> ID of the thread calling the callback and check against the thread entering
>> at module initialization time...I don't know whether pythread.h has a cheap
>> way of achieving this.
>
> I'm not sure how expensive PyThread_get_thread_ident(), for pthreads
> it calls pthread_self() is. So it's either such a call for every with
> gil, or initializing the GIL... I'll try to do some tests later today
> and report back.
>
>> --
>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>
>> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>
>>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>>> we want to >> initialize the GIL for either case, or only for the with gil
>>> >> functions? I'm not entirely sure what the overhead is for >>
>>> single-threaded code, but I'd say we need to initialize it for both >>
>>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>>> a > function signature -- it's a bit too magic and obscure. I don't really
>>> like this either. The reason why the GIL is not being initialised by default
>>> is that it has a runtime impact. Just because code is aware of threading
>>> does not mean it is going to be used with threads. Especially library code
>>> (which is a common place for using Cython) shouldn't impose such an impact
>>> on the rest of the runtime. Stefan
>>> ________________________________
>>> cython-devel mailing list cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>

Ok, I attached a little script that does some computation
(naive_convolve from the docs). This is the result:

? python convolve.py
5.61997413635
? python convolve.py --with-gil
5.96432781219

So that mean you get approximately a 6% slowdown with the GIL
initialized. The alternative is that we do a check for every with gil
in Cython code for the thread ident + PyEval_ThreadsInitialized()
(haven't tested how much that will slow down with gil blocks, but I
assume the difference would be nearly negligible).

So I suppose that check + abort would be the way to go?

From markflorisson88 at gmail.com  Tue Aug 16 12:50:40 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 16 Aug 2011 12:50:40 +0200
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <CANg26EUc4NqZ_C9jmiuD_id=E=tMrRuGYxJAkMSzVZLHLFQcSA@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
	<CANg26EXoQK48PtvP3G67PN9Uep6D3ynMqg0VQyHE=QTcdu5kgg@mail.gmail.com>
	<CANg26EUc4NqZ_C9jmiuD_id=E=tMrRuGYxJAkMSzVZLHLFQcSA@mail.gmail.com>
Message-ID: <CANg26EV_NNJ=wm_3LHGGhexoTFgzq5T71DY6SmvzNT5cZLrSNw@mail.gmail.com>

On 16 August 2011 12:49, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 August 2011 11:39, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 15 August 2011 23:33, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> Would it be horribly expensive to generate a better runtime error, or even
>>> initialize the gil on demand? If the gil is not initialized, get the thread
>>> ID of the thread calling the callback and check against the thread entering
>>> at module initialization time...I don't know whether pythread.h has a cheap
>>> way of achieving this.
>>
>> I'm not sure how expensive PyThread_get_thread_ident(), for pthreads
>> it calls pthread_self() is. So it's either such a call for every with
>> gil, or initializing the GIL... I'll try to do some tests later today
>> and report back.
>>
>>> --
>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>
>>> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>
>>>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>>>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>>>> we want to >> initialize the GIL for either case, or only for the with gil
>>>> >> functions? I'm not entirely sure what the overhead is for >>
>>>> single-threaded code, but I'd say we need to initialize it for both >>
>>>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>>>> a > function signature -- it's a bit too magic and obscure. I don't really
>>>> like this either. The reason why the GIL is not being initialised by default
>>>> is that it has a runtime impact. Just because code is aware of threading
>>>> does not mean it is going to be used with threads. Especially library code
>>>> (which is a common place for using Cython) shouldn't impose such an impact
>>>> on the rest of the runtime. Stefan
>>>> ________________________________
>>>> cython-devel mailing list cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>
> Ok, I attached a little script that does some computation
> (naive_convolve from the docs). This is the result:
>
> ? python convolve.py
> 5.61997413635
> ? python convolve.py --with-gil
> 5.96432781219
>
> So that mean you get approximately a 6% slowdown with the GIL
> initialized. The alternative is that we do a check for every with gil
> in Cython code for the thread ident + PyEval_ThreadsInitialized()
> (haven't tested how much that will slow down with gil blocks, but I
> assume the difference would be nearly negligible).
>
> So I suppose that check + abort would be the way to go?
>

Oops, forgot the script.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: convolve.py
Type: application/octet-stream
Size: 1907 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110816/f2ba5569/attachment.obj>

From robertwb at math.washington.edu  Tue Aug 16 20:22:47 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Aug 2011 11:22:47 -0700
Subject: [Cython] Segfault in PyThread_release_lock
In-Reply-To: <CANg26EUc4NqZ_C9jmiuD_id=E=tMrRuGYxJAkMSzVZLHLFQcSA@mail.gmail.com>
References: <87mxfev69a.fsf@inspiron.ap.columbia.edu>
	<CANg26EUDyJBnsZBk9+PS=NzKzuGzzeoZ=+5F=0vjOoRdBodKqA@mail.gmail.com>
	<87ipq0ouvc.fsf@vostro.rath.org>
	<CANg26EWBG_o6AmfqJDmRiQoD3_43Ek1VewRg7LvP8CB55voOGg@mail.gmail.com>
	<878vqved66.fsf@vostro.rath.org>
	<CANg26EU5d+0e5FB_02G7Sp7b0ZicWGg8C6zgv5bvYMHqW6Y11w@mail.gmail.com>
	<4E48ECC2.3000806@astro.uio.no> <4E497F52.3040807@behnel.de>
	<a63c61df-0560-4223-a3b5-7ba5eadc0b45@email.android.com>
	<CANg26EXoQK48PtvP3G67PN9Uep6D3ynMqg0VQyHE=QTcdu5kgg@mail.gmail.com>
	<CANg26EUc4NqZ_C9jmiuD_id=E=tMrRuGYxJAkMSzVZLHLFQcSA@mail.gmail.com>
Message-ID: <CADiQ+QDejpLqB9Fyw+_J4W9Hrpmgy2HnmZ0pNRCAu71O6SxvGg@mail.gmail.com>

On Tue, Aug 16, 2011 at 3:49 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 16 August 2011 11:39, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 15 August 2011 23:33, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> Would it be horribly expensive to generate a better runtime error, or even
>>> initialize the gil on demand? If the gil is not initialized, get the thread
>>> ID of the thread calling the callback and check against the thread entering
>>> at module initialization time...I don't know whether pythread.h has a cheap
>>> way of achieving this.
>>
>> I'm not sure how expensive PyThread_get_thread_ident(), for pthreads
>> it calls pthread_self() is. So it's either such a call for every with
>> gil, or initializing the GIL... I'll try to do some tests later today
>> and report back.
>>
>>> --
>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>
>>> Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>
>>>> Dag Sverre Seljebotn, 15.08.2011 11:54: > On 08/15/2011 11:42 AM, mark
>>>> florisson wrote: >> @Cython-dev: Do we merely want to update the docs, or do
>>>> we want to >> initialize the GIL for either case, or only for the with gil
>>>> >> functions? I'm not entirely sure what the overhead is for >>
>>>> single-threaded code, but I'd say we need to initialize it for both >>
>>>> cases. > > I'm not sure if I like magic related to having used "with gil" on
>>>> a > function signature -- it's a bit too magic and obscure. I don't really
>>>> like this either. The reason why the GIL is not being initialised by default
>>>> is that it has a runtime impact. Just because code is aware of threading
>>>> does not mean it is going to be used with threads. Especially library code
>>>> (which is a common place for using Cython) shouldn't impose such an impact
>>>> on the rest of the runtime. Stefan
>>>> ________________________________
>>>> cython-devel mailing list cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>
> Ok, I attached a little script that does some computation
> (naive_convolve from the docs). This is the result:
>
> ? python convolve.py
> 5.61997413635
> ? python convolve.py --with-gil
> 5.96432781219
>
> So that mean you get approximately a 6% slowdown with the GIL
> initialized.

Thanks for the timings. Yeah, that's too high a price to pay by default.

> The alternative is that we do a check for every with gil
> in Cython code for the thread ident + PyEval_ThreadsInitialized()
> (haven't tested how much that will slow down with gil blocks, but I
> assume the difference would be nearly negligible).

You're already doing work there (by request). If there was
non-negligible overhead we could add a directive to initialize threads
at module load time and skip some of the per-withgil checks.

> So I suppose that check + abort would be the way to go?

Yep.

- Robert

From vitja.makarov at gmail.com  Tue Aug 16 21:54:16 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 16 Aug 2011 23:54:16 +0400
Subject: [Cython] Entry lookup order
Message-ID: <CAKGHGPQDHkNhUwyoQKT0Kx1z2Ysg=R2=BHAeawa6drbQE+jajA@mail.gmail.com>

Trying to silence compiler warning about taking address of variable
I've found again that when CreateControlFlow is ran lhs entries are
resolved and rhs aren't.

As I understand analyse_declarations() creates scope entries for lhs.
At this point we don't know much about scope name reference belongs
to.
Next analyse_expressions() fills the rest entries.

Sine CreateControlFlow couldn't be run after analyse_expressions() is
it correct to lookup for entry in the current scope?

-- 
vitja.

From robertwb at math.washington.edu  Wed Aug 17 08:02:54 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Aug 2011 23:02:54 -0700
Subject: [Cython] Problems with decorated methods in cdef classes
In-Reply-To: <4E47E37B.5010706@behnel.de>
References: <4E47E37B.5010706@behnel.de>
Message-ID: <CADiQ+QDVOmNxwGfOzAdwctCpXaDMXfZ9adqq9GVpL_8AY79VKg@mail.gmail.com>

On Sun, Aug 14, 2011 at 8:02 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I've taken another stab at #593 and changed the way decorators are currently
> evaluated.
>
> http://trac.cython.org/cython_trac/ticket/593
>
> https://github.com/cython/cython/commit/c40ff48f84b5e5841e4e2d2c6dcce3e6494e4c25
>
> We previously had
>
> ? ?@deco
> ? ?def func(): pass
>
> turn into this:
>
> ? ?def func(): pass
> ? ?func = deco(func)
>
> Note how this binds the name more than once and even looks it up in between,
> which is problematic in class scopes and some other special cases. For
> example, this doesn't work:
>
> class Foo(object):
> ? ?@property
> ? ?def x(self):
> ? ? ? ?...
> ? ?@x.setter
> ? ?def x(self, value):
> ? ? ? ?...
>
> because "x.setter" is looked up *after* binding the second function "x" to
> its method name, thus overwriting the initial property.
>
> The correct way to do it is to create the function object in a temp, pass it
> through the decorator call chain, and then assign whatever result this
> produces to the method name. This works nicely, but it triggered a crash in
> problematic code of another test case, namely "closure_decorators_T478".
> That test does this:
>
> def print_args(func):
> ? ?def f(*args, **kwds):
> ? ? ? ?print "args", args, "kwds", kwds
> ? ? ? ?return func(*args, **kwds)
> ? ?return f
>
> cdef class Num:
> ? ?@print_args
> ? ?def is_prime(self, bint print_factors=False):
> ? ? ? ?...
>
> Now, the problem is that Cython considers "is_prime" to be a method of a
> cdef class, although it actually is not. It's only an arbitrary function
> that happens to be defined inside of a cdef class body and that happens to
> be *called* by a method, namely "f". It now crashes for me because the
> "self" argument is not being passed into is_prime() as a C method argument
> when called by the wrapper function - and that's correct, because it's not a
> method call but a regular function call at that point.
>
> The correct way to fix this is to turn all decorated methods in cdef classes
> into plain functions. However, this has huge drawbacks, especially that the
> first argument ('self') can no longer be typed as the surrounding extension
> type. But, after all, you could do this:
>
> def swap_args(func):
> ? ?def f(*args):
> ? ? ? ?return func(*args[::-1])
> ? ?return f
>
> cdef class Num:
> ? ?@swap_args
> ? ?def is_prime(arg, self):
> ? ? ? ?...
>
> I'm not sure what to make of this. Does it make sense to go this route? Or
> does anyone see a way to make this "mostly" work, e.g. by somehow
> restricting cdef classes and their methods? Or should we just add runtime
> checks to prevent bad behaviour of decorators?

I would be happy in making decorated methods into "ordinary"
functions--this will probably play more nicely with many real-world
decorators as well. I say we leave the arguments of a decorated method
completely untyped--the user can type the first argument if need be
(inserting the appropriate runtime check).

Now that we support decorators, perhaps the extra static/classmethod
magic could go away (or at least in the case that there are other
decorators). i suppose CyFunction could have a bit set to emulate
these behaviors for efficiency purposes, but only if it's the
immediate decorator.
(As a side note, it could be valuable and probably not too hard to
support these decorators on cdef methods as well. I'm thinking in
particular of static create* methods that take raw C data.)

- Robert

From robertwb at math.washington.edu  Wed Aug 17 09:12:10 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Aug 2011 00:12:10 -0700
Subject: [Cython] [cython-users] Calling gil-requiring function not
 allowed without gil
In-Reply-To: <4E452709.6000404@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
Message-ID: <CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>

On Fri, Aug 12, 2011 at 6:13 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/12/2011 02:45 PM, Stefan Behnel wrote:
>> [second try in moving this discussion to cython-devel]
>>
>> Dag Sverre Seljebotn, 12.08.2011 08:50:
>>> On 08/12/2011 06:44 AM, Robert Bradshaw wrote:
>>>> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>> Are you still against this mini-CEP?:
>>>>>
>>>>> with cython.global_lock():
>>>>> ...
>>>>>
>>>>> Where global_lock() is GIL-requiring noop.
>>>>
>>>> Just reading this, it's not immediately obvious what this means. (I
>>>> thought at first this was different syntax for "with gil"...)
>>>
>>> True. "cython.synchronized"? The synchronized keyword in Java does not
>>> quite the same thing but almost.
>>>
>>>>> This
>>>>>
>>>>> a) Improves code readability vastly. Having a critical section take
>>>>> effect
>>>>> because of the *lack* of a keyword is just very odd to anyone who's not
>>>>> shoulder deep in CPython internals
>>>>
>>>> I'm not following you here. The only way to run into this is if you
>>>> have explicitly release it. Presumably you can learn both keywords at
>>>> the same time. Perhaps extern function could be nogil by default.
>>>
>>> I'll try to explain again. Consider code like this:
>>>
>>> cdef call_c():
>>> magic_c_function(3)
>>>
>>> This code is perfectly fine even if magic_c_function is not reentrant --
>>> but that's hardly well documented! So it's conceivable that another
>>> programmer comes along (who doesn't know the C library well) and decides
>>> that "somebody has just been too lazy to add the nogil specifier" and
>>> slaps
>>> it on -- at which point you have a bug.
>>>
>>> This is more to the point in creating readable code IMO:
>>>
>>> cdef call_c():
>>> with cython.synchronized():
>>> magic_c_function(3)
>>
>> I think I'm as confused as Robert here. Is that the GIL or some separate
>> lock?
>>
>> If the latter, where would that lock be stored? Module-wide? While there
>> are certainly use cases for that, I think it's much more common to
>> either a) use the GIL or b) use an explicit lock at some well defined
>> point, e.g. at an object or class level.
>
> I intended it to be the GIL in current CPython. In GIL-less environments
> (whether Java, .NET or some future CPython) one would manuall ensure a fully
> global lock (inserting a _cython module in sys.path etc.).
>
> But:
>
>> I don't think we disagree when I say that locking should be explicit,
>> but that includes the "thing" that keeps the lock during its lifetime.
>>
>>
>>> Also, consider your classical race conditions:
>>>
>>> cdef init():
>>> global resource
>>> if resource == NULL:
>>> resource = malloc(sizeof(resource_t)) ...
>>>
>>> This is safe code.
>>
>> Well, it's safe code as long as there is no Python code interaction in
>> between. Calling back into the interpreter may trigger a thread switch.
>>
>> Admittedly, this is not obvious and tends to introduce bugs. I learned
>> that the hard way in lxml, actually twice.
>>
>> If your "synchronised" directive refers to the GIL, then it would suffer
>> from that problem as well. I think that's very undesirable for an
>> explicit critical section statement.
>
> This is a good point.
>
> Hmm. This all started with your comment "Even a trivial function may require
> an exclusive global lock for some reason, and it's common to use the GIL for
> that. So the programmer must be explicit here."
>
> And my problem is that it's too implicit -- when you /require/ the global
> lock, you /leave out/ the nogil modifier.

That's a good point.

> I just wanted to find some way about being explicit about what you already
> do. So it may be a question of naming. As you mention, using the GIL has
> drawbacks, but it performs better than a seperate lock when you know there
> won't be callbacks into Python...

So your proposal is that "with cython.synchronized" has the same
effect as a Python operation, in that its a compile time error to do
it while not holding the gil? (As opposed to "with gil" which actively
acquires the lock (which is more similar to java synchronized)?) Would
this cause the function to have a GIL requiring/acquiring signature? I
suppose the reason that one implicitly holds the gil is that it allows
you to do arbitrary Python operations by default. Having to explicitly
mark when one wants to hold the GIL would be a big regression for
applications that do a lot of Python interaction (e.g. wrappers).

I'm still thinking it could be reasonable for extern functions to be
nogil by default, as a user of C libraries is expected to manually and
externally respect its threading constraints. (This would require a
long deprecation period.)

- Robert

From d.s.seljebotn at astro.uio.no  Wed Aug 17 10:08:03 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Aug 2011 10:08:03 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>
	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
Message-ID: <4E4B76E3.3030808@astro.uio.no>

On 08/17/2011 09:12 AM, Robert Bradshaw wrote:
> On Fri, Aug 12, 2011 at 6:13 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 08/12/2011 02:45 PM, Stefan Behnel wrote:
>>> [second try in moving this discussion to cython-devel]
>>>
>>> Dag Sverre Seljebotn, 12.08.2011 08:50:
>>>> On 08/12/2011 06:44 AM, Robert Bradshaw wrote:
>>>>> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>>>> Are you still against this mini-CEP?:
>>>>>>
>>>>>> with cython.global_lock():
>>>>>> ...
>>>>>>
>>>>>> Where global_lock() is GIL-requiring noop.
>>>>>
>>>>> Just reading this, it's not immediately obvious what this means. (I
>>>>> thought at first this was different syntax for "with gil"...)
>>>>
>>>> True. "cython.synchronized"? The synchronized keyword in Java does not
>>>> quite the same thing but almost.
>>>>
>>>>>> This
>>>>>>
>>>>>> a) Improves code readability vastly. Having a critical section take
>>>>>> effect
>>>>>> because of the *lack* of a keyword is just very odd to anyone who's not
>>>>>> shoulder deep in CPython internals
>>>>>
>>>>> I'm not following you here. The only way to run into this is if you
>>>>> have explicitly release it. Presumably you can learn both keywords at
>>>>> the same time. Perhaps extern function could be nogil by default.
>>>>
>>>> I'll try to explain again. Consider code like this:
>>>>
>>>> cdef call_c():
>>>> magic_c_function(3)
>>>>
>>>> This code is perfectly fine even if magic_c_function is not reentrant --
>>>> but that's hardly well documented! So it's conceivable that another
>>>> programmer comes along (who doesn't know the C library well) and decides
>>>> that "somebody has just been too lazy to add the nogil specifier" and
>>>> slaps
>>>> it on -- at which point you have a bug.
>>>>
>>>> This is more to the point in creating readable code IMO:
>>>>
>>>> cdef call_c():
>>>> with cython.synchronized():
>>>> magic_c_function(3)
>>>
>>> I think I'm as confused as Robert here. Is that the GIL or some separate
>>> lock?
>>>
>>> If the latter, where would that lock be stored? Module-wide? While there
>>> are certainly use cases for that, I think it's much more common to
>>> either a) use the GIL or b) use an explicit lock at some well defined
>>> point, e.g. at an object or class level.
>>
>> I intended it to be the GIL in current CPython. In GIL-less environments
>> (whether Java, .NET or some future CPython) one would manuall ensure a fully
>> global lock (inserting a _cython module in sys.path etc.).
>>
>> But:
>>
>>> I don't think we disagree when I say that locking should be explicit,
>>> but that includes the "thing" that keeps the lock during its lifetime.
>>>
>>>
>>>> Also, consider your classical race conditions:
>>>>
>>>> cdef init():
>>>> global resource
>>>> if resource == NULL:
>>>> resource = malloc(sizeof(resource_t)) ...
>>>>
>>>> This is safe code.
>>>
>>> Well, it's safe code as long as there is no Python code interaction in
>>> between. Calling back into the interpreter may trigger a thread switch.
>>>
>>> Admittedly, this is not obvious and tends to introduce bugs. I learned
>>> that the hard way in lxml, actually twice.
>>>
>>> If your "synchronised" directive refers to the GIL, then it would suffer
>>> from that problem as well. I think that's very undesirable for an
>>> explicit critical section statement.
>>
>> This is a good point.
>>
>> Hmm. This all started with your comment "Even a trivial function may require
>> an exclusive global lock for some reason, and it's common to use the GIL for
>> that. So the programmer must be explicit here."
>>
>> And my problem is that it's too implicit -- when you /require/ the global
>> lock, you /leave out/ the nogil modifier.
>
> That's a good point.
>
>> I just wanted to find some way about being explicit about what you already
>> do. So it may be a question of naming. As you mention, using the GIL has
>> drawbacks, but it performs better than a seperate lock when you know there
>> won't be callbacks into Python...
>
> So your proposal is that "with cython.synchronized" has the same
> effect as a Python operation, in that its a compile time error to do
> it while not holding the gil? (As opposed to "with gil" which actively
> acquires the lock (which is more similar to java synchronized)?) Would

Yes, the former. It does not acquire a lock under curreent CPython.

But the intention is very much to prepare the ground for 
IronPython/Jython/GIL-less CPython/autogil-in-Cython too. Basically a 
tool to protect & document logic that depends on GIL to avoid races, so 
that it is future-safe.

So in Cython-for-Jython it would have to acquire a lock, and be more 
like Java synchronized.

> this cause the function to have a GIL requiring/acquiring signature? I

Not in isolation.

But, IF (after a long deprecation cycle) we start to infer "nogil" on 
function signatures, then the presence of this Python statement would 
stop nogil from getting inferred.

The intention is simply: If you have a cdef function which uses no 
Python operations, but still relies on holding the GIL to avoid race 
conditions (exactly the situation Stefan described initially), then you 
can use cython.synchronized to avoid any future inference from removing 
the GIL.

> suppose the reason that one implicitly holds the gil is that it allows
> you to do arbitrary Python operations by default. Having to explicitly
> mark when one wants to hold the GIL would be a big regression for
> applications that do a lot of Python interaction (e.g. wrappers).

Yes, I'm not proposing that at all.

> I'm still thinking it could be reasonable for extern functions to be
> nogil by default, as a user of C libraries is expected to manually and
> externally respect its threading constraints. (This would require a
> long deprecation period.)

+1.

Perhaps this would be safer if we introduce "expects gil" (explicit 
declaration for neither "nogil" or "with gil"), and then start to emit 
warnings if a GIL-mode isn't declared on cdef functions which has their 
address taken.

Thus, if you pass a callback to a C library, you'll get a reminder to 
declare a GIL mode.

Dag Sverre

From robertwb at math.washington.edu  Wed Aug 17 20:19:59 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Aug 2011 11:19:59 -0700
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4B76E3.3030808@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
	<4E4B76E3.3030808@astro.uio.no>
Message-ID: <CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>

On Wed, Aug 17, 2011 at 1:08 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/17/2011 09:12 AM, Robert Bradshaw wrote:
>> So your proposal is that "with cython.synchronized" has the same
>> effect as a Python operation, in that its a compile time error to do
>> it while not holding the gil? (As opposed to "with gil" which actively
>> acquires the lock (which is more similar to java synchronized)?) Would
>
> Yes, the former. It does not acquire a lock under curreent CPython.
>
> But the intention is very much to prepare the ground for
> IronPython/Jython/GIL-less CPython/autogil-in-Cython too. Basically a tool
> to protect & document logic that depends on GIL to avoid races, so that it
> is future-safe.
>
> So in Cython-for-Jython it would have to acquire a lock, and be more like
> Java synchronized.

I think a with gil statement (always acquiring the lock) has more
consistent semantics and accomplishes the same goal. Is "with gil"
expensive if one already has the gil? I'm assuming here that the gil
is re-entrant (or could be made to see so).

>> this cause the function to have a GIL requiring/acquiring signature? I
>
> Not in isolation.
>
> But, IF (after a long deprecation cycle) we start to infer "nogil" on
> function signatures, then the presence of this Python statement would stop
> nogil from getting inferred.

Knowing the subtleties of whether any Python operations are used,
especially as the specific optimizations done continue to evolve, this
seems like a dangerous thing to infer.

> The intention is simply: If you have a cdef function which uses no Python
> operations, but still relies on holding the GIL to avoid race conditions
> (exactly the situation Stefan described initially), then you can use
> cython.synchronized to avoid any future inference from removing the GIL.

Again, with gil, if acceptable from a gil-holding context, seems
sufficient. Then if one did such inference, one could isolate it to
the important block.

>> suppose the reason that one implicitly holds the gil is that it allows
>> you to do arbitrary Python operations by default. Having to explicitly
>> mark when one wants to hold the GIL would be a big regression for
>> applications that do a lot of Python interaction (e.g. wrappers).
>
> Yes, I'm not proposing that at all.
>
>> I'm still thinking it could be reasonable for extern functions to be
>> nogil by default, as a user of C libraries is expected to manually and
>> externally respect its threading constraints. (This would require a
>> long deprecation period.)
>
> +1.

Actually, it's relatively safe to switch, as it will only make
currently illegal code legal, but won't break existing code.

> Perhaps this would be safer if we introduce "expects gil" (explicit
> declaration for neither "nogil" or "with gil"), and then start to emit
> warnings if a GIL-mode isn't declared on cdef functions which has their
> address taken.
>
> Thus, if you pass a callback to a C library, you'll get a reminder to
> declare a GIL mode.

That's a nice idea. I have to admit that all these special gil
declarations are a bit messy. I'd also rather introduce clear
decorators, e.g.

@cython.requires_gil  # expects gil
cdef a(): ...

@cython.requires.gil(False) # nogil
cdef b(): ...

@cython.aquires_gil  # with gil
cdef c(): ...

(Actually, now that we have the "with gil" statement, it could be
worth considering simply noticing the pattern of the entire function
body in a with gil block/as the first statement and acquiring the GIL
before argument parsing.)

Note that we need to declare functions as requiring the GIL to allow
for declaring cpython.pxd if extern functions are implicitly nogil.

- Robert

From d.s.seljebotn at astro.uio.no  Wed Aug 17 20:46:24 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Aug 2011 20:46:24 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>
	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
Message-ID: <4E4C0C80.6090604@astro.uio.no>

On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
> That's a nice idea. I have to admit that all these special gil
> declarations are a bit messy. I'd also rather introduce clear
> decorators, e.g.
>
> @cython.requires_gil  # expects gil
> cdef a(): ...
>
> @cython.requires.gil(False) # nogil
> cdef b(): ...
>
> @cython.aquires_gil  # with gil
> cdef c(): ...
>
> (Actually, now that we have the "with gil" statement, it could be
> worth considering simply noticing the pattern of the entire function
> body in a with gil block/as the first statement and acquiring the GIL
> before argument parsing.)
>
> Note that we need to declare functions as requiring the GIL to allow
> for declaring cpython.pxd if extern functions are implicitly nogil.

I agree, it's messy in the current situation, simplifying would be good.

Assuming we can't acquire the GIL in every single function just to be 
sure, I have a hunch that the "acquires_gil" aspect of a function is 
just declared in the wrong place. I mean, the same function might be 
passed as a callback to C both while holding the GIL and while not 
holding the GIL -- it would be nice to automatically wrap it in a 
GIL-acquiring wrapper only when needed.

So to me it makes more sense to have acquires_gil be part of function 
pointer types, or of the C call where the pointer is passed, or similar. 
Sort of like an FFI. Can't think of a practical scheme that's more 
user-friendly than the current way though...

Dag Sverre

From robertwb at math.washington.edu  Wed Aug 17 21:21:46 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Aug 2011 12:21:46 -0700
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4C0C80.6090604@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
	<4E4C0C80.6090604@astro.uio.no>
Message-ID: <CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>

On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>
>> That's a nice idea. I have to admit that all these special gil
>> declarations are a bit messy. I'd also rather introduce clear
>> decorators, e.g.
>>
>> @cython.requires_gil ?# expects gil
>> cdef a(): ...
>>
>> @cython.requires.gil(False) # nogil
>> cdef b(): ...
>>
>> @cython.aquires_gil ?# with gil
>> cdef c(): ...
>>
>> (Actually, now that we have the "with gil" statement, it could be
>> worth considering simply noticing the pattern of the entire function
>> body in a with gil block/as the first statement and acquiring the GIL
>> before argument parsing.)
>>
>> Note that we need to declare functions as requiring the GIL to allow
>> for declaring cpython.pxd if extern functions are implicitly nogil.
>
> I agree, it's messy in the current situation, simplifying would be good.
>
> Assuming we can't acquire the GIL in every single function just to be sure,
> I have a hunch that the "acquires_gil" aspect of a function is just declared
> in the wrong place. I mean, the same function might be passed as a callback
> to C both while holding the GIL and while not holding the GIL -- it would be
> nice to automatically wrap it in a GIL-acquiring wrapper only when needed.
>
> So to me it makes more sense to have acquires_gil be part of function
> pointer types, or of the C call where the pointer is passed, or similar.
> Sort of like an FFI. Can't think of a practical scheme that's more
> user-friendly than the current way though...

I was thinking the opposite, "aquires_gil" should be completely
transparent to the caller (assuming it's cheap enough to
check-or-acquire, but that's an optimization not API issue). On the
other hand requires/does not require does need to be visible to the
caller though, which argues for it being part of the signature.

Regarding inference, are you thinking the semantics being that we
(re)-acquire the GIL for every individual operation that needs it
(including python operations or entire with gil blocks), with the
obvious optimization that we may choose to not release it between
(nearly) consecutive blocks of code that all need the GIL? That would
be truer to Python semantics, but would require risky guesswork at
compile time or some kind of periodic runtime checks. (It would also
be a strongly backwards incompatible move, so as mentioned you'd need
some kind of a explicit marker and deprecation period.)

- Robert

From stefan_ml at behnel.de  Thu Aug 18 07:32:23 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Aug 2011 07:32:23 +0200
Subject: [Cython] What now?
In-Reply-To: <201108111624.37503@a>
References: <201108111624.37503@a>
Message-ID: <4E4CA3E7.2020904@behnel.de>

Stefano, 11.08.2011 16:24:
> now that I've nailed Cython code, I'd like to get into something more funny.
> Currently, I'm working on a set of macros to seamlessy integrate Cython into
> CMake build process (in fact, I love CMake). But, I'd like to work also on
> something more essential, so...

Here's something truly essential that's been on our TODO list for ages, and 
it's not even that hard to do, given today's infrastructure (inline 
functions in .pxd files, function signature overloading and all that).

http://wiki.cython.org/enhancements/overlaypythonmodules

Basically, the idea is to use a .pxd for existing Python modules 
(especially stdlib modules) and to override *some* names in it with fast C 
functions.

Approach:

- for each normally "import"-ed module (except for relative imports), 
search for a corresponding .pxd file in both the PYTHONPATH and under 
Cython/Includes/cpython/

- if found, build a scope for it that falls through to the scope of the 
normally imported Python module, but looks up names in the .pxd namespace 
first.

For testing, write a cpython/math.pxd that contains replacements for *some* 
of the functions and constants in Python's math module.

There is obviously a bit more to it than the short wrap-up above. For 
example, if none of the signatures of a function in the .pxd matches, it 
would have to fall through to the module as well. That isn't all that easy 
to accomplish with just a name lookup. But it's also not strictly required 
for an initial implementation, it would just mean that your math.pxd 
implementation would have to be provide a complete set of signatures for 
the functions it offers. I think the math module is particularly friendly here.

Interested?

Stefan

From d.s.seljebotn at astro.uio.no  Thu Aug 18 08:39:04 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Aug 2011 08:39:04 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>
	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>	<4E4C0C80.6090604@astro.uio.no>
	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
Message-ID: <4E4CB388.8040303@astro.uio.no>

On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>
>>> That's a nice idea. I have to admit that all these special gil
>>> declarations are a bit messy. I'd also rather introduce clear
>>> decorators, e.g.
>>>
>>> @cython.requires_gil  # expects gil
>>> cdef a(): ...
>>>
>>> @cython.requires.gil(False) # nogil
>>> cdef b(): ...
>>>
>>> @cython.aquires_gil  # with gil
>>> cdef c(): ...
>>>
>>> (Actually, now that we have the "with gil" statement, it could be
>>> worth considering simply noticing the pattern of the entire function
>>> body in a with gil block/as the first statement and acquiring the GIL
>>> before argument parsing.)
>>>
>>> Note that we need to declare functions as requiring the GIL to allow
>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>
>> I agree, it's messy in the current situation, simplifying would be good.
>>
>> Assuming we can't acquire the GIL in every single function just to be sure,
>> I have a hunch that the "acquires_gil" aspect of a function is just declared
>> in the wrong place. I mean, the same function might be passed as a callback
>> to C both while holding the GIL and while not holding the GIL -- it would be
>> nice to automatically wrap it in a GIL-acquiring wrapper only when needed.
>>
>> So to me it makes more sense to have acquires_gil be part of function
>> pointer types, or of the C call where the pointer is passed, or similar.
>> Sort of like an FFI. Can't think of a practical scheme that's more
>> user-friendly than the current way though...
>
> I was thinking the opposite, "aquires_gil" should be completely
> transparent to the caller (assuming it's cheap enough to
> check-or-acquire, but that's an optimization not API issue). On the
> other hand requires/does not require does need to be visible to the
> caller though, which argues for it being part of the signature.

Are you saying that every single function cdef function, ever, that are 
not "nogil" should have acqusition semantics?

Those semantics would be great, but I worry a bit about performance -- 
we just agreed to make function GIL-acquisition even a bit more 
expensive with that check...

Hmm. How about this:

  i) Every cdef function that needs the GIL to be held generates a 
GIL-acquiring wrapper function as well.

  ii) Whenever taking the address of such a function, you get the 
address of the GIL-requiring wrapper, which can safely be used from any 
code, whether holding the GIL or not.

  iii) However, Cython code that already holds the GIL can call the 
inner function directly, as an optimization. Should be possible to do 
this across pxd's as well.

  iv) We may introduce a cython.nogil_function_address or similar just 
to provide an option to get around ii).

  v) The "with gil" on functions simply ceases to take effect, and is 
deprecated from the language.

> Regarding inference, are you thinking the semantics being that we
> (re)-acquire the GIL for every individual operation that needs it
> (including python operations or entire with gil blocks), with the
> obvious optimization that we may choose to not release it between
> (nearly) consecutive blocks of code that all need the GIL? That would
> be truer to Python semantics, but would require risky guesswork at
> compile time or some kind of periodic runtime checks. (It would also
> be a strongly backwards incompatible move, so as mentioned you'd need
> some kind of a explicit marker and deprecation period.)
>

No, I wasn't talking about this at all (this time on the list -- I did 
mention that I wished for this behaviour in Munich, but that's really 
long term).

All I wished for now was for simple code like this:

cdef double f(double x): return x**2 + x

to be callable without holding the GIL. In particular since this is the 
major reason why prange users need to learn the "nogil" keyword.

The semantics are simple: For all cdef functions, if "nogil" could have 
been applied without a syntax error, then it gets automatically applied.

The only time this isn't completely safe is when the GIL is 
intentionally being used as a lock.

Dag Sverre

From stefan_ml at behnel.de  Thu Aug 18 08:52:44 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Aug 2011 08:52:44 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>
	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
Message-ID: <4E4CB6BC.3070705@behnel.de>

Robert Bradshaw, 17.08.2011 20:19:
> I have to admit that all these special gil
> declarations are a bit messy. I'd also rather introduce clear
> decorators, e.g.
>
> @cython.requires_gil  # expects gil
> cdef a(): ...
>
> @cython.requires.gil(False) # nogil
> cdef b(): ...
>
> @cython.aquires_gil  # with gil
> cdef c(): ...

The last would spell "@cython.aquire_gil", and yes, it totally makes sense 
to turn these into decorators. There's no reason at all why pure mode needs 
to differ from Cython mode here (however tricky it may be to write a GIL 
free function in pure mode...).


> (Actually, now that we have the "with gil" statement, it could be
> worth considering simply noticing the pattern of the entire function
> body in a with gil block/as the first statement and acquiring the GIL
> before argument parsing.)

Agreed.


> Note that we need to declare functions as requiring the GIL to allow
> for declaring cpython.pxd if extern functions are implicitly nogil.

Yes, I thought of that, too. In the specific case of .pxd files for 
external C libraries, a special "requires gil" syntax makes sense, as you'd 
want to put it at the "cdef extern" declaration once, instead of having to 
add a decorator to each of the declared functions.

Stefan

From d.s.seljebotn at astro.uio.no  Thu Aug 18 08:58:41 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Aug 2011 08:58:41 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4CB388.8040303@astro.uio.no>
References: <4E43C051.4080507@gmail.com>	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>	<4E4C0C80.6090604@astro.uio.no>	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
	<4E4CB388.8040303@astro.uio.no>
Message-ID: <4E4CB821.1020400@astro.uio.no>

On 08/18/2011 08:39 AM, Dag Sverre Seljebotn wrote:
> On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>>
>>>> That's a nice idea. I have to admit that all these special gil
>>>> declarations are a bit messy. I'd also rather introduce clear
>>>> decorators, e.g.
>>>>
>>>> @cython.requires_gil # expects gil
>>>> cdef a(): ...
>>>>
>>>> @cython.requires.gil(False) # nogil
>>>> cdef b(): ...
>>>>
>>>> @cython.aquires_gil # with gil
>>>> cdef c(): ...
>>>>
>>>> (Actually, now that we have the "with gil" statement, it could be
>>>> worth considering simply noticing the pattern of the entire function
>>>> body in a with gil block/as the first statement and acquiring the GIL
>>>> before argument parsing.)
>>>>
>>>> Note that we need to declare functions as requiring the GIL to allow
>>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>>
>>> I agree, it's messy in the current situation, simplifying would be good.
>>>
>>> Assuming we can't acquire the GIL in every single function just to be
>>> sure,
>>> I have a hunch that the "acquires_gil" aspect of a function is just
>>> declared
>>> in the wrong place. I mean, the same function might be passed as a
>>> callback
>>> to C both while holding the GIL and while not holding the GIL -- it
>>> would be
>>> nice to automatically wrap it in a GIL-acquiring wrapper only when
>>> needed.
>>>
>>> So to me it makes more sense to have acquires_gil be part of function
>>> pointer types, or of the C call where the pointer is passed, or similar.
>>> Sort of like an FFI. Can't think of a practical scheme that's more
>>> user-friendly than the current way though...
>>
>> I was thinking the opposite, "aquires_gil" should be completely
>> transparent to the caller (assuming it's cheap enough to
>> check-or-acquire, but that's an optimization not API issue). On the
>> other hand requires/does not require does need to be visible to the
>> caller though, which argues for it being part of the signature.
>
> Are you saying that every single function cdef function, ever, that are
> not "nogil" should have acqusition semantics?
>
> Those semantics would be great, but I worry a bit about performance --
> we just agreed to make function GIL-acquisition even a bit more
> expensive with that check...
>
> Hmm. How about this:
>
> i) Every cdef function that needs the GIL to be held generates a
> GIL-acquiring wrapper function as well.
>
> ii) Whenever taking the address of such a function, you get the address
> of the GIL-requiring wrapper, which can safely be used from any code,
> whether holding the GIL or not.
>
> iii) However, Cython code that already holds the GIL can call the inner
> function directly, as an optimization. Should be possible to do this
> across pxd's as well.
>
> iv) We may introduce a cython.nogil_function_address or similar just to
> provide an option to get around ii).
>
> v) The "with gil" on functions simply ceases to take effect, and is
> deprecated from the language.

Relevant benchmarks on my laptop:

Calling an almost-noop cdef function (in a loop): 0.7 ns per call

Calling same function with "with gil": 58 ns per call

A dict lookup + Python integer-object addition: 57 ns per iteration

I think the cost (basically 2 python ops) is too high to go for the 
simpler scheme of simply making everything "with gil" that is not 
"nogil"? Also, the GIL-initialization check will make it even more 
expensive.

cdef int noop(int i) nogil:
     return 2 * i

cdef int withgil(int i) with gil:
     return 2 * i


def call_noop(int reps):
     cdef int i
     cdef int s = 0
     for i in range(reps):
         s += noop(i)
     return s

def call_withgil(int reps):
     cdef int i
     cdef int s = 0
     for i in range(reps):
         s += withgil(i)
     return s

def call_dict(int reps):
     cdef int i
     cdef object s = 0
     for i in range(reps):
         s += d['2']
     return s

d = {}

for i in range(15):
     d[str(i)] = i

Dag SVerre

From stefan_ml at behnel.de  Thu Aug 18 09:06:39 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Aug 2011 09:06:39 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4CB388.8040303@astro.uio.no>
References: <4E43C051.4080507@gmail.com>	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>	<4E4C0C80.6090604@astro.uio.no>	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
	<4E4CB388.8040303@astro.uio.no>
Message-ID: <4E4CB9FF.1040009@behnel.de>

Dag Sverre Seljebotn, 18.08.2011 08:39:
> The semantics are simple: For all cdef functions, if "nogil" could have
> been applied without a syntax error, then it gets automatically applied.
>
> The only time this isn't completely safe is when the GIL is intentionally
> being used as a lock.

In which case, IMHO, it's actually a good idea to require an explicit 
declaration, because it makes this intention explicitly visible in the 
source code.

I take it that auto-nogil semantics for a function wouldn't automatically 
*release* the GIL. They'd just not require it, right? So all that would 
change is that you wouldn't get a compiler error when calling such a 
function without holding the GIL.

That shouldn't break all that much code...

Stefan

From d.s.seljebotn at astro.uio.no  Thu Aug 18 09:47:39 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Aug 2011 09:47:39 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4CB9FF.1040009@behnel.de>
References: <4E43C051.4080507@gmail.com>	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>	<4E4C0C80.6090604@astro.uio.no>	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>	<4E4CB388.8040303@astro.uio.no>
	<4E4CB9FF.1040009@behnel.de>
Message-ID: <4E4CC39B.9060701@astro.uio.no>

On 08/18/2011 09:06 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 18.08.2011 08:39:
>> The semantics are simple: For all cdef functions, if "nogil" could have
>> been applied without a syntax error, then it gets automatically applied.
>>
>> The only time this isn't completely safe is when the GIL is intentionally
>> being used as a lock.
>
> In which case, IMHO, it's actually a good idea to require an explicit
> declaration, because it makes this intention explicitly visible in the
> source code.
>
> I take it that auto-nogil semantics for a function wouldn't
> automatically *release* the GIL. They'd just not require it, right? So

Right. Sorry about not making it clearer earlier.

> all that would change is that you wouldn't get a compiler error when
> calling such a function without holding the GIL.
>
> That shouldn't break all that much code...

Happy that we agree!

Dag Sverre

From robertwb at math.washington.edu  Thu Aug 18 21:27:30 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Aug 2011 12:27:30 -0700
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4CB388.8040303@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
	<4E4C0C80.6090604@astro.uio.no>
	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
	<4E4CB388.8040303@astro.uio.no>
Message-ID: <CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>

On Wed, Aug 17, 2011 at 11:39 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
>>
>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>>
>>>> That's a nice idea. I have to admit that all these special gil
>>>> declarations are a bit messy. I'd also rather introduce clear
>>>> decorators, e.g.
>>>>
>>>> @cython.requires_gil ?# expects gil
>>>> cdef a(): ...
>>>>
>>>> @cython.requires.gil(False) # nogil
>>>> cdef b(): ...
>>>>
>>>> @cython.aquires_gil ?# with gil
>>>> cdef c(): ...
>>>>
>>>> (Actually, now that we have the "with gil" statement, it could be
>>>> worth considering simply noticing the pattern of the entire function
>>>> body in a with gil block/as the first statement and acquiring the GIL
>>>> before argument parsing.)
>>>>
>>>> Note that we need to declare functions as requiring the GIL to allow
>>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>>
>>> I agree, it's messy in the current situation, simplifying would be good.
>>>
>>> Assuming we can't acquire the GIL in every single function just to be
>>> sure,
>>> I have a hunch that the "acquires_gil" aspect of a function is just
>>> declared
>>> in the wrong place. I mean, the same function might be passed as a
>>> callback
>>> to C both while holding the GIL and while not holding the GIL -- it would
>>> be
>>> nice to automatically wrap it in a GIL-acquiring wrapper only when
>>> needed.
>>>
>>> So to me it makes more sense to have acquires_gil be part of function
>>> pointer types, or of the C call where the pointer is passed, or similar.
>>> Sort of like an FFI. Can't think of a practical scheme that's more
>>> user-friendly than the current way though...
>>
>> I was thinking the opposite, "aquires_gil" should be completely
>> transparent to the caller (assuming it's cheap enough to
>> check-or-acquire, but that's an optimization not API issue). On the
>> other hand requires/does not require does need to be visible to the
>> caller though, which argues for it being part of the signature.
>
> Are you saying that every single function cdef function, ever, that are not
> "nogil" should have acqusition semantics?

No, I was saying that the distinction between with gil and nogil
should be transparent (semantically) to the caller.

> Those semantics would be great, but I worry a bit about performance -- we
> just agreed to make function GIL-acquisition even a bit more expensive with
> that check...

According to your timings, the performance argument is a good one.

> Hmm. How about this:
>
> ?i) Every cdef function that needs the GIL to be held generates a
> GIL-acquiring wrapper function as well.
>
> ?ii) Whenever taking the address of such a function, you get the address of
> the GIL-requiring wrapper, which can safely be used from any code, whether
> holding the GIL or not.

You mean GIL-acquiring wrapper, right?

> ?iii) However, Cython code that already holds the GIL can call the inner
> function directly, as an optimization. Should be possible to do this across
> pxd's as well.
>
> ?iv) We may introduce a cython.nogil_function_address or similar just to
> provide an option to get around ii).
>
> ?v) The "with gil" on functions simply ceases to take effect, and is
> deprecated from the language.
>
>> Regarding inference, are you thinking the semantics being that we
>> (re)-acquire the GIL for every individual operation that needs it
>> (including python operations or entire with gil blocks), with the
>> obvious optimization that we may choose to not release it between
>> (nearly) consecutive blocks of code that all need the GIL? That would
>> be truer to Python semantics, but would require risky guesswork at
>> compile time or some kind of periodic runtime checks. (It would also
>> be a strongly backwards incompatible move, so as mentioned you'd need
>> some kind of a explicit marker and deprecation period.)
>>
>
> No, I wasn't talking about this at all (this time on the list -- I did
> mention that I wished for this behaviour in Munich, but that's really long
> term).
>
> All I wished for now was for simple code like this:
>
> cdef double f(double x): return x**2 + x
>
> to be callable without holding the GIL. In particular since this is the
> major reason why prange users need to learn the "nogil" keyword.
>
> The semantics are simple: For all cdef functions, if "nogil" could have been
> applied without a syntax error, then it gets automatically applied.

Give that the implicit nogil marker is a function of the function's
body, how would this information be propagated across pxd files?
Perhaps I'm confused, we have

cdef double f(double x):   # implicitly nogil
    return x**2 + x

cdef double f(double x):    # implicitly "with gil" or implicitly
"requires gil"?
    print x
    return x**2 + x

If "with gil" then constant, implicit gil acquisition could be a
non-obvious performance hit; if "requires gil" then I don't see how to
propagate accross .pxd files. (I do like the direction this is
heading, just pointing out some cons.)

> The only time this isn't completely safe is when the GIL is intentionally
> being used as a lock.

Or unintentionally being used as a lock... (Yes, that'd be user
error.) The idea that adding a print statement to a function's body
can produce such a large change is somewhat worrisome, but perhaps
worth it.

- Robert

From robertwb at math.washington.edu  Thu Aug 18 21:34:10 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Aug 2011 12:34:10 -0700
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
	<4E4C0C80.6090604@astro.uio.no>
	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
	<4E4CB388.8040303@astro.uio.no>
	<CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>
Message-ID: <CADiQ+QD6w242bLBbsScMc5Yx0JpPR1KTBc-k4mhqV=oB29ZoYw@mail.gmail.com>

On Thu, Aug 18, 2011 at 12:27 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Wed, Aug 17, 2011 at 11:39 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
>>>
>>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>>>
>>>>> That's a nice idea. I have to admit that all these special gil
>>>>> declarations are a bit messy. I'd also rather introduce clear
>>>>> decorators, e.g.
>>>>>
>>>>> @cython.requires_gil ?# expects gil
>>>>> cdef a(): ...
>>>>>
>>>>> @cython.requires.gil(False) # nogil
>>>>> cdef b(): ...
>>>>>
>>>>> @cython.aquires_gil ?# with gil
>>>>> cdef c(): ...
>>>>>
>>>>> (Actually, now that we have the "with gil" statement, it could be
>>>>> worth considering simply noticing the pattern of the entire function
>>>>> body in a with gil block/as the first statement and acquiring the GIL
>>>>> before argument parsing.)
>>>>>
>>>>> Note that we need to declare functions as requiring the GIL to allow
>>>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>>>
>>>> I agree, it's messy in the current situation, simplifying would be good.
>>>>
>>>> Assuming we can't acquire the GIL in every single function just to be
>>>> sure,
>>>> I have a hunch that the "acquires_gil" aspect of a function is just
>>>> declared
>>>> in the wrong place. I mean, the same function might be passed as a
>>>> callback
>>>> to C both while holding the GIL and while not holding the GIL -- it would
>>>> be
>>>> nice to automatically wrap it in a GIL-acquiring wrapper only when
>>>> needed.
>>>>
>>>> So to me it makes more sense to have acquires_gil be part of function
>>>> pointer types, or of the C call where the pointer is passed, or similar.
>>>> Sort of like an FFI. Can't think of a practical scheme that's more
>>>> user-friendly than the current way though...
>>>
>>> I was thinking the opposite, "aquires_gil" should be completely
>>> transparent to the caller (assuming it's cheap enough to
>>> check-or-acquire, but that's an optimization not API issue). On the
>>> other hand requires/does not require does need to be visible to the
>>> caller though, which argues for it being part of the signature.
>>
>> Are you saying that every single function cdef function, ever, that are not
>> "nogil" should have acqusition semantics?
>
> No, I was saying that the distinction between with gil and nogil
> should be transparent (semantically) to the caller.
>
>> Those semantics would be great, but I worry a bit about performance -- we
>> just agreed to make function GIL-acquisition even a bit more expensive with
>> that check...
>
> According to your timings, the performance argument is a good one.
>
>> Hmm. How about this:
>>
>> ?i) Every cdef function that needs the GIL to be held generates a
>> GIL-acquiring wrapper function as well.
>>
>> ?ii) Whenever taking the address of such a function, you get the address of
>> the GIL-requiring wrapper, which can safely be used from any code, whether
>> holding the GIL or not.
>
> You mean GIL-acquiring wrapper, right?
>
>> ?iii) However, Cython code that already holds the GIL can call the inner
>> function directly, as an optimization. Should be possible to do this across
>> pxd's as well.
>>
>> ?iv) We may introduce a cython.nogil_function_address or similar just to
>> provide an option to get around ii).
>>
>> ?v) The "with gil" on functions simply ceases to take effect, and is
>> deprecated from the language.
>>
>>> Regarding inference, are you thinking the semantics being that we
>>> (re)-acquire the GIL for every individual operation that needs it
>>> (including python operations or entire with gil blocks), with the
>>> obvious optimization that we may choose to not release it between
>>> (nearly) consecutive blocks of code that all need the GIL? That would
>>> be truer to Python semantics, but would require risky guesswork at
>>> compile time or some kind of periodic runtime checks. (It would also
>>> be a strongly backwards incompatible move, so as mentioned you'd need
>>> some kind of a explicit marker and deprecation period.)
>>>
>>
>> No, I wasn't talking about this at all (this time on the list -- I did
>> mention that I wished for this behaviour in Munich, but that's really long
>> term).
>>
>> All I wished for now was for simple code like this:
>>
>> cdef double f(double x): return x**2 + x
>>
>> to be callable without holding the GIL. In particular since this is the
>> major reason why prange users need to learn the "nogil" keyword.
>>
>> The semantics are simple: For all cdef functions, if "nogil" could have been
>> applied without a syntax error, then it gets automatically applied.
>
> Give that the implicit nogil marker is a function of the function's
> body, how would this information be propagated across pxd files?
> Perhaps I'm confused, we have
>
> cdef double f(double x): ? # implicitly nogil
> ? ?return x**2 + x
>
> cdef double f(double x): ? ?# implicitly "with gil" or implicitly
> "requires gil"?
> ? ?print x
> ? ?return x**2 + x
>
> If "with gil" then constant, implicit gil acquisition could be a
> non-obvious performance hit; if "requires gil" then I don't see how to
> propagate accross .pxd files. (I do like the direction this is
> heading, just pointing out some cons.)

On second though, "with gil" (= "acquires gil") isn't that bad,
because the mantra of "look for unneeded Python operations" to speed
up your code directly applies here, and acquiring the GIL is about as
expensive as the one or two operations that requested it.

>> The only time this isn't completely safe is when the GIL is intentionally
>> being used as a lock.
>
> Or unintentionally being used as a lock... (Yes, that'd be user
> error.) The idea that adding a print statement to a function's body
> can produce such a large change is somewhat worrisome, but perhaps
> worth it.
>
> - Robert
>

From robertwb at math.washington.edu  Thu Aug 18 23:35:02 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Aug 2011 14:35:02 -0700
Subject: [Cython] What now?
In-Reply-To: <4E4CA3E7.2020904@behnel.de>
References: <201108111624.37503@a> <4E4CA3E7.2020904@behnel.de>
Message-ID: <CADiQ+QDRZs_DjOe0gdLghDV-FdkTeWUU-dux4a_N-=-YBGDVqw@mail.gmail.com>

On Wed, Aug 17, 2011 at 10:32 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefano, 11.08.2011 16:24:
>>
>> now that I've nailed Cython code, I'd like to get into something more
>> funny.
>> Currently, I'm working on a set of macros to seamlessy integrate Cython
>> into
>> CMake build process (in fact, I love CMake). But, I'd like to work also on
>> something more essential, so...
>
> Here's something truly essential that's been on our TODO list for ages, and
> it's not even that hard to do, given today's infrastructure (inline
> functions in .pxd files, function signature overloading and all that).
>
> http://wiki.cython.org/enhancements/overlaypythonmodules

+1, this is a high-impact feature that should be relatively easy to dig into.

> Basically, the idea is to use a .pxd for existing Python modules (especially
> stdlib modules) and to override *some* names in it with fast C functions.
>
> Approach:
>
> - for each normally "import"-ed module (except for relative imports), search
> for a corresponding .pxd file in both the PYTHONPATH and under
> Cython/Includes/cpython/
>
> - if found, build a scope for it that falls through to the scope of the
> normally imported Python module, but looks up names in the .pxd namespace
> first.
>
> For testing, write a cpython/math.pxd that contains replacements for *some*
> of the functions and constants in Python's math module.
>
> There is obviously a bit more to it than the short wrap-up above. For
> example, if none of the signatures of a function in the .pxd matches, it
> would have to fall through to the module as well. That isn't all that easy
> to accomplish with just a name lookup. But it's also not strictly required
> for an initial implementation, it would just mean that your math.pxd
> implementation would have to be provide a complete set of signatures for the
> functions it offers. I think the math module is particularly friendly here.
>
> Interested?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefano.k.sanfilippo at gmail.com  Fri Aug 19 07:50:28 2011
From: stefano.k.sanfilippo at gmail.com (Stefano Sanfilippo)
Date: Fri, 19 Aug 2011 07:50:28 +0200
Subject: [Cython] What now?
In-Reply-To: <CADiQ+QDRZs_DjOe0gdLghDV-FdkTeWUU-dux4a_N-=-YBGDVqw@mail.gmail.com>
References: <201108111624.37503@a> <4E4CA3E7.2020904@behnel.de>
	<CADiQ+QDRZs_DjOe0gdLghDV-FdkTeWUU-dux4a_N-=-YBGDVqw@mail.gmail.com>
Message-ID: <CAAQ0_PYh2SLpwj=G7eAEHG1_xdF4FXb9jkJGTi09OOtGxE0t4g@mail.gmail.com>

Sounds really interesting to me. Thanks for your proposal :) I may
have to dig the internals, so, first I'll study the Compiler code.

--SKS

From d.s.seljebotn at astro.uio.no  Fri Aug 19 08:35:55 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Aug 2011 08:35:55 +0200
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>
References: <4E43C051.4080507@gmail.com>
	<4E43C3F2.5070609@astro.uio.no>	<4E43C759.2040509@behnel.de>
	<4E43D0CA.1060900@astro.uio.no>	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>	<4E44CD52.5040504@astro.uio.no>
	<4E452079.9010204@behnel.de>	<4E452709.6000404@astro.uio.no>	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>	<4E4B76E3.3030808@astro.uio.no>	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>	<4E4C0C80.6090604@astro.uio.no>	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>	<4E4CB388.8040303@astro.uio.no>
	<CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>
Message-ID: <4E4E044B.1040801@astro.uio.no>

On 08/18/2011 09:27 PM, Robert Bradshaw wrote:
> On Wed, Aug 17, 2011 at 11:39 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
>>>
>>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no>    wrote:
>>>>
>>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>>>
>>>>> That's a nice idea. I have to admit that all these special gil
>>>>> declarations are a bit messy. I'd also rather introduce clear
>>>>> decorators, e.g.
>>>>>
>>>>> @cython.requires_gil  # expects gil
>>>>> cdef a(): ...
>>>>>
>>>>> @cython.requires.gil(False) # nogil
>>>>> cdef b(): ...
>>>>>
>>>>> @cython.aquires_gil  # with gil
>>>>> cdef c(): ...
>>>>>
>>>>> (Actually, now that we have the "with gil" statement, it could be
>>>>> worth considering simply noticing the pattern of the entire function
>>>>> body in a with gil block/as the first statement and acquiring the GIL
>>>>> before argument parsing.)
>>>>>
>>>>> Note that we need to declare functions as requiring the GIL to allow
>>>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>>>
>>>> I agree, it's messy in the current situation, simplifying would be good.
>>>>
>>>> Assuming we can't acquire the GIL in every single function just to be
>>>> sure,
>>>> I have a hunch that the "acquires_gil" aspect of a function is just
>>>> declared
>>>> in the wrong place. I mean, the same function might be passed as a
>>>> callback
>>>> to C both while holding the GIL and while not holding the GIL -- it would
>>>> be
>>>> nice to automatically wrap it in a GIL-acquiring wrapper only when
>>>> needed.
>>>>
>>>> So to me it makes more sense to have acquires_gil be part of function
>>>> pointer types, or of the C call where the pointer is passed, or similar.
>>>> Sort of like an FFI. Can't think of a practical scheme that's more
>>>> user-friendly than the current way though...
>>>
>>> I was thinking the opposite, "aquires_gil" should be completely
>>> transparent to the caller (assuming it's cheap enough to
>>> check-or-acquire, but that's an optimization not API issue). On the
>>> other hand requires/does not require does need to be visible to the
>>> caller though, which argues for it being part of the signature.
>>
>> Are you saying that every single function cdef function, ever, that are not
>> "nogil" should have acqusition semantics?
>
> No, I was saying that the distinction between with gil and nogil
> should be transparent (semantically) to the caller.
>
>> Those semantics would be great, but I worry a bit about performance -- we
>> just agreed to make function GIL-acquisition even a bit more expensive with
>> that check...
>
> According to your timings, the performance argument is a good one.
>
>> Hmm. How about this:
>>
>>   i) Every cdef function that needs the GIL to be held generates a
>> GIL-acquiring wrapper function as well.
>>
>>   ii) Whenever taking the address of such a function, you get the address of
>> the GIL-requiring wrapper, which can safely be used from any code, whether
>> holding the GIL or not.
>
> You mean GIL-acquiring wrapper, right?

Yes, sorry.

>
>>   iii) However, Cython code that already holds the GIL can call the inner
>> function directly, as an optimization. Should be possible to do this across
>> pxd's as well.
>>
>>   iv) We may introduce a cython.nogil_function_address or similar just to
>> provide an option to get around ii).
>>
>>   v) The "with gil" on functions simply ceases to take effect, and is
>> deprecated from the language.
>>
>>> Regarding inference, are you thinking the semantics being that we
>>> (re)-acquire the GIL for every individual operation that needs it
>>> (including python operations or entire with gil blocks), with the
>>> obvious optimization that we may choose to not release it between
>>> (nearly) consecutive blocks of code that all need the GIL? That would
>>> be truer to Python semantics, but would require risky guesswork at
>>> compile time or some kind of periodic runtime checks. (It would also
>>> be a strongly backwards incompatible move, so as mentioned you'd need
>>> some kind of a explicit marker and deprecation period.)
>>>
>>
>> No, I wasn't talking about this at all (this time on the list -- I did
>> mention that I wished for this behaviour in Munich, but that's really long
>> term).
>>
>> All I wished for now was for simple code like this:
>>
>> cdef double f(double x): return x**2 + x
>>
>> to be callable without holding the GIL. In particular since this is the
>> major reason why prange users need to learn the "nogil" keyword.
>>
>> The semantics are simple: For all cdef functions, if "nogil" could have been
>> applied without a syntax error, then it gets automatically applied.
>
> Give that the implicit nogil marker is a function of the function's
> body, how would this information be propagated across pxd files?
> Perhaps I'm confused, we have
>
> cdef double f(double x):   # implicitly nogil
>      return x**2 + x
>
> cdef double f(double x):    # implicitly "with gil" or implicitly
> "requires gil"?
>      print x
>      return x**2 + x
>
> If "with gil" then constant, implicit gil acquisition could be a
> non-obvious performance hit; if "requires gil" then I don't see how to
> propagate accross .pxd files. (I do like the direction this is
> heading, just pointing out some cons.)

True, I didn't think about the pxd issue.

Brainstorming:

I) Downsize my proposal to within-module calls, require nogil for 
cross-module.

II) Change Cython so that any function can be called from within "nogil" 
sections (and then their GIL-acquiring wrapper, if any, will be called). 
Of course this is a little fussy regarding exceptions etc. and need some 
thinking through (which I haven't done).

Together with the proposal above to deprecate the "with gil" modifier 
and always generate GIL-acquiring wrappers, this works across modules: 
The module function pointer table gets two slots per cdef function; one 
to call if the GIL is held, and one to call if the GIL is not held. 
Functions that are nogil (explicitly or implicitly) then simply use the 
same pointer in both slots.

>> The only time this isn't completely safe is when the GIL is intentionally
>> being used as a lock.
>
> Or unintentionally being used as a lock... (Yes, that'd be user
> error.) The idea that adding a print statement to a function's body
> can produce such a large change is somewhat worrisome, but perhaps
> worth it.

My hunch is to postpone the question of inferring the nogil modifier. 
Instead, first focus on making the "with gil" modifier optional 
(GIL-acquiring wrapper on all cdef functions), and see where that takes us.

I guess I can raise the question again with a formal CEP when I have 
time to do something about it in terms of code...

Dag Sverre

From markflorisson88 at gmail.com  Sat Aug 20 00:34:17 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 20 Aug 2011 00:34:17 +0200
Subject: [Cython] except clause on nogil functions
Message-ID: <CANg26EXjba1v7Zc-BM=1K9Jq8w5C4KHJgdaDEw-xyfBMej5Sxw@mail.gmail.com>

Hey,

I think I found a rather serious bug: if an error label is used in a
nogil function, it tries to build a traceback. So if the GIL is
released you will immediately segfault, and otherwise it will work
fine! Here is a snippet:

cdef int with_gil_func() except 0 with gil:
    raise Exception("error!")

cdef int nogil_func() nogil except 0:
    with_gil_func()

with nogil:
    nogil_func()


[0] [00:03] ~  ? python -c 'import test'
[1]    6888 segmentation fault  python -c 'import test'

I've tried as far back as 0.12 but all versions down to 0.12 have it.

BTW, it's really weird that the except clause on the function has to
come after nogil but before with gil, and otherwise it's a syntax
error.

Should we fix it and do a bugfix release?

From markflorisson88 at gmail.com  Sat Aug 20 11:55:19 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 20 Aug 2011 11:55:19 +0200
Subject: [Cython] except clause on nogil functions
In-Reply-To: <CANg26EXjba1v7Zc-BM=1K9Jq8w5C4KHJgdaDEw-xyfBMej5Sxw@mail.gmail.com>
References: <CANg26EXjba1v7Zc-BM=1K9Jq8w5C4KHJgdaDEw-xyfBMej5Sxw@mail.gmail.com>
Message-ID: <CANg26EW+EpAispDjdNBYVszP0BTK8fBDEjvjnZkTpr697E153Q@mail.gmail.com>

On 20 August 2011 00:34, mark florisson <markflorisson88 at gmail.com> wrote:
> Hey,
>
> I think I found a rather serious bug: if an error label is used in a
> nogil function, it tries to build a traceback. So if the GIL is
> released you will immediately segfault, and otherwise it will work
> fine! Here is a snippet:
>
> cdef int with_gil_func() except 0 with gil:
> ? ?raise Exception("error!")
>
> cdef int nogil_func() nogil except 0:
> ? ?with_gil_func()
>
> with nogil:
> ? ?nogil_func()
>
>
> [0] [00:03] ~ ?? python -c 'import test'
> [1] ? ?6888 segmentation fault ?python -c 'import test'
>
> I've tried as far back as 0.12 but all versions down to 0.12 have it.
>
> BTW, it's really weird that the except clause on the function has to
> come after nogil but before with gil, and otherwise it's a syntax
> error.
>
> Should we fix it and do a bugfix release?
>

Ok I made a pull request for the fix: https://github.com/cython/cython/pull/56
Considering that it has been around this long and nobody has reported
it (?), I assume nobody has had a problem with it, so it's perhaps not
as urgent as I previously stated.

What about the syntax issue though? Should we fix that?

From dalcinl at gmail.com  Sat Aug 20 21:07:46 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 20 Aug 2011 16:07:46 -0300
Subject: [Cython] except clause on nogil functions
In-Reply-To: <CANg26EW+EpAispDjdNBYVszP0BTK8fBDEjvjnZkTpr697E153Q@mail.gmail.com>
References: <CANg26EXjba1v7Zc-BM=1K9Jq8w5C4KHJgdaDEw-xyfBMej5Sxw@mail.gmail.com>
	<CANg26EW+EpAispDjdNBYVszP0BTK8fBDEjvjnZkTpr697E153Q@mail.gmail.com>
Message-ID: <CAEcYPwDBsP=6vC8Un5NRfbN1B=D9=TGwUNiA0BHbLuK5+rStag@mail.gmail.com>

On 20 August 2011 06:55, mark florisson <markflorisson88 at gmail.com> wrote:
>
> What about the syntax issue though? Should we fix that?
>

Long ago I also complained about that. Greg commented that we should
really accept "except" and "gil/nogil" in any position, so that

cdef int foo() except 0 with gil
cdef int foo() with gil except 0
cdef int bar() nogil except 0
cdef int bar() except 0 nogil

should be all valid.



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Sun Aug 21 06:36:26 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Aug 2011 21:36:26 -0700
Subject: [Cython] except clause on nogil functions
In-Reply-To: <CANg26EW+EpAispDjdNBYVszP0BTK8fBDEjvjnZkTpr697E153Q@mail.gmail.com>
References: <CANg26EXjba1v7Zc-BM=1K9Jq8w5C4KHJgdaDEw-xyfBMej5Sxw@mail.gmail.com>
	<CANg26EW+EpAispDjdNBYVszP0BTK8fBDEjvjnZkTpr697E153Q@mail.gmail.com>
Message-ID: <CADiQ+QCZ4YMVAedZvq-+vtFzRd8D27WgCZ9y39Jqb_SJhuRBNQ@mail.gmail.com>

On Sat, Aug 20, 2011 at 2:55 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 20 August 2011 00:34, mark florisson <markflorisson88 at gmail.com> wrote:
>> Hey,
>>
>> I think I found a rather serious bug: if an error label is used in a
>> nogil function, it tries to build a traceback. So if the GIL is
>> released you will immediately segfault, and otherwise it will work
>> fine! Here is a snippet:
>>
>> cdef int with_gil_func() except 0 with gil:
>> ? ?raise Exception("error!")
>>
>> cdef int nogil_func() nogil except 0:
>> ? ?with_gil_func()
>>
>> with nogil:
>> ? ?nogil_func()
>>
>>
>> [0] [00:03] ~ ?? python -c 'import test'
>> [1] ? ?6888 segmentation fault ?python -c 'import test'
>>
>> I've tried as far back as 0.12 but all versions down to 0.12 have it.
>>
>> BTW, it's really weird that the except clause on the function has to
>> come after nogil but before with gil, and otherwise it's a syntax
>> error.
>>
>> Should we fix it and do a bugfix release?
>>
>
> Ok I made a pull request for the fix: https://github.com/cython/cython/pull/56

Thanks.

> Considering that it has been around this long and nobody has reported
> it (?), I assume nobody has had a problem with it, so it's perhaps not
> as urgent as I previously stated.

I agree.

> What about the syntax issue though? Should we fix that?

Sure, I don't see any reason the asymmetry of these options.

- Robert

From robertwb at math.washington.edu  Sun Aug 21 06:44:27 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Aug 2011 21:44:27 -0700
Subject: [Cython] Calling gil-requiring function not allowed without gil
In-Reply-To: <4E4E044B.1040801@astro.uio.no>
References: <4E43C051.4080507@gmail.com> <4E43C3F2.5070609@astro.uio.no>
	<4E43C759.2040509@behnel.de> <4E43D0CA.1060900@astro.uio.no>
	<CADiQ+QA-8UKopOeDEphyhLHAr8Qs109EfwniMW+XK7oidiFp1g@mail.gmail.com>
	<4E44CD52.5040504@astro.uio.no> <4E452079.9010204@behnel.de>
	<4E452709.6000404@astro.uio.no>
	<CADiQ+QA49cZ9AgRnLYVrOKZvwQGc6j+KsY7Qyn7EQoTbwxp9pA@mail.gmail.com>
	<4E4B76E3.3030808@astro.uio.no>
	<CADiQ+QAXQAWkwf-npc7g_zEZ-_fkxwaE9jgYQz9gWtx9-H7pRw@mail.gmail.com>
	<4E4C0C80.6090604@astro.uio.no>
	<CADiQ+QA1ToVYiAUBD7FK43ec7vhPN3NgdOJZsReNTLe3S_o2Ng@mail.gmail.com>
	<4E4CB388.8040303@astro.uio.no>
	<CADiQ+QDcHbtW5yGa4hMST78Hd+E2KEWifGbUZRSRjZQ9ThgqtQ@mail.gmail.com>
	<4E4E044B.1040801@astro.uio.no>
Message-ID: <CADiQ+QDA_BFbWcGvZm=j+Yvadw+DGM04c8nCn0D-OL3x-WXDyQ@mail.gmail.com>

On Thu, Aug 18, 2011 at 11:35 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/18/2011 09:27 PM, Robert Bradshaw wrote:
>>
>> On Wed, Aug 17, 2011 at 11:39 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 08/17/2011 09:21 PM, Robert Bradshaw wrote:
>>>>
>>>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>
>>>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote:
>>>>>>
>>>>>> That's a nice idea. I have to admit that all these special gil
>>>>>> declarations are a bit messy. I'd also rather introduce clear
>>>>>> decorators, e.g.
>>>>>>
>>>>>> @cython.requires_gil ?# expects gil
>>>>>> cdef a(): ...
>>>>>>
>>>>>> @cython.requires.gil(False) # nogil
>>>>>> cdef b(): ...
>>>>>>
>>>>>> @cython.aquires_gil ?# with gil
>>>>>> cdef c(): ...
>>>>>>
>>>>>> (Actually, now that we have the "with gil" statement, it could be
>>>>>> worth considering simply noticing the pattern of the entire function
>>>>>> body in a with gil block/as the first statement and acquiring the GIL
>>>>>> before argument parsing.)
>>>>>>
>>>>>> Note that we need to declare functions as requiring the GIL to allow
>>>>>> for declaring cpython.pxd if extern functions are implicitly nogil.
>>>>>
>>>>> I agree, it's messy in the current situation, simplifying would be
>>>>> good.
>>>>>
>>>>> Assuming we can't acquire the GIL in every single function just to be
>>>>> sure,
>>>>> I have a hunch that the "acquires_gil" aspect of a function is just
>>>>> declared
>>>>> in the wrong place. I mean, the same function might be passed as a
>>>>> callback
>>>>> to C both while holding the GIL and while not holding the GIL -- it
>>>>> would
>>>>> be
>>>>> nice to automatically wrap it in a GIL-acquiring wrapper only when
>>>>> needed.
>>>>>
>>>>> So to me it makes more sense to have acquires_gil be part of function
>>>>> pointer types, or of the C call where the pointer is passed, or
>>>>> similar.
>>>>> Sort of like an FFI. Can't think of a practical scheme that's more
>>>>> user-friendly than the current way though...
>>>>
>>>> I was thinking the opposite, "aquires_gil" should be completely
>>>> transparent to the caller (assuming it's cheap enough to
>>>> check-or-acquire, but that's an optimization not API issue). On the
>>>> other hand requires/does not require does need to be visible to the
>>>> caller though, which argues for it being part of the signature.
>>>
>>> Are you saying that every single function cdef function, ever, that are
>>> not
>>> "nogil" should have acqusition semantics?
>>
>> No, I was saying that the distinction between with gil and nogil
>> should be transparent (semantically) to the caller.
>>
>>> Those semantics would be great, but I worry a bit about performance -- we
>>> just agreed to make function GIL-acquisition even a bit more expensive
>>> with
>>> that check...
>>
>> According to your timings, the performance argument is a good one.
>>
>>> Hmm. How about this:
>>>
>>> ?i) Every cdef function that needs the GIL to be held generates a
>>> GIL-acquiring wrapper function as well.
>>>
>>> ?ii) Whenever taking the address of such a function, you get the address
>>> of
>>> the GIL-requiring wrapper, which can safely be used from any code,
>>> whether
>>> holding the GIL or not.
>>
>> You mean GIL-acquiring wrapper, right?
>
> Yes, sorry.
>
>>
>>> ?iii) However, Cython code that already holds the GIL can call the inner
>>> function directly, as an optimization. Should be possible to do this
>>> across
>>> pxd's as well.
>>>
>>> ?iv) We may introduce a cython.nogil_function_address or similar just to
>>> provide an option to get around ii).
>>>
>>> ?v) The "with gil" on functions simply ceases to take effect, and is
>>> deprecated from the language.
>>>
>>>> Regarding inference, are you thinking the semantics being that we
>>>> (re)-acquire the GIL for every individual operation that needs it
>>>> (including python operations or entire with gil blocks), with the
>>>> obvious optimization that we may choose to not release it between
>>>> (nearly) consecutive blocks of code that all need the GIL? That would
>>>> be truer to Python semantics, but would require risky guesswork at
>>>> compile time or some kind of periodic runtime checks. (It would also
>>>> be a strongly backwards incompatible move, so as mentioned you'd need
>>>> some kind of a explicit marker and deprecation period.)
>>>>
>>>
>>> No, I wasn't talking about this at all (this time on the list -- I did
>>> mention that I wished for this behaviour in Munich, but that's really
>>> long
>>> term).
>>>
>>> All I wished for now was for simple code like this:
>>>
>>> cdef double f(double x): return x**2 + x
>>>
>>> to be callable without holding the GIL. In particular since this is the
>>> major reason why prange users need to learn the "nogil" keyword.
>>>
>>> The semantics are simple: For all cdef functions, if "nogil" could have
>>> been
>>> applied without a syntax error, then it gets automatically applied.
>>
>> Give that the implicit nogil marker is a function of the function's
>> body, how would this information be propagated across pxd files?
>> Perhaps I'm confused, we have
>>
>> cdef double f(double x): ? # implicitly nogil
>> ? ? return x**2 + x
>>
>> cdef double f(double x): ? ?# implicitly "with gil" or implicitly
>> "requires gil"?
>> ? ? print x
>> ? ? return x**2 + x
>>
>> If "with gil" then constant, implicit gil acquisition could be a
>> non-obvious performance hit; if "requires gil" then I don't see how to
>> propagate accross .pxd files. (I do like the direction this is
>> heading, just pointing out some cons.)
>
> True, I didn't think about the pxd issue.
>
> Brainstorming:
>
> I) Downsize my proposal to within-module calls, require nogil for
> cross-module.
>
> II) Change Cython so that any function can be called from within "nogil"
> sections (and then their GIL-acquiring wrapper, if any, will be called). Of
> course this is a little fussy regarding exceptions etc. and need some
> thinking through (which I haven't done).
>
> Together with the proposal above to deprecate the "with gil" modifier and
> always generate GIL-acquiring wrappers, this works across modules: The
> module function pointer table gets two slots per cdef function; one to call
> if the GIL is held, and one to call if the GIL is not held. Functions that
> are nogil (explicitly or implicitly) then simply use the same pointer in
> both slots.

Sounds reasonable. Is there any scenario where one would explicitly
not want the GIL-acquiring wrapper to be created/potentially used?

>>> The only time this isn't completely safe is when the GIL is intentionally
>>> being used as a lock.
>>
>> Or unintentionally being used as a lock... (Yes, that'd be user
>> error.) The idea that adding a print statement to a function's body
>> can produce such a large change is somewhat worrisome, but perhaps
>> worth it.
>
> My hunch is to postpone the question of inferring the nogil modifier.
> Instead, first focus on making the "with gil" modifier optional
> (GIL-acquiring wrapper on all cdef functions), and see where that takes us.
>
> I guess I can raise the question again with a formal CEP when I have time to
> do something about it in terms of code...

Makes sense.

- Robert

From stefan_ml at behnel.de  Sun Aug 21 12:31:53 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Aug 2011 12:31:53 +0200
Subject: [Cython] Entry.utility_code_definitions vs. Entry.utility_code
Message-ID: <4E50DE99.1090104@behnel.de>

Hi,

is there a reason for having the two?

The "uc_definition" is set to instances of "CythonUtilityCode" in 
UtilityCode.py, whereas the other is used to keep references to 
"UtilityCode" instances, but both inject their code in the same way, it 
appears. The first also has dedicated support in the pipeline, whereas 
support for the latter is spread across the syntax tree nodes.

It seems cleaner to me to use the first approach, i.e. to simply walk all 
used entries at some late point in the pipeline and to inject their utility 
code, rather than to leave that responsibility to each node that uses them.

Anything I misunderstood here? If not, any objections to merging the latter 
into the first?

Stefan

From markflorisson88 at gmail.com  Sun Aug 21 12:54:47 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 21 Aug 2011 12:54:47 +0200
Subject: [Cython] Entry.utility_code_definitions vs. Entry.utility_code
In-Reply-To: <4E50DE99.1090104@behnel.de>
References: <4E50DE99.1090104@behnel.de>
Message-ID: <CANg26EV-Nw71uV9QqLaqkJuQAhuhL+T6Ny6hmAKvFwzW9Y5ksQ@mail.gmail.com>

On 21 August 2011 12:31, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> is there a reason for having the two?
>
> The "uc_definition" is set to instances of "CythonUtilityCode" in
> UtilityCode.py, whereas the other is used to keep references to
> "UtilityCode" instances, but both inject their code in the same way, it
> appears. The first also has dedicated support in the pipeline, whereas
> support for the latter is spread across the syntax tree nodes.
>
> It seems cleaner to me to use the first approach, i.e. to simply walk all
> used entries at some late point in the pipeline and to inject their utility
> code, rather than to leave that responsibility to each node that uses them.
>
> Anything I misunderstood here? If not, any objections to merging the latter
> into the first?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

That would be fine with me, as long as it meant no modifications to
(Cython)UtilityCode or the code in the pipeline, as I have modified
all of those in _memview.

I see only two cases in ExprNodes of code checking entry.utility_code
and using the utilities, though. So if I understand correctly, it's
about those few cases?

From markflorisson88 at gmail.com  Sun Aug 21 12:57:32 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 21 Aug 2011 12:57:32 +0200
Subject: [Cython] Entry.utility_code_definitions vs. Entry.utility_code
In-Reply-To: <CANg26EV-Nw71uV9QqLaqkJuQAhuhL+T6Ny6hmAKvFwzW9Y5ksQ@mail.gmail.com>
References: <4E50DE99.1090104@behnel.de>
	<CANg26EV-Nw71uV9QqLaqkJuQAhuhL+T6Ny6hmAKvFwzW9Y5ksQ@mail.gmail.com>
Message-ID: <CANg26EUr7GyNk7CthAtf7Ur2=fvnwsiqZ1p=G3OEmzi7sjKkGw@mail.gmail.com>

On 21 August 2011 12:54, mark florisson <markflorisson88 at gmail.com> wrote:
> On 21 August 2011 12:31, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> is there a reason for having the two?
>>
>> The "uc_definition" is set to instances of "CythonUtilityCode" in
>> UtilityCode.py, whereas the other is used to keep references to
>> "UtilityCode" instances, but both inject their code in the same way, it
>> appears. The first also has dedicated support in the pipeline, whereas
>> support for the latter is spread across the syntax tree nodes.
>>
>> It seems cleaner to me to use the first approach, i.e. to simply walk all
>> used entries at some late point in the pipeline and to inject their utility
>> code, rather than to leave that responsibility to each node that uses them.
>>
>> Anything I misunderstood here? If not, any objections to merging the latter
>> into the first?
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> That would be fine with me, as long as it meant no modifications to
> (Cython)UtilityCode or the code in the pipeline, as I have modified
> all of those in _memview.
>
> I see only two cases in ExprNodes of code checking entry.utility_code
> and using the utilities, though. So if I understand correctly, it's
> about those few cases?
>

The code in the pipeline is tailored for CythonUtilityCode only, so I
suppose it would need to be modified. I believe the _memview branch is
nearly finished though, so could you wait for that merge? Otherwise
modifying the pipeline wouldn't be too bad either, it might only give
a few conflicts.

From stefan_ml at behnel.de  Sun Aug 21 14:35:59 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Aug 2011 14:35:59 +0200
Subject: [Cython] Entry.utility_code_definitions vs. Entry.utility_code
In-Reply-To: <CANg26EUr7GyNk7CthAtf7Ur2=fvnwsiqZ1p=G3OEmzi7sjKkGw@mail.gmail.com>
References: <4E50DE99.1090104@behnel.de>	<CANg26EV-Nw71uV9QqLaqkJuQAhuhL+T6Ny6hmAKvFwzW9Y5ksQ@mail.gmail.com>
	<CANg26EUr7GyNk7CthAtf7Ur2=fvnwsiqZ1p=G3OEmzi7sjKkGw@mail.gmail.com>
Message-ID: <4E50FBAF.9080006@behnel.de>

mark florisson, 21.08.2011 12:57:
> On 21 August 2011 12:54, mark florisson wrote:
>> On 21 August 2011 12:31, Stefan Behnel wrote:
>>> is there a reason for having the two?
>>>
>>> The "uc_definition" is set to instances of "CythonUtilityCode" in
>>> UtilityCode.py, whereas the other is used to keep references to
>>> "UtilityCode" instances, but both inject their code in the same way, it
>>> appears. The first also has dedicated support in the pipeline, whereas
>>> support for the latter is spread across the syntax tree nodes.
>>>
>>> It seems cleaner to me to use the first approach, i.e. to simply walk all
>>> used entries at some late point in the pipeline and to inject their utility
>>> code, rather than to leave that responsibility to each node that uses them.
>>>
>>> Anything I misunderstood here? If not, any objections to merging the latter
>>> into the first?
>>
>> That would be fine with me, as long as it meant no modifications to
>> (Cython)UtilityCode or the code in the pipeline, as I have modified
>> all of those in _memview.
>>
>> I see only two cases in ExprNodes of code checking entry.utility_code
>> and using the utilities, though. So if I understand correctly, it's
>> about those few cases?

I see three or four (depending on how you count), and the main one of them 
is in NameNode.analyse_types(), which is way too early in the pipeline.

The current problem with NameNode is that it usually just executes 
calculate_result_code() during code generation (at least for all 
interesting cases), so it has no way to inject utility code at that point.

Also, "scope.entries" and "entry.used" may be outdated at code generation 
time due to dead code removal (for early created entries) and post-analysis 
optimisations.


> The code in the pipeline is tailored for CythonUtilityCode only, so I
> suppose it would need to be modified.

Sure. It would happen at a point between optimisations and code generation, 
and it would walk the whole tree, take a look at all NameNodes and 
AttributeNodes that have entries marked as used, and inject their utility 
code into the environment. AFAICT, the current pipeline step is just a 
special case of that.

I also thought about making that a dedicated "inject_utility_code" or 
"inject_external_code" phase that nodes can interact with, i.e. we'd call a 
method on each node and it would either do nothing or inject some code into 
the global scope. However, that's obviously much more costly and likely not 
needed. Most nodes can just inject their required code during code 
generation themselves. It's mainly a problem with NameNode and friends.


> I believe the _memview branch is
> nearly finished though, so could you wait for that merge? Otherwise
> modifying the pipeline wouldn't be too bad either, it might only give
> a few conflicts.

I think it can wait a little longer.

Stefan

From stefan_ml at behnel.de  Mon Aug 22 13:36:33 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 22 Aug 2011 13:36:33 +0200
Subject: [Cython] Problems with decorated methods in cdef classes
In-Reply-To: <CADiQ+QDVOmNxwGfOzAdwctCpXaDMXfZ9adqq9GVpL_8AY79VKg@mail.gmail.com>
References: <4E47E37B.5010706@behnel.de>
	<CADiQ+QDVOmNxwGfOzAdwctCpXaDMXfZ9adqq9GVpL_8AY79VKg@mail.gmail.com>
Message-ID: <4E523F41.8040908@behnel.de>

Robert Bradshaw, 17.08.2011 08:02:
> On Sun, Aug 14, 2011 at 8:02 AM, Stefan Behnel wrote:
>> def print_args(func):
>>     def f(*args, **kwds):
>>         print "args", args, "kwds", kwds
>>         return func(*args, **kwds)
>>     return f
>>
>> cdef class Num:
>>     @print_args
>>     def is_prime(self, bint print_factors=False):
>>         ...
>>
>> Now, the problem is that Cython considers "is_prime" to be a method of a
>> cdef class, although it actually is not. It's only an arbitrary function
>> that happens to be defined inside of a cdef class body and that happens to
>> be *called* by a method, namely "f". It now crashes for me because the
>> "self" argument is not being passed into is_prime() as a C method argument
>> when called by the wrapper function - and that's correct, because it's not a
>> method call but a regular function call at that point.
>>
>> The correct way to fix this is to turn all decorated methods in cdef classes
>> into plain functions. However, this has huge drawbacks, especially that the
>> first argument ('self') can no longer be typed as the surrounding extension
>> type. But, after all, you could do this:
>>
>> def swap_args(func):
>>     def f(*args):
>>         return func(*args[::-1])
>>     return f
>>
>> cdef class Num:
>>     @swap_args
>>     def is_prime(arg, self):
>>         ...
>>
>> I'm not sure what to make of this. Does it make sense to go this route? Or
>> does anyone see a way to make this "mostly" work, e.g. by somehow
>> restricting cdef classes and their methods? Or should we just add runtime
>> checks to prevent bad behaviour of decorators?
>
> I would be happy in making decorated methods into "ordinary"
> functions--this will probably play more nicely with many real-world
> decorators as well.

I created a ticket for this:

http://trac.cython.org/cython_trac/ticket/719

Stefan

From stefan_ml at behnel.de  Mon Aug 22 13:37:23 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 22 Aug 2011 13:37:23 +0200
Subject: [Cython] Entry.utility_code_definitions vs. Entry.utility_code
In-Reply-To: <4E50FBAF.9080006@behnel.de>
References: <4E50DE99.1090104@behnel.de>	<CANg26EV-Nw71uV9QqLaqkJuQAhuhL+T6Ny6hmAKvFwzW9Y5ksQ@mail.gmail.com>	<CANg26EUr7GyNk7CthAtf7Ur2=fvnwsiqZ1p=G3OEmzi7sjKkGw@mail.gmail.com>
	<4E50FBAF.9080006@behnel.de>
Message-ID: <4E523F73.8000706@behnel.de>

Stefan Behnel, 21.08.2011 14:35:
> mark florisson, 21.08.2011 12:57:
>> I believe the _memview branch is
>> nearly finished though, so could you wait for that merge? Otherwise
>> modifying the pipeline wouldn't be too bad either, it might only give
>> a few conflicts.
>
> I think it can wait a little longer.

http://trac.cython.org/cython_trac/ticket/718

Stefan

From stefan_ml at behnel.de  Mon Aug 22 22:11:05 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 22 Aug 2011 22:11:05 +0200
Subject: [Cython] Fwd: PEP 393 Summer of Code Project
Message-ID: <4E52B7D9.6010008@behnel.de>

Sounds like he's at a point where this becomes interesting for us.

Stefan

-------- Original-Message --------
Subject: PEP 393 Summer of Code Project
Date: Mon, 22 Aug 2011 14:58:51 -0400
From: Torsten Becker <torsten.becker... at gmail.com>
To: python-dev... at python.org

Hello all,

I have implemented an initial version of PEP 393 -- "Flexible String
Representation" as part of my Google Summer of Code project.  My patch
is hosted as a repository on bitbucket [1] and I created a related
issue on the bug tracker [2].  I posted documentation for the current
state of the development in the wiki [3].

Current tests show a potential reduction of memory by about 20% and
CPU by 50% for a join micro benchmark.  Starting a new interpreter
still causes 3244 calls to create compatibility Py_UNICODE
representations, 263 strings are created using the old API while 62719
are created using the new API.  More measurements are on the wiki page
[3].

If there is interest, I would like to continue working on the patch
with the goal of getting it into Python 3.3.  Any and all feedback is
welcome.


Regards,
Torsten

[1]: http://www.python.org/dev/peps/pep-0393
[2]: http://bugs.python.org/issue12819
[3]: http://wiki.python.org/moin/SummerOfCode/2011/PEP393

From vitja.makarov at gmail.com  Tue Aug 23 06:26:27 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 23 Aug 2011 08:26:27 +0400
Subject: [Cython] pyregr testsuite regression
Message-ID: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>

Hudson shows regression for last 4 pyregr builds.

-- 
vitja.

From stefan_ml at behnel.de  Tue Aug 23 06:41:54 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Aug 2011 06:41:54 +0200
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
Message-ID: <4E532F92.3040808@behnel.de>

Vitja Makarov, 23.08.2011 06:26:
> Hudson shows regression for last 4 pyregr builds.

Yes, I enabled the "always_allow_keywords" option for better Python 
compatibility, which disables the signature optimisation into METH_O & 
friends for def functions. That helped in a couple of cases but broke 
others. The regression appears to be mostly because generator expressions 
do not work with this option:

http://trac.cython.org/cython_trac/ticket/720

CyFunction may or may not drop the need for that option eventually, but for 
now, it would be good to fix it.

I also wouldn't object to switching it off again, in case it's not an easy 
fix (I tried but didn't find the right place to fix it).

Stefan

From vitja.makarov at gmail.com  Tue Aug 23 07:06:34 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 23 Aug 2011 09:06:34 +0400
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <4E532F92.3040808@behnel.de>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
	<4E532F92.3040808@behnel.de>
Message-ID: <CAKGHGPRbqx_Q+NHShgRJiR2na-pPbMu2dEpZrVMCSqmLh1EGUw@mail.gmail.com>

2011/8/23 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 23.08.2011 06:26:
>>
>> Hudson shows regression for last 4 pyregr builds.
>
> Yes, I enabled the "always_allow_keywords" option for better Python
> compatibility, which disables the signature optimisation into METH_O &
> friends for def functions. That helped in a couple of cases but broke
> others. The regression appears to be mostly because generator expressions do
> not work with this option:
>
> http://trac.cython.org/cython_trac/ticket/720
>
> CyFunction may or may not drop the need for that option eventually, but for
> now, it would be good to fix it.
>
> I also wouldn't object to switching it off again, in case it's not an easy
> fix (I tried but didn't find the right place to fix it).
>

Ok. I'll take a look later.


-- 
vitja.

From robertwb at math.washington.edu  Tue Aug 23 07:15:38 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Aug 2011 22:15:38 -0700
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <4E532F92.3040808@behnel.de>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
	<4E532F92.3040808@behnel.de>
Message-ID: <CADiQ+QA4J1ba4YC0gQY0vsR7Ub3eEUSdAEB52SioeQvsmZZBZA@mail.gmail.com>

On Mon, Aug 22, 2011 at 9:41 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 23.08.2011 06:26:
>>
>> Hudson shows regression for last 4 pyregr builds.
>
> Yes, I enabled the "always_allow_keywords" option for better Python
> compatibility, which disables the signature optimisation into METH_O &
> friends for def functions. That helped in a couple of cases but broke
> others. The regression appears to be mostly because generator expressions do
> not work with this option:
>
> http://trac.cython.org/cython_trac/ticket/720
>
> CyFunction may or may not drop the need for that option eventually, but for
> now, it would be good to fix it.
>
> I also wouldn't object to switching it off again, in case it's not an easy
> fix (I tried but didn't find the right place to fix it).

IIRC, it is a significant performance penalty to turn that off (which
is why it's disabled by default). We should at least allow one to
disable it.

- Robert

From vitja.makarov at gmail.com  Tue Aug 23 07:20:43 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 23 Aug 2011 09:20:43 +0400
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <CADiQ+QA4J1ba4YC0gQY0vsR7Ub3eEUSdAEB52SioeQvsmZZBZA@mail.gmail.com>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
	<4E532F92.3040808@behnel.de>
	<CADiQ+QA4J1ba4YC0gQY0vsR7Ub3eEUSdAEB52SioeQvsmZZBZA@mail.gmail.com>
Message-ID: <CAKGHGPQa0108m=8gFqj4RxzRD64=wsJ6tLAaQgF1p=wyw6EugQ@mail.gmail.com>

2011/8/23 Robert Bradshaw <robertwb at math.washington.edu>:
> On Mon, Aug 22, 2011 at 9:41 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Vitja Makarov, 23.08.2011 06:26:
>>>
>>> Hudson shows regression for last 4 pyregr builds.
>>
>> Yes, I enabled the "always_allow_keywords" option for better Python
>> compatibility, which disables the signature optimisation into METH_O &
>> friends for def functions. That helped in a couple of cases but broke
>> others. The regression appears to be mostly because generator expressions do
>> not work with this option:
>>
>> http://trac.cython.org/cython_trac/ticket/720
>>
>> CyFunction may or may not drop the need for that option eventually, but for
>> now, it would be good to fix it.
>>
>> I also wouldn't object to switching it off again, in case it's not an easy
>> fix (I tried but didn't find the right place to fix it).
>
> IIRC, it is a significant performance penalty to turn that off (which
> is why it's disabled by default). We should at least allow one to
> disable it.
>

That's switched on only for pyregr tests


-- 
vitja.

From vitja.makarov at gmail.com  Tue Aug 23 07:29:12 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 23 Aug 2011 09:29:12 +0400
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <CAKGHGPQa0108m=8gFqj4RxzRD64=wsJ6tLAaQgF1p=wyw6EugQ@mail.gmail.com>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>
	<4E532F92.3040808@behnel.de>
	<CADiQ+QA4J1ba4YC0gQY0vsR7Ub3eEUSdAEB52SioeQvsmZZBZA@mail.gmail.com>
	<CAKGHGPQa0108m=8gFqj4RxzRD64=wsJ6tLAaQgF1p=wyw6EugQ@mail.gmail.com>
Message-ID: <CAKGHGPTWACiZ94JmE4yJKtkTxYYzFSW6cbynyzyjQOnEyEpUnw@mail.gmail.com>

2011/8/23 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/23 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Mon, Aug 22, 2011 at 9:41 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Vitja Makarov, 23.08.2011 06:26:
>>>>
>>>> Hudson shows regression for last 4 pyregr builds.
>>>
>>> Yes, I enabled the "always_allow_keywords" option for better Python
>>> compatibility, which disables the signature optimisation into METH_O &
>>> friends for def functions. That helped in a couple of cases but broke
>>> others. The regression appears to be mostly because generator expressions do
>>> not work with this option:
>>>
>>> http://trac.cython.org/cython_trac/ticket/720
>>>
>>> CyFunction may or may not drop the need for that option eventually, but for
>>> now, it would be good to fix it.
>>>
>>> I also wouldn't object to switching it off again, in case it's not an easy
>>> fix (I tried but didn't find the right place to fix it).
>>
>> IIRC, it is a significant performance penalty to turn that off (which
>> is why it's disabled by default). We should at least allow one to
>> disable it.
>>
>
> That's switched on only for pyregr tests
>

I've fixed it here
https://github.com/cython/cython/commit/7c01e8488e12f5e7581a356df5882d5329457369

-- 
vitja.

From stefan_ml at behnel.de  Tue Aug 23 08:15:09 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Aug 2011 08:15:09 +0200
Subject: [Cython] pyregr testsuite regression
In-Reply-To: <CAKGHGPTWACiZ94JmE4yJKtkTxYYzFSW6cbynyzyjQOnEyEpUnw@mail.gmail.com>
References: <CAKGHGPQF=9F5kUBooEMHkqN4AWKcGdcGDKv1dmFVq9Nra+So2Q@mail.gmail.com>	<4E532F92.3040808@behnel.de>	<CADiQ+QA4J1ba4YC0gQY0vsR7Ub3eEUSdAEB52SioeQvsmZZBZA@mail.gmail.com>	<CAKGHGPQa0108m=8gFqj4RxzRD64=wsJ6tLAaQgF1p=wyw6EugQ@mail.gmail.com>
	<CAKGHGPTWACiZ94JmE4yJKtkTxYYzFSW6cbynyzyjQOnEyEpUnw@mail.gmail.com>
Message-ID: <4E53456D.2080403@behnel.de>

Vitja Makarov, 23.08.2011 07:29:
> 2011/8/23 Vitja Makarov:
>> 2011/8/23 Robert Bradshaw:
>>> On Mon, Aug 22, 2011 at 9:41 PM, Stefan Behnel wrote:
>>>> Vitja Makarov, 23.08.2011 06:26:
>>>>>
>>>>> Hudson shows regression for last 4 pyregr builds.
>>>>
>>>> Yes, I enabled the "always_allow_keywords" option for better Python
>>>> compatibility, which disables the signature optimisation into METH_O&
>>>> friends for def functions. That helped in a couple of cases but broke
>>>> others. The regression appears to be mostly because generator expressions do
>>>> not work with this option:
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/720
>>>>
>>>> CyFunction may or may not drop the need for that option eventually, but for
>>>> now, it would be good to fix it.
>>>>
>>>> I also wouldn't object to switching it off again, in case it's not an easy
>>>> fix (I tried but didn't find the right place to fix it).
>>>
>>> IIRC, it is a significant performance penalty to turn that off (which
>>> is why it's disabled by default). We should at least allow one to
>>> disable it.
>>
>> That's switched on only for pyregr tests

Right, but it's still a useful option for Python compatibility in general.


> I've fixed it here
> https://github.com/cython/cython/commit/7c01e8488e12f5e7581a356df5882d5329457369

Ah, given the right pair of eyeballs ...

Thanks!

Stefan

From stefano.k.sanfilippo at gmail.com  Tue Aug 23 23:16:00 2011
From: stefano.k.sanfilippo at gmail.com (Stefano)
Date: Tue, 23 Aug 2011 23:16:00 +0200
Subject: [Cython]  Away: automagical (c)import plans
Message-ID: <201108232316.00615@a>

Hi folks,

I'm not dead, I'm just on holiday :)
Meanwhile, I've started investigating the CEP I'll be working on 
(http://wiki.cython.org/enhancements/overlaypythonmodules).

Brainstorming: if I understood well (code is not documented) I'll have to work 
on Cython.Compiler.SymTab, in particular, on Scope and Entry classes. That 
doesn't seem difficult, but I won't say so until work is over :D

Regards,

--SKS

From stefan_ml at behnel.de  Wed Aug 24 08:19:06 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Aug 2011 08:19:06 +0200
Subject: [Cython] Away: automagical (c)import plans
In-Reply-To: <201108232316.00615@a>
References: <201108232316.00615@a>
Message-ID: <4E5497DA.7090407@behnel.de>

Stefano, 23.08.2011 23:16:
> I'm not dead, I'm just on holiday :)
> Meanwhile, I've started investigating the CEP I'll be working on
> (http://wiki.cython.org/enhancements/overlaypythonmodules).
>
> Brainstorming: if I understood well (code is not documented) I'll have to work
> on Cython.Compiler.SymTab, in particular, on Scope and Entry classes. That
> doesn't seem difficult, but I won't say so until work is over :D

My guess is that you'll rather have to work with them than on them. I don't 
even think you need a new scope class to implement the fall-through. It 
should be enough to use the pxd's ModuleScope and add a Python module's 
ModuleScope as parent_scope.

I'm not sure what happens when you currently cimport *and* import a module 
(e.g. numpy), that may account for a similar setup already.

I would expect that most of what you have to change applies to the parser 
and somewhat to the ModuleScope. Those are the places where .?xd file 
search happens.

Stefan

From vitja.makarov at gmail.com  Wed Aug 24 21:00:12 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 24 Aug 2011 23:00:12 +0400
Subject: [Cython] non-virtual methods
Message-ID: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>

Recently I used cython in my project and I came with idea that
sometimes virtual methods are overkill.

What's about adding non-virtual decorator for c[p]def methods?

cdef class Foo:
     @cython.nonvirtual
     cpdef int is_active(self):
             return clib.obj_is_active(self._cobj)

-- 
vitja.

From stefan_ml at behnel.de  Wed Aug 24 21:06:17 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Aug 2011 21:06:17 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
Message-ID: <4E554BA9.3020309@behnel.de>

Vitja Makarov, 24.08.2011 21:00:
> Recently I used cython in my project and I came with idea that
> sometimes virtual methods are overkill.
>
> What's about adding non-virtual decorator for c[p]def methods?
>
> cdef class Foo:
>       @cython.nonvirtual
>       cpdef int is_active(self):
>               return clib.obj_is_active(self._cobj)

How does that differ from the "final" decorator?

Stefan

From robertwb at math.washington.edu  Wed Aug 24 21:06:50 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 24 Aug 2011 12:06:50 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
Message-ID: <CADiQ+QDcQ6t5R8w55hzVjNPPga1=4FLXLMHtOZT_kP0vkr2xFg@mail.gmail.com>

On Wed, Aug 24, 2011 at 12:00 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Recently I used cython in my project and I came with idea that
> sometimes virtual methods are overkill.
>
> What's about adding non-virtual decorator for c[p]def methods?
>
> cdef class Foo:
> ? ? @cython.nonvirtual
> ? ? cpdef int is_active(self):
> ? ? ? ? ? ? return clib.obj_is_active(self._cobj)

We have toyed with the idea of using a "final" modifier. Note that
this would only work for a final class or a pure cdef method.

- Robert

From vitja.makarov at gmail.com  Wed Aug 24 21:17:52 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 24 Aug 2011 23:17:52 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E554BA9.3020309@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
Message-ID: <CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>

2011/8/24 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 24.08.2011 21:00:
>>
>> Recently I used cython in my project and I came with idea that
>> sometimes virtual methods are overkill.
>>
>> What's about adding non-virtual decorator for c[p]def methods?
>>
>> cdef class Foo:
>> ? ? ?@cython.nonvirtual
>> ? ? ?cpdef int is_active(self):
>> ? ? ? ? ? ? ?return clib.obj_is_active(self._cobj)
>
> How does that differ from the "final" decorator?
>

I tried final classes:
1. Final decorator doesn't work at pxd level (compiler crash)
2. In this example foo call is done through virtual table

cimport cython

@cython.final
cdef class Foo:
    cdef foo(self):
        print 'haha'

def test():
    cdef Foo a = Foo()
    a.foo()

  __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
*)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}

3. I can't use final decorator for methods (error reported)


-- 
vitja.

From robertwb at math.washington.edu  Wed Aug 24 21:20:30 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 24 Aug 2011 12:20:30 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
Message-ID: <CADiQ+QB9hU+_FENjwYQGGymiOC9+4b3ThBRvPiYd7g=gRodCRg@mail.gmail.com>

On Wed, Aug 24, 2011 at 12:17 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/8/24 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 24.08.2011 21:00:
>>>
>>> Recently I used cython in my project and I came with idea that
>>> sometimes virtual methods are overkill.
>>>
>>> What's about adding non-virtual decorator for c[p]def methods?
>>>
>>> cdef class Foo:
>>> ? ? ?@cython.nonvirtual
>>> ? ? ?cpdef int is_active(self):
>>> ? ? ? ? ? ? ?return clib.obj_is_active(self._cobj)
>>
>> How does that differ from the "final" decorator?
>>
>
> I tried final classes:
> 1. Final decorator doesn't work at pxd level (compiler crash)
> 2. In this example foo call is done through virtual table
>
> cimport cython
>
> @cython.final
> cdef class Foo:
> ? ?cdef foo(self):
> ? ? ? ?print 'haha'
>
> def test():
> ? ?cdef Foo a = Foo()
> ? ?a.foo()
>
> ?__pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
>
> 3. I can't use final decorator for methods (error reported)

I don't think "final methods" were actually implemented, it's just a
question of naming.

- Robert

From vitja.makarov at gmail.com  Wed Aug 24 21:20:56 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 24 Aug 2011 23:20:56 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
Message-ID: <CAKGHGPT1nm85k4TK5qmwSxsabhcaQmHpTa4rr5tbH7sDgbm1uQ@mail.gmail.com>

2011/8/24 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/24 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 24.08.2011 21:00:
>>>
>>> Recently I used cython in my project and I came with idea that
>>> sometimes virtual methods are overkill.
>>>
>>> What's about adding non-virtual decorator for c[p]def methods?
>>>
>>> cdef class Foo:
>>> ? ? ?@cython.nonvirtual
>>> ? ? ?cpdef int is_active(self):
>>> ? ? ? ? ? ? ?return clib.obj_is_active(self._cobj)
>>
>> How does that differ from the "final" decorator?
>>
>
> I tried final classes:
> 1. Final decorator doesn't work at pxd level (compiler crash)
> 2. In this example foo call is done through virtual table
>
> cimport cython
>
> @cython.final
> cdef class Foo:
> ? ?cdef foo(self):
> ? ? ? ?print 'haha'
>
> def test():
> ? ?cdef Foo a = Foo()
> ? ?a.foo()
>
> ?__pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
>
> 3. I can't use final decorator for methods (error reported)
>
>

I think this ticket is about it http://trac.cython.org/cython_trac/ticket/586

-- 
vitja.

From stefan_ml at behnel.de  Wed Aug 24 21:27:00 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Aug 2011 21:27:00 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
Message-ID: <4E555084.2000602@behnel.de>

Vitja Makarov, 24.08.2011 21:17:
> 2011/8/24 Stefan Behnel:
>> Vitja Makarov, 24.08.2011 21:00:
>>>
>>> Recently I used cython in my project and I came with idea that
>>> sometimes virtual methods are overkill.
>>>
>>> What's about adding non-virtual decorator for c[p]def methods?
>>>
>>> cdef class Foo:
>>>       @cython.nonvirtual
>>>       cpdef int is_active(self):
>>>               return clib.obj_is_active(self._cobj)
>>
>> How does that differ from the "final" decorator?
>>
>
> I tried final classes:
> 1. Final decorator doesn't work at pxd level (compiler crash)

Sounds like a bug.


> 2. In this example foo call is done through virtual table
>
> cimport cython
>
> @cython.final
> cdef class Foo:
>      cdef foo(self):
>          print 'haha'
>
> def test():
>      cdef Foo a = Foo()
>      a.foo()
>
>    __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}

Right, this is not implemented yet. Feel free to do so. Also see

http://trac.cython.org/cython_trac/ticket/474


> 3. I can't use final decorator for methods (error reported)

http://trac.cython.org/cython_trac/ticket/586

Stefan

From vitja.makarov at gmail.com  Thu Aug 25 18:11:31 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Aug 2011 20:11:31 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E555084.2000602@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
Message-ID: <CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>

2011/8/24 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 24.08.2011 21:17:
>>
>> 2011/8/24 Stefan Behnel:
>>>
>>> Vitja Makarov, 24.08.2011 21:00:
>>>>
>>>> Recently I used cython in my project and I came with idea that
>>>> sometimes virtual methods are overkill.
>>>>
>>>> What's about adding non-virtual decorator for c[p]def methods?
>>>>
>>>> cdef class Foo:
>>>> ? ? ?@cython.nonvirtual
>>>> ? ? ?cpdef int is_active(self):
>>>> ? ? ? ? ? ? ?return clib.obj_is_active(self._cobj)
>>>
>>> How does that differ from the "final" decorator?
>>>
>>
>> I tried final classes:
>> 1. Final decorator doesn't work at pxd level (compiler crash)
>
> Sounds like a bug.
>
>
>> 2. In this example foo call is done through virtual table
>>
>> cimport cython
>>
>> @cython.final
>> cdef class Foo:
>> ? ? cdef foo(self):
>> ? ? ? ? print 'haha'
>>
>> def test():
>> ? ? cdef Foo a = Foo()
>> ? ? a.foo()
>>
>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>> __LINE__; goto __pyx_L1_error;}
>
> Right, this is not implemented yet. Feel free to do so. Also see
>
> http://trac.cython.org/cython_trac/ticket/474
>
>
>> 3. I can't use final decorator for methods (error reported)
>
> http://trac.cython.org/cython_trac/ticket/586
>

What is the prefered syntax keyword inline or final decorator?


-- 
vitja.

From stefan_ml at behnel.de  Thu Aug 25 19:08:16 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Aug 2011 19:08:16 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E554BA9.3020309@behnel.de>	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
Message-ID: <4E568180.5030104@behnel.de>

Vitja Makarov, 25.08.2011 18:11:
> 2011/8/24 Stefan Behnel:
>> Vitja Makarov, 24.08.2011 21:17:
>>> I tried final classes:
>>> 2. In this example foo call is done through virtual table
>>>
>>> cimport cython
>>>
>>> @cython.final
>>> cdef class Foo:
>>>      cdef foo(self):
>>>          print 'haha'
>>>
>>> def test():
>>>      cdef Foo a = Foo()
>>>      a.foo()
>>>
>>>    __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>> __LINE__; goto __pyx_L1_error;}
>>
>> Right, this is not implemented yet. Feel free to do so. Also see
>>
>> http://trac.cython.org/cython_trac/ticket/474
>>
>>
>>> 3. I can't use final decorator for methods (error reported)
>>
>> http://trac.cython.org/cython_trac/ticket/586
>
> What is the prefered syntax keyword inline or final decorator?

"final" is the right option here.

They are orthogonal concepts. Only because you declare a method "final" 
does not mean you want to inline it, and just because you declare it 
"inline" does not (necessarily) mean that you cannot override it. 
Admittedly, the semantics of an overridable inline method may turn out to 
be somewhat obscure and error prone, so I think it's a good idea to let 
"inline" imply "final". But not the other way round.

Stefan

From vitja.makarov at gmail.com  Thu Aug 25 20:32:19 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Aug 2011 22:32:19 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E568180.5030104@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
Message-ID: <CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>

2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.08.2011 18:11:
>>
>> 2011/8/24 Stefan Behnel:
>>>
>>> Vitja Makarov, 24.08.2011 21:17:
>>>>
>>>> I tried final classes:
>>>> 2. In this example foo call is done through virtual table
>>>>
>>>> cimport cython
>>>>
>>>> @cython.final
>>>> cdef class Foo:
>>>> ? ? cdef foo(self):
>>>> ? ? ? ? print 'haha'
>>>>
>>>> def test():
>>>> ? ? cdef Foo a = Foo()
>>>> ? ? a.foo()
>>>>
>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>> __LINE__; goto __pyx_L1_error;}
>>>
>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>
>>> http://trac.cython.org/cython_trac/ticket/474
>>>
>>>
>>>> 3. I can't use final decorator for methods (error reported)
>>>
>>> http://trac.cython.org/cython_trac/ticket/586
>>
>> What is the prefered syntax keyword inline or final decorator?
>
> "final" is the right option here.
>
> They are orthogonal concepts. Only because you declare a method "final" does
> not mean you want to inline it, and just because you declare it "inline"
> does not (necessarily) mean that you cannot override it. Admittedly, the
> semantics of an overridable inline method may turn out to be somewhat
> obscure and error prone, so I think it's a good idea to let "inline" imply
> "final". But not the other way round.
>

But both inline and final methods should bypass vtab, right? Also I'm
not sure about C inline qualifier here.

I see three options:

 - non-virtual: bypass vtab
 - final: non-virtual, non-overridable
 - inline: non-virtual, C inline qualifier is used

-- 
vitja.

From stefan_ml at behnel.de  Thu Aug 25 20:58:03 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Aug 2011 20:58:03 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E554BA9.3020309@behnel.de>	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>	<4E555084.2000602@behnel.de>	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
Message-ID: <4E569B3B.8060201@behnel.de>

Vitja Makarov, 25.08.2011 20:32:
> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>> Vitja Makarov, 25.08.2011 18:11:
>>>
>>> 2011/8/24 Stefan Behnel:
>>>>
>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>
>>>>> I tried final classes:
>>>>> 2. In this example foo call is done through virtual table
>>>>>
>>>>> cimport cython
>>>>>
>>>>> @cython.final
>>>>> cdef class Foo:
>>>>>      cdef foo(self):
>>>>>          print 'haha'
>>>>>
>>>>> def test():
>>>>>      cdef Foo a = Foo()
>>>>>      a.foo()
>>>>>
>>>>>    __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>> __LINE__; goto __pyx_L1_error;}
>>>>
>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>
>>>>
>>>>> 3. I can't use final decorator for methods (error reported)
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/586
>>>
>>> What is the prefered syntax keyword inline or final decorator?
>>
>> "final" is the right option here.
>>
>> They are orthogonal concepts. Only because you declare a method "final" does
>> not mean you want to inline it, and just because you declare it "inline"
>> does not (necessarily) mean that you cannot override it. Admittedly, the
>> semantics of an overridable inline method may turn out to be somewhat
>> obscure and error prone, so I think it's a good idea to let "inline" imply
>> "final". But not the other way round.
>
> But both inline and final methods should bypass vtab, right?

Yes. But in the "final" case, it's always clear which method implementation 
to use - it's not overridable, so there is only one choice. In the "inline" 
case, it could still be overridable and we may have a subtype of the 
declared type in our hands at runtime, thus choosing the wrong method at 
compile time. That's why only the "final" case is safe.

Note that I'm only talking about the semantics of the qualifier themselves 
here. If we allow "inline" methods, I think we should force them to be 
"final" as well. But that's a practical choice, not a semantic implication.


> Also I'm not sure about C inline qualifier here.

That's what "inline" requests.


> I see three options:
>
>   - non-virtual: bypass vtab
>   - final: non-virtual, non-overridable

How would you want to bypass the vtable in the "non-virtual" case if the 
method is overridable?


>   - inline: non-virtual, C inline qualifier is used

Correct.

Stefan

From vitja.makarov at gmail.com  Thu Aug 25 21:07:53 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Aug 2011 23:07:53 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E569B3B.8060201@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
Message-ID: <CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>

2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.08.2011 20:32:
>>
>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 25.08.2011 18:11:
>>>>
>>>> 2011/8/24 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>
>>>>>> I tried final classes:
>>>>>> 2. In this example foo call is done through virtual table
>>>>>>
>>>>>> cimport cython
>>>>>>
>>>>>> @cython.final
>>>>>> cdef class Foo:
>>>>>> ? ? cdef foo(self):
>>>>>> ? ? ? ? print 'haha'
>>>>>>
>>>>>> def test():
>>>>>> ? ? cdef Foo a = Foo()
>>>>>> ? ? a.foo()
>>>>>>
>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>
>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>
>>>>>
>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>
>>>> What is the prefered syntax keyword inline or final decorator?
>>>
>>> "final" is the right option here.
>>>
>>> They are orthogonal concepts. Only because you declare a method "final"
>>> does
>>> not mean you want to inline it, and just because you declare it "inline"
>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>> semantics of an overridable inline method may turn out to be somewhat
>>> obscure and error prone, so I think it's a good idea to let "inline"
>>> imply
>>> "final". But not the other way round.
>>
>> But both inline and final methods should bypass vtab, right?
>
> Yes. But in the "final" case, it's always clear which method implementation
> to use - it's not overridable, so there is only one choice. In the "inline"
> case, it could still be overridable and we may have a subtype of the
> declared type in our hands at runtime, thus choosing the wrong method at
> compile time. That's why only the "final" case is safe.
>
> Note that I'm only talking about the semantics of the qualifier themselves
> here. If we allow "inline" methods, I think we should force them to be
> "final" as well. But that's a practical choice, not a semantic implication.
>
>
>> Also I'm not sure about C inline qualifier here.
>
> That's what "inline" requests.
>
>
>> I see three options:
>>
>> ?- non-virtual: bypass vtab
>> ?- final: non-virtual, non-overridable
>
> How would you want to bypass the vtable in the "non-virtual" case if the
> method is overridable?
>
>
>> ?- inline: non-virtual, C inline qualifier is used
>
> Correct.
>

Ok.

I think it's better to implement final method then user could choose
to use inline qualifier or not.

-- 
vitja.

From vitja.makarov at gmail.com  Thu Aug 25 21:56:36 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Aug 2011 23:56:36 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
Message-ID: <CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>

2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 25.08.2011 20:32:
>>>
>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>
>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>
>>>>> 2011/8/24 Stefan Behnel:
>>>>>>
>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>
>>>>>>> I tried final classes:
>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>
>>>>>>> cimport cython
>>>>>>>
>>>>>>> @cython.final
>>>>>>> cdef class Foo:
>>>>>>> ? ? cdef foo(self):
>>>>>>> ? ? ? ? print 'haha'
>>>>>>>
>>>>>>> def test():
>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>> ? ? a.foo()
>>>>>>>
>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>
>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>
>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>
>>>>>>
>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>
>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>
>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>
>>>> "final" is the right option here.
>>>>
>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>> does
>>>> not mean you want to inline it, and just because you declare it "inline"
>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>> semantics of an overridable inline method may turn out to be somewhat
>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>> imply
>>>> "final". But not the other way round.
>>>
>>> But both inline and final methods should bypass vtab, right?
>>
>> Yes. But in the "final" case, it's always clear which method implementation
>> to use - it's not overridable, so there is only one choice. In the "inline"
>> case, it could still be overridable and we may have a subtype of the
>> declared type in our hands at runtime, thus choosing the wrong method at
>> compile time. That's why only the "final" case is safe.
>>
>> Note that I'm only talking about the semantics of the qualifier themselves
>> here. If we allow "inline" methods, I think we should force them to be
>> "final" as well. But that's a practical choice, not a semantic implication.
>>
>>
>>> Also I'm not sure about C inline qualifier here.
>>
>> That's what "inline" requests.
>>
>>
>>> I see three options:
>>>
>>> ?- non-virtual: bypass vtab
>>> ?- final: non-virtual, non-overridable
>>
>> How would you want to bypass the vtable in the "non-virtual" case if the
>> method is overridable?
>>
>>
>>> ?- inline: non-virtual, C inline qualifier is used
>>
>> Correct.
>>
>
> Ok.
>
> I think it's better to implement final method then user could choose
> to use inline qualifier or not.
>

I tried it here:
https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02

There is one problem: vtab bypassing should be enabled if final method
is defined in the same module.
I don't know how to check that final method comes from cimport (it's
okay with pxd, the problem is pyx)

-- 
vitja.

From vitja.makarov at gmail.com  Fri Aug 26 06:30:20 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 26 Aug 2011 08:30:20 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
Message-ID: <CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>

2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>> Vitja Makarov, 25.08.2011 20:32:
>>>>
>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>
>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>
>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>
>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>
>>>>>>>> I tried final classes:
>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>
>>>>>>>> cimport cython
>>>>>>>>
>>>>>>>> @cython.final
>>>>>>>> cdef class Foo:
>>>>>>>> ? ? cdef foo(self):
>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>
>>>>>>>> def test():
>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>> ? ? a.foo()
>>>>>>>>
>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>
>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>
>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>
>>>>>>>
>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>
>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>
>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>
>>>>> "final" is the right option here.
>>>>>
>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>> does
>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>> imply
>>>>> "final". But not the other way round.
>>>>
>>>> But both inline and final methods should bypass vtab, right?
>>>
>>> Yes. But in the "final" case, it's always clear which method implementation
>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>> case, it could still be overridable and we may have a subtype of the
>>> declared type in our hands at runtime, thus choosing the wrong method at
>>> compile time. That's why only the "final" case is safe.
>>>
>>> Note that I'm only talking about the semantics of the qualifier themselves
>>> here. If we allow "inline" methods, I think we should force them to be
>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>
>>>
>>>> Also I'm not sure about C inline qualifier here.
>>>
>>> That's what "inline" requests.
>>>
>>>
>>>> I see three options:
>>>>
>>>> ?- non-virtual: bypass vtab
>>>> ?- final: non-virtual, non-overridable
>>>
>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>> method is overridable?
>>>
>>>
>>>> ?- inline: non-virtual, C inline qualifier is used
>>>
>>> Correct.
>>>
>>
>> Ok.
>>
>> I think it's better to implement final method then user could choose
>> to use inline qualifier or not.
>>
>
> I tried it here:
> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>
> There is one problem: vtab bypassing should be enabled if final method
> is defined in the same module.
> I don't know how to check that final method comes from cimport (it's
> okay with pxd, the problem is pyx)
>


https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425

Here is update version. I've add tree asserts and final method's prototypes.

-- 
vitja.

From vitja.makarov at gmail.com  Sat Aug 27 11:21:15 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Aug 2011 13:21:15 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
Message-ID: <CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>

2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>
>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>
>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>
>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>
>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>
>>>>>>>>> I tried final classes:
>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>
>>>>>>>>> cimport cython
>>>>>>>>>
>>>>>>>>> @cython.final
>>>>>>>>> cdef class Foo:
>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>
>>>>>>>>> def test():
>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>> ? ? a.foo()
>>>>>>>>>
>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>
>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>
>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>
>>>>>>>>
>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>
>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>
>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>
>>>>>> "final" is the right option here.
>>>>>>
>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>> does
>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>> imply
>>>>>> "final". But not the other way round.
>>>>>
>>>>> But both inline and final methods should bypass vtab, right?
>>>>
>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>> case, it could still be overridable and we may have a subtype of the
>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>> compile time. That's why only the "final" case is safe.
>>>>
>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>> here. If we allow "inline" methods, I think we should force them to be
>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>
>>>>
>>>>> Also I'm not sure about C inline qualifier here.
>>>>
>>>> That's what "inline" requests.
>>>>
>>>>
>>>>> I see three options:
>>>>>
>>>>> ?- non-virtual: bypass vtab
>>>>> ?- final: non-virtual, non-overridable
>>>>
>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>> method is overridable?
>>>>
>>>>
>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>
>>>> Correct.
>>>>
>>>
>>> Ok.
>>>
>>> I think it's better to implement final method then user could choose
>>> to use inline qualifier or not.
>>>
>>
>> I tried it here:
>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>
>> There is one problem: vtab bypassing should be enabled if final method
>> is defined in the same module.
>> I don't know how to check that final method comes from cimport (it's
>> okay with pxd, the problem is pyx)
>>
>
>
> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>
> Here is update version. I've add tree asserts and final method's prototypes.
>
> --
> vitja.
>

I've created ticket for compiler crash when cython.final is used
inside pxd file:

http://trac.cython.org/cython_trac/ticket/722

Also I've updated final methods test case (added tree path assertions)
https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74


About declaration origin detection may be it's a good idea to have a
flag at scope level something like is_pxd_scope or
is_declaration_scope?

-- 
vitja.

From vitja.makarov at gmail.com  Sat Aug 27 13:10:41 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Aug 2011 15:10:41 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
Message-ID: <CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>

2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>
>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>
>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>
>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>
>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>
>>>>>>>>>> I tried final classes:
>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>
>>>>>>>>>> cimport cython
>>>>>>>>>>
>>>>>>>>>> @cython.final
>>>>>>>>>> cdef class Foo:
>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>
>>>>>>>>>> def test():
>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>
>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>
>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>
>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>
>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>
>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>
>>>>>>> "final" is the right option here.
>>>>>>>
>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>> does
>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>> imply
>>>>>>> "final". But not the other way round.
>>>>>>
>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>
>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>> case, it could still be overridable and we may have a subtype of the
>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>> compile time. That's why only the "final" case is safe.
>>>>>
>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>
>>>>>
>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>
>>>>> That's what "inline" requests.
>>>>>
>>>>>
>>>>>> I see three options:
>>>>>>
>>>>>> ?- non-virtual: bypass vtab
>>>>>> ?- final: non-virtual, non-overridable
>>>>>
>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>> method is overridable?
>>>>>
>>>>>
>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>
>>>>> Correct.
>>>>>
>>>>
>>>> Ok.
>>>>
>>>> I think it's better to implement final method then user could choose
>>>> to use inline qualifier or not.
>>>>
>>>
>>> I tried it here:
>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>
>>> There is one problem: vtab bypassing should be enabled if final method
>>> is defined in the same module.
>>> I don't know how to check that final method comes from cimport (it's
>>> okay with pxd, the problem is pyx)
>>>
>>
>>
>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>
>> Here is update version. I've add tree asserts and final method's prototypes.
>>
>> --
>> vitja.
>>
>
> I've created ticket for compiler crash when cython.final is used
> inside pxd file:
>
> http://trac.cython.org/cython_trac/ticket/722
>
> Also I've updated final methods test case (added tree path assertions)
> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>
>
> About declaration origin detection may be it's a good idea to have a
> flag at scope level something like is_pxd_scope or
> is_declaration_scope?
>

It seems to me that I found a way to fix pxd/pyx cimport problem. I've
created pull request:

https://github.com/cython/cython/pull/59


-- 
vitja.

From stefan_ml at behnel.de  Sat Aug 27 21:55:52 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Aug 2011 21:55:52 +0200
Subject: [Cython] Thread on "with atomic" in python-dev
Message-ID: <4E594BC8.7070908@behnel.de>

Just copying this here as it correlates somewhat with the recent GIL 
discussions on this list.

http://thread.gmane.org/gmane.comp.python.devel/126336

Stefan

From vitja.makarov at gmail.com  Sun Aug 28 13:00:37 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 28 Aug 2011 15:00:37 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
Message-ID: <CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>

2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>
>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>
>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>
>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>
>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>
>>>>>>>>>>> I tried final classes:
>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>
>>>>>>>>>>> cimport cython
>>>>>>>>>>>
>>>>>>>>>>> @cython.final
>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>
>>>>>>>>>>> def test():
>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>
>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>
>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>
>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>
>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>
>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>
>>>>>>>> "final" is the right option here.
>>>>>>>>
>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>> does
>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>> imply
>>>>>>>> "final". But not the other way round.
>>>>>>>
>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>
>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>
>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>
>>>>>>
>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>
>>>>>> That's what "inline" requests.
>>>>>>
>>>>>>
>>>>>>> I see three options:
>>>>>>>
>>>>>>> ?- non-virtual: bypass vtab
>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>
>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>> method is overridable?
>>>>>>
>>>>>>
>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>
>>>>>> Correct.
>>>>>>
>>>>>
>>>>> Ok.
>>>>>
>>>>> I think it's better to implement final method then user could choose
>>>>> to use inline qualifier or not.
>>>>>
>>>>
>>>> I tried it here:
>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>
>>>> There is one problem: vtab bypassing should be enabled if final method
>>>> is defined in the same module.
>>>> I don't know how to check that final method comes from cimport (it's
>>>> okay with pxd, the problem is pyx)
>>>>
>>>
>>>
>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>
>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>
>>> --
>>> vitja.
>>>
>>
>> I've created ticket for compiler crash when cython.final is used
>> inside pxd file:
>>
>> http://trac.cython.org/cython_trac/ticket/722
>>
>> Also I've updated final methods test case (added tree path assertions)
>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>
>>
>> About declaration origin detection may be it's a good idea to have a
>> flag at scope level something like is_pxd_scope or
>> is_declaration_scope?
>>
>
> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
> created pull request:
>
> https://github.com/cython/cython/pull/59
>

I've add support for inline methods, now you can declare inline method
in pxd file:

cdef class Foo:
   cdef inline foo(self):
       return 1




-- 
vitja.

From vitja.makarov at gmail.com  Sun Aug 28 22:19:38 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 29 Aug 2011 00:19:38 +0400
Subject: [Cython] ComprehensionNode problem
Message-ID: <CAKGHGPQ2Ted5J23FbO2hoeSXXU6n4+WSBiU=XKBB0t85kDTd0w@mail.gmail.com>

I've started #715 ticket investigation.


Here is minimal test case:

# cython: language_level=3
def foo(target):
    return [(e for e in t) for t in target]

Crash in the ticket is related to GeneratorExpressionScope (name is
not correct, actually ScopedExprScope or NestedScope)

If you set languange_level to 2 you will see next error:
...
Compiler crash traceback from this point on:
  File "/home/vitja/work/cython-vitek-git/Cython/Compiler/ExprNodes.py",
line 7732, in __init__
    if not result_type.create_from_py_utility_code(env):
AttributeError: 'UnspecifiedType' object has no attribute
'create_from_py_utility_code'


Since ComprehensionNode.append and ComprehensionNode.loop.body is the
same generator body is created twice in MarkClosureVisitor

So the issue could be solved in two ways:

 - Write special handler for ComprehensionNode in MarkClosureVisitor
 - Remove append child attribute from ComprehensionNode (it can be
removed completely or just from children attribute list)


-- 
vitja.

From stefan_ml at behnel.de  Mon Aug 29 16:33:15 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 29 Aug 2011 16:33:15 +0200
Subject: [Cython] History of SWIG and applicability to Cython
Message-ID: <4E5BA32B.3030709@behnel.de>

Hi,

here's an interesting history wrap-up of SWIG, by its original author.

One thing that stroke me when I read this (or rather when I was half-way 
through) was that it might be possible to use SWIG as a glue code generator 
for .pxd files and trivial wrapper code. Not sure how hard that would be - 
it does sound like such a complex system could also be somewhat tricky to 
extend ...

Stefan


-------- Original-Message --------
Subject: SWIG (was Re: Ctypes and the stdlib)
Date: Mon, 29 Aug 2011 07:41:23 -0500
From: David Beazley <dave... at dabeaz.com>
To: python-dev... at python.org
CC: David Beazley <dave... at dabeaz.com>

On Mon, Aug 29, 2011 at 12:27 PM, Guido van Rossum <guido... at python.org> wrote:

> I wonder if for
> this particular purpose SWIG isn't the better match. (If SWIG weren't
> universally hated, even by its original author. :-)

Hate is probably a strong word, but as the author of Swig, let me chime in 
here ;-).   I think there are probably some lessons to be learned from Swig.

As Nick noted, Swig is best suited when you have control over both sides 
(C/C++ and Python) of whatever code you're working with.  In fact, the 
original motivation for  Swig was to give application programmers 
(scientists in my case), a means for automatically generating the Python 
bindings to their code.  However, there was one other important 
assumption--and that was the fact that all of your "real code" was going to 
be written in C/C++ and that the Python scripting interface was just an 
optional add-on (perhaps even just a throw-away thing).  Keep in mind, Swig 
was first created in 1995 and at that time, the use of Python (or any 
similar language) was a pretty radical idea in the sciences.  Moreover, 
there was a lot of legacy code that people just weren't going to abandon. 
Thus, I alwa
  ys viewed Swig as a kind of transitional vehicle for getting people to 
use Python who might otherwise not even consider it.   Getting back to 
Nick's point though, to really use Swig effectiv
  ely, it was always known that you might have to reorganize or refactor 
your C/C++ code to make it more Python friendly.  However, due to the 
automatic wrapper generation, you didn't have to do it all at once. 
Basically your code could organically evolve and Swig would just keep up 
with whatever you were doing.  In my projects, we'd usually just tuck Swig 
away in some Makefile somewhere and forget about it.

One of the major complexities of Swig is the fact that it attempts to parse 
C/C++ header files.   This very notion is actually a dangerous trap waiting 
for anyone who wants to wander into it.  You might look at a header file 
and say, well how hard could it be to just grab a few definitions out of 
there?   I'll just write a few regexs or come up with some simple hack for 
recognizing function definitions or something.   Yes, you can do that, but 
you're immediately going to find that whatever approach you take starts to 
break down into horrible corner cases.   Swig started out like this and 
quickly turned into a quagmire of esoteric bug reports.  All sorts of 
problems with preprocessor macros, typedefs, missing headers, and other 
things.  For awhile, I would get these bug reports that would g
  o something like "I had this C++ class inside a namespace with an 
abstract method taking a typedef'd const reference to this smart pointer 
..... and Swig broke."   Hell, I can't even underst
  and the bug report let alone know how to fix it.  Almost all of these 
bugs were due to the fact that Swig started out as a hack and didn't really 
have any kind of solid conceptual foundation for how it should be put together.

If you flash forward a bit, from about 2001-2004 there was a very serious 
push to fix these kinds of issues.  Although it was not a complete rewrite 
of Swig, there were a huge number of changes to how it worked during this 
time.  Swig grew a fully compatible C++ preprocessor that fully supported 
macros   A complete C++ type system was implemented including support for 
namespaces, templates, and even such things as template partial 
specialization.  Swig evolved into a multi-pass compiler that was doing all 
sorts of global analysis of the interface.   Just to give you an idea, Swig 
would do things such as automatically detect/wrap C++ smart pointers.  It 
could wrap overloaded C++ methods/function.  Also, if you had a C++ class 
with virtual methods, it would only make one Python wrapper funct
  ion and then reuse across all wrapped subclasses.

Under the covers of all of this, the implementation basically evolved into 
a sophisticated macro preprocessor coupled with a pattern matching engine 
built on top of the C++ type system.   For example, you could write 
patterns that matched specific C++ types (the much hated "typemap" feature) 
and you could write patterns that matched entire C++ declarations.  This 
whole pattern matching approach had a huge power if you knew what you were 
doing.  For example, I had a graduate student working on adding "contracts" 
to Swig--something that was being funded by a NSF grant.   It was cool and 
mind boggling all at once.

In hindsight however, I think the complexity of Swig has exceeded anyone's 
ability to fully understand it (including my own).  For example, to even 
make sense of what's happening, you have to have a pretty solid grasp of 
the C/C++ type system (easier said than done).   Couple that with all sorts 
of crazy pattern matching, low-level code fragments, and a ton of macro 
definitions, your head will literally explode if you try to figure out 
what's happening.   So far as I know, recent versions of Swig have even 
combined all of this type-pattern matching with regular expressions.  I 
can't even fathom it.

Sadly, my involvement was Swig was an unfortunate casualty of my academic 
career biting the dust.  By 2005, I was so burned out of working on it and 
so sick of what I was doing, I quite literally put all of my computer stuff 
aside to go play in a band for a few years.   After a few years, I came 
back to programming (obviously), but not to keep working on the same stuff. 
   In particularly, I will die quite happy if I never have to look at 
another line of C++ code ever again.  No, I would much rather fling my 
toddlers, ride my bike, play piano, or do just about anything than ever do 
that again.   Although I still subscribe the Swig mailing lists and watch 
what's happening, I'm not active with it at the moment.

I've sometimes thought it might be interesting to create a Swig replacement 
purely in Python.  When I work on the PLY project, this is often what I 
think about.   In that project, I've actually built a number of the parsing 
tools that would be useful in creating such a thing.   The only catch is 
that when I start thinking along these lines, I usually reach a point where 
I say "nah, I'll just write the whole application in Python."

Anyways, this is probably way more than anyone wants to know about Swig. 
Getting back to the original topic of using it to make standard library 
modules, I just don't know.   I think you probably could have some success 
with an automatic code generator of some kind.  I'm just not sure it should 
take the Swig approach of parsing C++ headers.  I think you could do better.

Cheers,
Dave

P.S. By the way, if people want to know a lot more about Swig internals, 
they should check out the PyCon 2008 presentation I gave about it. 
http://www.dabeaz.com/SwigMaster/


From stefan_ml at behnel.de  Mon Aug 29 19:25:51 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 29 Aug 2011 19:25:51 +0200
Subject: [Cython] History of SWIG and applicability to Cython
In-Reply-To: <4E5BA32B.3030709@behnel.de>
References: <4E5BA32B.3030709@behnel.de>
Message-ID: <4E5BCB9F.50700@behnel.de>

Stefan Behnel, 29.08.2011 16:33:
> here's an interesting history wrap-up of SWIG, by its original author.
> [...]

And an interesting reply:

http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440

I didn't know clang even had Python bindings for its parser.

Stefan

From markflorisson88 at gmail.com  Mon Aug 29 22:31:55 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 29 Aug 2011 22:31:55 +0200
Subject: [Cython] History of SWIG and applicability to Cython
In-Reply-To: <4E5BCB9F.50700@behnel.de>
References: <4E5BA32B.3030709@behnel.de>
	<4E5BCB9F.50700@behnel.de>
Message-ID: <CANg26EVjm1N6YAC2taqXaPLWzB68UaVUEZkkPUrO9S92QiecOQ@mail.gmail.com>

On 29 August 2011 19:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 29.08.2011 16:33:
>>
>> here's an interesting history wrap-up of SWIG, by its original author.
>> [...]
>
> And an interesting reply:
>
> http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440
>
> I didn't know clang even had Python bindings for its parser.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Last time I checked I think they didn't support the preprocessor yet,
and I do think you want access to macros. Macro constants are easy if
you have access to the preprocessor, I don't know about macro
functions... perhaps those should remain the user's burden. I would
certainly welcome a good pxd generator.

There is one such generator out there, can't seem to find it right
now. I think it uses gccxml (which parses C++ and generates xml).
There's also ctypesgen, which is also based on gccxml.

I think clang or gccxml would be easier and less of a hack than swig,
although I haven't looked at swig's implementation.

From cournape at gmail.com  Mon Aug 29 22:42:36 2011
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 29 Aug 2011 22:42:36 +0200
Subject: [Cython] History of SWIG and applicability to Cython
In-Reply-To: <CANg26EVjm1N6YAC2taqXaPLWzB68UaVUEZkkPUrO9S92QiecOQ@mail.gmail.com>
References: <4E5BA32B.3030709@behnel.de> <4E5BCB9F.50700@behnel.de>
	<CANg26EVjm1N6YAC2taqXaPLWzB68UaVUEZkkPUrO9S92QiecOQ@mail.gmail.com>
Message-ID: <CAGY4rcUJy4K+UNY1CAK==qwos97wTXT8uKU6jQ5o-UZhrAFkFQ@mail.gmail.com>

On Mon, Aug 29, 2011 at 10:31 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 29 August 2011 19:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Stefan Behnel, 29.08.2011 16:33:
>>>
>>> here's an interesting history wrap-up of SWIG, by its original author.
>>> [...]
>>
>> And an interesting reply:
>>
>> http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440
>>
>> I didn't know clang even had Python bindings for its parser.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> Last time I checked I think they didn't support the preprocessor yet,
> and I do think you want access to macros. Macro constants are easy if
> you have access to the preprocessor, I don't know about macro
> functions... perhaps those should remain the user's burden. I would
> certainly welcome a good pxd generator.
>
> There is one such generator out there, can't seem to find it right
> now.

I did one such thing, but it is more of an hack than a real solution:

https://github.com/cournape/cython-codegen

Using gccxml is a PITA:
  - you cannot force C mode (always assume C++)
  - building gccxml is not pleasant

clang is much better for this kind of things,

cheers,

David

From sccolbert at gmail.com  Mon Aug 29 22:55:49 2011
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 29 Aug 2011 15:55:49 -0500
Subject: [Cython] History of SWIG and applicability to Cython
In-Reply-To: <CAGY4rcUJy4K+UNY1CAK==qwos97wTXT8uKU6jQ5o-UZhrAFkFQ@mail.gmail.com>
References: <4E5BA32B.3030709@behnel.de> <4E5BCB9F.50700@behnel.de>
	<CANg26EVjm1N6YAC2taqXaPLWzB68UaVUEZkkPUrO9S92QiecOQ@mail.gmail.com>
	<CAGY4rcUJy4K+UNY1CAK==qwos97wTXT8uKU6jQ5o-UZhrAFkFQ@mail.gmail.com>
Message-ID: <CAPc_zRXX88w24g7ja-m42Buw+-Ker15aNTRrxteA3mn=wOtnKQ@mail.gmail.com>

I've been working on CWrap in my spare time.

https://github.com/enthought/cwrap

The goal is to be SWIG for Cython.

It currently outputs pxd files for the headers of a C library. It's not
mature, but it works. (It can output pxd files for the whole of Intel's IPP
and MKL libraries). I hope to eventually output pyx implementation files as
well.

It currently uses gccxml, but I hope to make the switch to Clang.

We've been talking amongst ourselves @ enthought to lobby you guys
eventually to merge it with Cython main, but I just haven't had the
bandwidth to get it up to that point.

Chris


On Mon, Aug 29, 2011 at 3:42 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Mon, Aug 29, 2011 at 10:31 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
> > On 29 August 2011 19:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
> >> Stefan Behnel, 29.08.2011 16:33:
> >>>
> >>> here's an interesting history wrap-up of SWIG, by its original author.
> >>> [...]
> >>
> >> And an interesting reply:
> >>
> >> http://thread.gmane.org/gmane.comp.python.devel/126425/focus=126440
> >>
> >> I didn't know clang even had Python bindings for its parser.
> >>
> >> Stefan
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> http://mail.python.org/mailman/listinfo/cython-devel
> >>
> >
> > Last time I checked I think they didn't support the preprocessor yet,
> > and I do think you want access to macros. Macro constants are easy if
> > you have access to the preprocessor, I don't know about macro
> > functions... perhaps those should remain the user's burden. I would
> > certainly welcome a good pxd generator.
> >
> > There is one such generator out there, can't seem to find it right
> > now.
>
> I did one such thing, but it is more of an hack than a real solution:
>
> https://github.com/cournape/cython-codegen
>
> Using gccxml is a PITA:
>  - you cannot force C mode (always assume C++)
>  - building gccxml is not pleasant
>
> clang is much better for this kind of things,
>
> cheers,
>
> David
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110829/189fa7a4/attachment-0001.html>

From robertwb at math.washington.edu  Tue Aug 30 07:25:11 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Aug 2011 22:25:11 -0700
Subject: [Cython] History of SWIG and applicability to Cython
In-Reply-To: <4E5BA32B.3030709@behnel.de>
References: <4E5BA32B.3030709@behnel.de>
Message-ID: <CADiQ+QD7_iDiMqGZ92RzOC0YzHhWo_6--=pCGSZAv8nPQtDaew@mail.gmail.com>

On Mon, Aug 29, 2011 at 7:33 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> here's an interesting history wrap-up of SWIG, by its original author.

Very interesting thread.

> One thing that stroke me when I read this (or rather when I was half-way
> through) was that it might be possible to use SWIG as a glue code generator
> for .pxd files and trivial wrapper code. Not sure how hard that would be -
> it does sound like such a complex system could also be somewhat tricky to
> extend ...

Yeah, I've thought about this before too. It's probably a project
worth pursuing (especially given all the swig wrapper and wrapped code
out there) but haven't ever looked seriously into it. Clang seems like
a more solid option long-term, and what I would pick if I were to
write a wrapper-generator from scratch (especially for C++). Of course
there's still all the messiness of pointer memory management, type
translation etc.

The idea of automatically parsing header files has been around for a
*long* time, and lots of people have come up with partial solutions
(e.g. "good enough" for their tasks) but solving the generic problem
can be quite hard. So they don't get lost, I've created a page at
http://wiki.cython.org/AutoPxd . Eventually it'd be nice if one such
solution became good enough to be the "standard" solution that
everyone put their weight behind. As for inclusion in Cython, I think
it's still to early to tell, but the a standalone tool to do the .h ->
.pxd/.pyx translation would be quite useful even if it remains a
separate project. (And the two could know about/interface with each
other, even if they weren't bundled together.)

- Robert


> -------- Original-Message --------
> Subject: SWIG (was Re: Ctypes and the stdlib)
> Date: Mon, 29 Aug 2011 07:41:23 -0500
> From: David Beazley <dave... at dabeaz.com>
> To: python-dev... at python.org
> CC: David Beazley <dave... at dabeaz.com>
>
> On Mon, Aug 29, 2011 at 12:27 PM, Guido van Rossum <guido... at python.org>
> wrote:
>
>> I wonder if for
>> this particular purpose SWIG isn't the better match. (If SWIG weren't
>> universally hated, even by its original author. :-)
>
> Hate is probably a strong word, but as the author of Swig, let me chime in
> here ;-). ? I think there are probably some lessons to be learned from Swig.
>
> As Nick noted, Swig is best suited when you have control over both sides
> (C/C++ and Python) of whatever code you're working with. ?In fact, the
> original motivation for ?Swig was to give application programmers
> (scientists in my case), a means for automatically generating the Python
> bindings to their code. ?However, there was one other important
> assumption--and that was the fact that all of your "real code" was going to
> be written in C/C++ and that the Python scripting interface was just an
> optional add-on (perhaps even just a throw-away thing). ?Keep in mind, Swig
> was first created in 1995 and at that time, the use of Python (or any
> similar language) was a pretty radical idea in the sciences. ?Moreover,
> there was a lot of legacy code that people just weren't going to abandon.
> Thus, I alwa
> ?ys viewed Swig as a kind of transitional vehicle for getting people to use
> Python who might otherwise not even consider it. ? Getting back to Nick's
> point though, to really use Swig effectiv
> ?ely, it was always known that you might have to reorganize or refactor your
> C/C++ code to make it more Python friendly. ?However, due to the automatic
> wrapper generation, you didn't have to do it all at once. Basically your
> code could organically evolve and Swig would just keep up with whatever you
> were doing. ?In my projects, we'd usually just tuck Swig away in some
> Makefile somewhere and forget about it.
>
> One of the major complexities of Swig is the fact that it attempts to parse
> C/C++ header files. ? This very notion is actually a dangerous trap waiting
> for anyone who wants to wander into it. ?You might look at a header file and
> say, well how hard could it be to just grab a few definitions out of there?
> ? I'll just write a few regexs or come up with some simple hack for
> recognizing function definitions or something. ? Yes, you can do that, but
> you're immediately going to find that whatever approach you take starts to
> break down into horrible corner cases. ? Swig started out like this and
> quickly turned into a quagmire of esoteric bug reports. ?All sorts of
> problems with preprocessor macros, typedefs, missing headers, and other
> things. ?For awhile, I would get these bug reports that would g
> ?o something like "I had this C++ class inside a namespace with an abstract
> method taking a typedef'd const reference to this smart pointer ..... and
> Swig broke." ? Hell, I can't even underst
> ?and the bug report let alone know how to fix it. ?Almost all of these bugs
> were due to the fact that Swig started out as a hack and didn't really have
> any kind of solid conceptual foundation for how it should be put together.
>
> If you flash forward a bit, from about 2001-2004 there was a very serious
> push to fix these kinds of issues. ?Although it was not a complete rewrite
> of Swig, there were a huge number of changes to how it worked during this
> time. ?Swig grew a fully compatible C++ preprocessor that fully supported
> macros ? A complete C++ type system was implemented including support for
> namespaces, templates, and even such things as template partial
> specialization. ?Swig evolved into a multi-pass compiler that was doing all
> sorts of global analysis of the interface. ? Just to give you an idea, Swig
> would do things such as automatically detect/wrap C++ smart pointers. ?It
> could wrap overloaded C++ methods/function. ?Also, if you had a C++ class
> with virtual methods, it would only make one Python wrapper funct
> ?ion and then reuse across all wrapped subclasses.
>
> Under the covers of all of this, the implementation basically evolved into a
> sophisticated macro preprocessor coupled with a pattern matching engine
> built on top of the C++ type system. ? For example, you could write patterns
> that matched specific C++ types (the much hated "typemap" feature) and you
> could write patterns that matched entire C++ declarations. ?This whole
> pattern matching approach had a huge power if you knew what you were doing.
> ?For example, I had a graduate student working on adding "contracts" to
> Swig--something that was being funded by a NSF grant. ? It was cool and mind
> boggling all at once.
>
> In hindsight however, I think the complexity of Swig has exceeded anyone's
> ability to fully understand it (including my own). ?For example, to even
> make sense of what's happening, you have to have a pretty solid grasp of the
> C/C++ type system (easier said than done). ? Couple that with all sorts of
> crazy pattern matching, low-level code fragments, and a ton of macro
> definitions, your head will literally explode if you try to figure out
> what's happening. ? So far as I know, recent versions of Swig have even
> combined all of this type-pattern matching with regular expressions. ?I
> can't even fathom it.
>
> Sadly, my involvement was Swig was an unfortunate casualty of my academic
> career biting the dust. ?By 2005, I was so burned out of working on it and
> so sick of what I was doing, I quite literally put all of my computer stuff
> aside to go play in a band for a few years. ? After a few years, I came back
> to programming (obviously), but not to keep working on the same stuff. ? In
> particularly, I will die quite happy if I never have to look at another line
> of C++ code ever again. ?No, I would much rather fling my toddlers, ride my
> bike, play piano, or do just about anything than ever do that again.
> Although I still subscribe the Swig mailing lists and watch what's
> happening, I'm not active with it at the moment.
>
> I've sometimes thought it might be interesting to create a Swig replacement
> purely in Python. ?When I work on the PLY project, this is often what I
> think about. ? In that project, I've actually built a number of the parsing
> tools that would be useful in creating such a thing. ? The only catch is
> that when I start thinking along these lines, I usually reach a point where
> I say "nah, I'll just write the whole application in Python."
>
> Anyways, this is probably way more than anyone wants to know about Swig.
> Getting back to the original topic of using it to make standard library
> modules, I just don't know. ? I think you probably could have some success
> with an automatic code generator of some kind. ?I'm just not sure it should
> take the Swig approach of parsing C++ headers. ?I think you could do better.
>
> Cheers,
> Dave
>
> P.S. By the way, if people want to know a lot more about Swig internals,
> they should check out the PyCon 2008 presentation I gave about it.
> http://www.dabeaz.com/SwigMaster/
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Tue Aug 30 07:32:52 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Aug 2011 22:32:52 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
Message-ID: <CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>

On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>
>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>
>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>
>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>
>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>
>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>
>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>
>>>>>>>>>>>> @cython.final
>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>
>>>>>>>>>>>> def test():
>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>
>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>
>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>
>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>
>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>
>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>
>>>>>>>>> "final" is the right option here.
>>>>>>>>>
>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>> does
>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>> imply
>>>>>>>>> "final". But not the other way round.
>>>>>>>>
>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>
>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>
>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>
>>>>>>>
>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>
>>>>>>> That's what "inline" requests.
>>>>>>>
>>>>>>>
>>>>>>>> I see three options:
>>>>>>>>
>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>
>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>> method is overridable?
>>>>>>>
>>>>>>>
>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>
>>>>>>> Correct.
>>>>>>>
>>>>>>
>>>>>> Ok.
>>>>>>
>>>>>> I think it's better to implement final method then user could choose
>>>>>> to use inline qualifier or not.
>>>>>>
>>>>>
>>>>> I tried it here:
>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>
>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>> is defined in the same module.
>>>>> I don't know how to check that final method comes from cimport (it's
>>>>> okay with pxd, the problem is pyx)
>>>>>
>>>>
>>>>
>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>
>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>
>>>> --
>>>> vitja.
>>>>
>>>
>>> I've created ticket for compiler crash when cython.final is used
>>> inside pxd file:
>>>
>>> http://trac.cython.org/cython_trac/ticket/722
>>>
>>> Also I've updated final methods test case (added tree path assertions)
>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>
>>>
>>> About declaration origin detection may be it's a good idea to have a
>>> flag at scope level something like is_pxd_scope or
>>> is_declaration_scope?
>>>
>>
>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>> created pull request:
>>
>> https://github.com/cython/cython/pull/59
>>
>
> I've add support for inline methods, now you can declare inline method
> in pxd file:
>
> cdef class Foo:
> ? cdef inline foo(self):
> ? ? ? return 1

Cool.

I suppose for cross-module calls, non-inline methods vtables are best
one can do short of actually linking the modules together. One
question about your code--I'm not seeing how you're disallowing
overriding final cpdef methods from Python. (Should we even allow
final cpdef methods on non-final classes?)

- Robert

From vitja.makarov at gmail.com  Tue Aug 30 07:57:58 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Aug 2011 09:57:58 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
Message-ID: <CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>

2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>>
>>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>>
>>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>>
>>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>>
>>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>>
>>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>>
>>>>>>>>>>>>> @cython.final
>>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>>
>>>>>>>>>>>>> def test():
>>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>>
>>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>>
>>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>>
>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>>
>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>>
>>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>>
>>>>>>>>>> "final" is the right option here.
>>>>>>>>>>
>>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>>> does
>>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>>> imply
>>>>>>>>>> "final". But not the other way round.
>>>>>>>>>
>>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>>
>>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>>
>>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>>
>>>>>>>> That's what "inline" requests.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I see three options:
>>>>>>>>>
>>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>>
>>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>>> method is overridable?
>>>>>>>>
>>>>>>>>
>>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>>
>>>>>>>> Correct.
>>>>>>>>
>>>>>>>
>>>>>>> Ok.
>>>>>>>
>>>>>>> I think it's better to implement final method then user could choose
>>>>>>> to use inline qualifier or not.
>>>>>>>
>>>>>>
>>>>>> I tried it here:
>>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>>
>>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>>> is defined in the same module.
>>>>>> I don't know how to check that final method comes from cimport (it's
>>>>>> okay with pxd, the problem is pyx)
>>>>>>
>>>>>
>>>>>
>>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>>
>>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>>
>>>>> --
>>>>> vitja.
>>>>>
>>>>
>>>> I've created ticket for compiler crash when cython.final is used
>>>> inside pxd file:
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/722
>>>>
>>>> Also I've updated final methods test case (added tree path assertions)
>>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>>
>>>>
>>>> About declaration origin detection may be it's a good idea to have a
>>>> flag at scope level something like is_pxd_scope or
>>>> is_declaration_scope?
>>>>
>>>
>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>> created pull request:
>>>
>>> https://github.com/cython/cython/pull/59
>>>
>>
>> I've add support for inline methods, now you can declare inline method
>> in pxd file:
>>
>> cdef class Foo:
>> ? cdef inline foo(self):
>> ? ? ? return 1
>
> Cool.
>
> I suppose for cross-module calls, non-inline methods vtables are best
> one can do short of actually linking the modules together.

Sure. This is how it works now. But noitice that final isn't supported
now at pxd scope. So it might not work.


https://github.com/cython/cython/pull/59/files#L5R1888

For subclassing I check that both 3parent and children are in the same scope.
When pxd is cimported vtable bypassing should be disabled in case it's
cimported from anpther module.
Not sure how to implement this. I think we should first fix final at pxd.

> One
> question about your code--I'm not seeing how you're disallowing
> overriding final cpdef methods from Python. (Should we even allow
> final cpdef methods on non-final classes?)
>

I think it's hard and tricky to disallow cpdef overriding but I think
it's better to have cpdef final methods.
And I wouldn't argue if you say that final cpdef methods shouldn't be
allowed at all.

-- 
vitja.

From stefan_ml at behnel.de  Tue Aug 30 10:07:37 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 30 Aug 2011 10:07:37 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E554BA9.3020309@behnel.de>	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>	<4E555084.2000602@behnel.de>	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>	<4E568180.5030104@behnel.de>	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>	<4E569B3B.8060201@behnel.de>	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
Message-ID: <4E5C9A49.4030904@behnel.de>

Robert Bradshaw, 30.08.2011 07:32:
> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov wrote:
>>> I've created pull request:
>>>
>>> https://github.com/cython/cython/pull/59
>>
>> I've add support for inline methods, now you can declare inline method
>> in pxd file:
>>
>> cdef class Foo:
>>    cdef inline foo(self):
>>        return 1
>
> Cool.

+1 :)


> I'm not seeing how you're disallowing
> overriding final cpdef methods from Python.

We could check for an override in the DefNode wrapper code at runtime. 
However, if it has been overridden, we won't even get there when called 
from Python, so we will never know if it has been overridden (well, we 
could add a check to tp_new(), but that would only catch overrides at class 
instantiation time, not attribute assignments).

It's certainly easier and safer to reject this case at compile time, 
providing a very specific error message.

Stefan

From ndbecker2 at gmail.com  Tue Aug 30 13:28:50 2011
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 30 Aug 2011 07:28:50 -0400
Subject: [Cython] History of SWIG and applicability to Cython
References: <4E5BA32B.3030709@behnel.de> <4E5BCB9F.50700@behnel.de>
	<CANg26EVjm1N6YAC2taqXaPLWzB68UaVUEZkkPUrO9S92QiecOQ@mail.gmail.com>
	<CAGY4rcUJy4K+UNY1CAK==qwos97wTXT8uKU6jQ5o-UZhrAFkFQ@mail.gmail.com>
	<CAPc_zRXX88w24g7ja-m42Buw+-Ker15aNTRrxteA3mn=wOtnKQ@mail.gmail.com>
Message-ID: <j3ihhi$lqi$1@dough.gmane.org>

gcc now has python bindings.  Maybe that would allow the desired introspection?  
That would be great, because it seems like python binding for gcc is going to be 
supported.


From robertwb at math.washington.edu  Tue Aug 30 18:11:13 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Aug 2011 09:11:13 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
Message-ID: <CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>

On Mon, Aug 29, 2011 at 10:57 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>>>
>>>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>>>
>>>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>>>
>>>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @cython.final
>>>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> def test():
>>>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>>>
>>>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>>>
>>>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>>>
>>>>>>>>>>> "final" is the right option here.
>>>>>>>>>>>
>>>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>>>> does
>>>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>>>> imply
>>>>>>>>>>> "final". But not the other way round.
>>>>>>>>>>
>>>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>>>
>>>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>>>
>>>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>>>
>>>>>>>>> That's what "inline" requests.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I see three options:
>>>>>>>>>>
>>>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>>>
>>>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>>>> method is overridable?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>>>
>>>>>>>>> Correct.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Ok.
>>>>>>>>
>>>>>>>> I think it's better to implement final method then user could choose
>>>>>>>> to use inline qualifier or not.
>>>>>>>>
>>>>>>>
>>>>>>> I tried it here:
>>>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>>>
>>>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>>>> is defined in the same module.
>>>>>>> I don't know how to check that final method comes from cimport (it's
>>>>>>> okay with pxd, the problem is pyx)
>>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>>>
>>>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>>>
>>>>>> --
>>>>>> vitja.
>>>>>>
>>>>>
>>>>> I've created ticket for compiler crash when cython.final is used
>>>>> inside pxd file:
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/722
>>>>>
>>>>> Also I've updated final methods test case (added tree path assertions)
>>>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>>>
>>>>>
>>>>> About declaration origin detection may be it's a good idea to have a
>>>>> flag at scope level something like is_pxd_scope or
>>>>> is_declaration_scope?
>>>>>
>>>>
>>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>>> created pull request:
>>>>
>>>> https://github.com/cython/cython/pull/59
>>>>
>>>
>>> I've add support for inline methods, now you can declare inline method
>>> in pxd file:
>>>
>>> cdef class Foo:
>>> ? cdef inline foo(self):
>>> ? ? ? return 1
>>
>> Cool.
>>
>> I suppose for cross-module calls, non-inline methods vtables are best
>> one can do short of actually linking the modules together.
>
> Sure. This is how it works now. But noitice that final isn't supported
> now at pxd scope. So it might not work.
>
>
> https://github.com/cython/cython/pull/59/files#L5R1888
>
> For subclassing I check that both 3parent and children are in the same scope.
> When pxd is cimported vtable bypassing should be disabled in case it's
> cimported from anpther module.
> Not sure how to implement this. I think we should first fix final at pxd.
>
>> One
>> question about your code--I'm not seeing how you're disallowing
>> overriding final cpdef methods from Python. (Should we even allow
>> final cpdef methods on non-final classes?)
>>
>
> I think it's hard and tricky to disallow cpdef overriding but I think
> it's better to have cpdef final methods.
> And I wouldn't argue if you say that final cpdef methods shouldn't be
> allowed at all.

Yes, disallowing it with an explicit compile-time error was what I was
thinking.

- Robert

From vitja.makarov at gmail.com  Tue Aug 30 18:14:23 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Aug 2011 20:14:23 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
Message-ID: <CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>

2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
> On Mon, Aug 29, 2011 at 10:57 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>>>>
>>>>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>>>>
>>>>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @cython.final
>>>>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> def test():
>>>>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>>>>
>>>>>>>>>>>> "final" is the right option here.
>>>>>>>>>>>>
>>>>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>>>>> does
>>>>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>>>>> imply
>>>>>>>>>>>> "final". But not the other way round.
>>>>>>>>>>>
>>>>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>>>>
>>>>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>>>>
>>>>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>>>>
>>>>>>>>>> That's what "inline" requests.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I see three options:
>>>>>>>>>>>
>>>>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>>>>
>>>>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>>>>> method is overridable?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>>>>
>>>>>>>>>> Correct.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok.
>>>>>>>>>
>>>>>>>>> I think it's better to implement final method then user could choose
>>>>>>>>> to use inline qualifier or not.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I tried it here:
>>>>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>>>>
>>>>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>>>>> is defined in the same module.
>>>>>>>> I don't know how to check that final method comes from cimport (it's
>>>>>>>> okay with pxd, the problem is pyx)
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>>>>
>>>>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>>>>
>>>>>>> --
>>>>>>> vitja.
>>>>>>>
>>>>>>
>>>>>> I've created ticket for compiler crash when cython.final is used
>>>>>> inside pxd file:
>>>>>>
>>>>>> http://trac.cython.org/cython_trac/ticket/722
>>>>>>
>>>>>> Also I've updated final methods test case (added tree path assertions)
>>>>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>>>>
>>>>>>
>>>>>> About declaration origin detection may be it's a good idea to have a
>>>>>> flag at scope level something like is_pxd_scope or
>>>>>> is_declaration_scope?
>>>>>>
>>>>>
>>>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>>>> created pull request:
>>>>>
>>>>> https://github.com/cython/cython/pull/59
>>>>>
>>>>
>>>> I've add support for inline methods, now you can declare inline method
>>>> in pxd file:
>>>>
>>>> cdef class Foo:
>>>> ? cdef inline foo(self):
>>>> ? ? ? return 1
>>>
>>> Cool.
>>>
>>> I suppose for cross-module calls, non-inline methods vtables are best
>>> one can do short of actually linking the modules together.
>>
>> Sure. This is how it works now. But noitice that final isn't supported
>> now at pxd scope. So it might not work.
>>
>>
>> https://github.com/cython/cython/pull/59/files#L5R1888
>>
>> For subclassing I check that both 3parent and children are in the same scope.
>> When pxd is cimported vtable bypassing should be disabled in case it's
>> cimported from anpther module.
>> Not sure how to implement this. I think we should first fix final at pxd.
>>
>>> One
>>> question about your code--I'm not seeing how you're disallowing
>>> overriding final cpdef methods from Python. (Should we even allow
>>> final cpdef methods on non-final classes?)
>>>
>>
>> I think it's hard and tricky to disallow cpdef overriding but I think
>> it's better to have cpdef final methods.
>> And I wouldn't argue if you say that final cpdef methods shouldn't be
>> allowed at all.
>
> Yes, disallowing it with an explicit compile-time error was what I was
> thinking.
>

What about final classes with cpdef methods?

@cython.final
class Foo:
    cpdef bar(self):
        pass

Should that raise an error?

-- 
vitja.

From robertwb at math.washington.edu  Tue Aug 30 18:18:47 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Aug 2011 09:18:47 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
Message-ID: <CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>

On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Mon, Aug 29, 2011 at 10:57 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>>>> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>>>>>
>>>>>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @cython.final
>>>>>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> def test():
>>>>>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>>>>>
>>>>>>>>>>>>> "final" is the right option here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>>>>>> does
>>>>>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>>>>>> imply
>>>>>>>>>>>>> "final". But not the other way round.
>>>>>>>>>>>>
>>>>>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>>>>>
>>>>>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>>>>>
>>>>>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>>>>>
>>>>>>>>>>> That's what "inline" requests.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I see three options:
>>>>>>>>>>>>
>>>>>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>>>>>
>>>>>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>>>>>> method is overridable?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>>>>>
>>>>>>>>>>> Correct.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok.
>>>>>>>>>>
>>>>>>>>>> I think it's better to implement final method then user could choose
>>>>>>>>>> to use inline qualifier or not.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I tried it here:
>>>>>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>>>>>
>>>>>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>>>>>> is defined in the same module.
>>>>>>>>> I don't know how to check that final method comes from cimport (it's
>>>>>>>>> okay with pxd, the problem is pyx)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>>>>>
>>>>>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>>>>>
>>>>>>>> --
>>>>>>>> vitja.
>>>>>>>>
>>>>>>>
>>>>>>> I've created ticket for compiler crash when cython.final is used
>>>>>>> inside pxd file:
>>>>>>>
>>>>>>> http://trac.cython.org/cython_trac/ticket/722
>>>>>>>
>>>>>>> Also I've updated final methods test case (added tree path assertions)
>>>>>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>>>>>
>>>>>>>
>>>>>>> About declaration origin detection may be it's a good idea to have a
>>>>>>> flag at scope level something like is_pxd_scope or
>>>>>>> is_declaration_scope?
>>>>>>>
>>>>>>
>>>>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>>>>> created pull request:
>>>>>>
>>>>>> https://github.com/cython/cython/pull/59
>>>>>>
>>>>>
>>>>> I've add support for inline methods, now you can declare inline method
>>>>> in pxd file:
>>>>>
>>>>> cdef class Foo:
>>>>> ? cdef inline foo(self):
>>>>> ? ? ? return 1
>>>>
>>>> Cool.
>>>>
>>>> I suppose for cross-module calls, non-inline methods vtables are best
>>>> one can do short of actually linking the modules together.
>>>
>>> Sure. This is how it works now. But noitice that final isn't supported
>>> now at pxd scope. So it might not work.
>>>
>>>
>>> https://github.com/cython/cython/pull/59/files#L5R1888
>>>
>>> For subclassing I check that both 3parent and children are in the same scope.
>>> When pxd is cimported vtable bypassing should be disabled in case it's
>>> cimported from anpther module.
>>> Not sure how to implement this. I think we should first fix final at pxd.
>>>
>>>> One
>>>> question about your code--I'm not seeing how you're disallowing
>>>> overriding final cpdef methods from Python. (Should we even allow
>>>> final cpdef methods on non-final classes?)
>>>>
>>>
>>> I think it's hard and tricky to disallow cpdef overriding but I think
>>> it's better to have cpdef final methods.
>>> And I wouldn't argue if you say that final cpdef methods shouldn't be
>>> allowed at all.
>>
>> Yes, disallowing it with an explicit compile-time error was what I was
>> thinking.
>>
>
> What about final classes with cpdef methods?
>
> @cython.final
> class Foo:
> ? ?cpdef bar(self):
> ? ? ? ?pass
>
> Should that raise an error?

That should be perfectly fine.

- Robert

From stefan_ml at behnel.de  Tue Aug 30 18:33:52 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 30 Aug 2011 18:33:52 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E568180.5030104@behnel.de>	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>	<4E569B3B.8060201@behnel.de>	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
Message-ID: <4E5D10F0.2010803@behnel.de>

Robert Bradshaw, 30.08.2011 18:18:
> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>> What about final classes with cpdef methods?
>>
>> @cython.final
>> class Foo:
>>     cpdef bar(self):
>>         pass
>>
>> Should that raise an error?
>
> That should be perfectly fine.

Well, the 'final' decorator shouldn't work on normal Python classes.

Regarding extension types, CPython has a way of declaring them 'final' with 
a type flag, which effectively prevents them from being subclassed in 
Python. So the above works as just fine for cdef classes.

Stefan

From vitja.makarov at gmail.com  Tue Aug 30 18:36:52 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Aug 2011 20:36:52 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E554BA9.3020309@behnel.de>
	<CAKGHGPSJxFtqD-KBQFq8TqRs_zbVA0A9Maxsoa8OzcA8ALfrCA@mail.gmail.com>
	<4E555084.2000602@behnel.de>
	<CAKGHGPSr8wj7q2x_CXcuhaSLbaaD0FkCCRjtV+QoZqALaVS3Dg@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
Message-ID: <CAKGHGPSvGaxwYVv0u+JzpLwxWgvQW-V8LmDYwrr=KXXct_wzfA@mail.gmail.com>

2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Mon, Aug 29, 2011 at 10:57 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2011/8/30 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>> On Sun, Aug 28, 2011 at 4:00 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2011/8/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>> 2011/8/26 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>> 2011/8/25 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>> 2011/8/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>>>>>>>>>> Vitja Makarov, 25.08.2011 20:32:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2011/8/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Vitja Makarov, 25.08.2011 18:11:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2011/8/24 Stefan Behnel:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vitja Makarov, 24.08.2011 21:17:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I tried final classes:
>>>>>>>>>>>>>>>>> 2. In this example foo call is done through virtual table
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cimport cython
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @cython.final
>>>>>>>>>>>>>>>>> cdef class Foo:
>>>>>>>>>>>>>>>>> ? ? cdef foo(self):
>>>>>>>>>>>>>>>>> ? ? ? ? print 'haha'
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> def test():
>>>>>>>>>>>>>>>>> ? ? cdef Foo a = Foo()
>>>>>>>>>>>>>>>>> ? ? a.foo()
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ? __pyx_t_1 = ((struct __pyx_vtabstruct_3yyy_Foo
>>>>>>>>>>>>>>>>> *)__pyx_v_a->__pyx_vtab)->foo(__pyx_v_a); if (unlikely(!__pyx_t_1))
>>>>>>>>>>>>>>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 12; __pyx_clineno =
>>>>>>>>>>>>>>>>> __LINE__; goto __pyx_L1_error;}
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Right, this is not implemented yet. Feel free to do so. Also see
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/474
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3. I can't use final decorator for methods (error reported)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://trac.cython.org/cython_trac/ticket/586
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What is the prefered syntax keyword inline or final decorator?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "final" is the right option here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> They are orthogonal concepts. Only because you declare a method "final"
>>>>>>>>>>>>>> does
>>>>>>>>>>>>>> not mean you want to inline it, and just because you declare it "inline"
>>>>>>>>>>>>>> does not (necessarily) mean that you cannot override it. Admittedly, the
>>>>>>>>>>>>>> semantics of an overridable inline method may turn out to be somewhat
>>>>>>>>>>>>>> obscure and error prone, so I think it's a good idea to let "inline"
>>>>>>>>>>>>>> imply
>>>>>>>>>>>>>> "final". But not the other way round.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But both inline and final methods should bypass vtab, right?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes. But in the "final" case, it's always clear which method implementation
>>>>>>>>>>>> to use - it's not overridable, so there is only one choice. In the "inline"
>>>>>>>>>>>> case, it could still be overridable and we may have a subtype of the
>>>>>>>>>>>> declared type in our hands at runtime, thus choosing the wrong method at
>>>>>>>>>>>> compile time. That's why only the "final" case is safe.
>>>>>>>>>>>>
>>>>>>>>>>>> Note that I'm only talking about the semantics of the qualifier themselves
>>>>>>>>>>>> here. If we allow "inline" methods, I think we should force them to be
>>>>>>>>>>>> "final" as well. But that's a practical choice, not a semantic implication.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Also I'm not sure about C inline qualifier here.
>>>>>>>>>>>>
>>>>>>>>>>>> That's what "inline" requests.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I see three options:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ?- non-virtual: bypass vtab
>>>>>>>>>>>>> ?- final: non-virtual, non-overridable
>>>>>>>>>>>>
>>>>>>>>>>>> How would you want to bypass the vtable in the "non-virtual" case if the
>>>>>>>>>>>> method is overridable?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> ?- inline: non-virtual, C inline qualifier is used
>>>>>>>>>>>>
>>>>>>>>>>>> Correct.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok.
>>>>>>>>>>>
>>>>>>>>>>> I think it's better to implement final method then user could choose
>>>>>>>>>>> to use inline qualifier or not.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I tried it here:
>>>>>>>>>> https://github.com/vitek/cython/commit/ddf80a80dc75aced2cd92dc32afa77a7bcf2de02
>>>>>>>>>>
>>>>>>>>>> There is one problem: vtab bypassing should be enabled if final method
>>>>>>>>>> is defined in the same module.
>>>>>>>>>> I don't know how to check that final method comes from cimport (it's
>>>>>>>>>> okay with pxd, the problem is pyx)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/vitek/cython/commit/6e91fc257a683018ba6be340d384f9a7c34ef425
>>>>>>>>>
>>>>>>>>> Here is update version. I've add tree asserts and final method's prototypes.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> vitja.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I've created ticket for compiler crash when cython.final is used
>>>>>>>> inside pxd file:
>>>>>>>>
>>>>>>>> http://trac.cython.org/cython_trac/ticket/722
>>>>>>>>
>>>>>>>> Also I've updated final methods test case (added tree path assertions)
>>>>>>>> https://github.com/vitek/cython/commit/92edb09419c9b77a792f7c43e6ddd760b00c4e74
>>>>>>>>
>>>>>>>>
>>>>>>>> About declaration origin detection may be it's a good idea to have a
>>>>>>>> flag at scope level something like is_pxd_scope or
>>>>>>>> is_declaration_scope?
>>>>>>>>
>>>>>>>
>>>>>>> It seems to me that I found a way to fix pxd/pyx cimport problem. I've
>>>>>>> created pull request:
>>>>>>>
>>>>>>> https://github.com/cython/cython/pull/59
>>>>>>>
>>>>>>
>>>>>> I've add support for inline methods, now you can declare inline method
>>>>>> in pxd file:
>>>>>>
>>>>>> cdef class Foo:
>>>>>> ? cdef inline foo(self):
>>>>>> ? ? ? return 1
>>>>>
>>>>> Cool.
>>>>>
>>>>> I suppose for cross-module calls, non-inline methods vtables are best
>>>>> one can do short of actually linking the modules together.
>>>>
>>>> Sure. This is how it works now. But noitice that final isn't supported
>>>> now at pxd scope. So it might not work.
>>>>
>>>>
>>>> https://github.com/cython/cython/pull/59/files#L5R1888
>>>>
>>>> For subclassing I check that both 3parent and children are in the same scope.
>>>> When pxd is cimported vtable bypassing should be disabled in case it's
>>>> cimported from anpther module.
>>>> Not sure how to implement this. I think we should first fix final at pxd.
>>>>
>>>>> One
>>>>> question about your code--I'm not seeing how you're disallowing
>>>>> overriding final cpdef methods from Python. (Should we even allow
>>>>> final cpdef methods on non-final classes?)
>>>>>
>>>>
>>>> I think it's hard and tricky to disallow cpdef overriding but I think
>>>> it's better to have cpdef final methods.
>>>> And I wouldn't argue if you say that final cpdef methods shouldn't be
>>>> allowed at all.
>>>
>>> Yes, disallowing it with an explicit compile-time error was what I was
>>> thinking.
>>>
>>
>> What about final classes with cpdef methods?
>>
>> @cython.final
>> class Foo:
>> ? ?cpdef bar(self):
>> ? ? ? ?pass
>>
>> Should that raise an error?
>
> That should be perfectly fine.
>

I'm not sure here. This would make users write code like this:


@final
cdef class Foo:
    cdef _is_foo(self):
        return self._foo

    def is_foo(self):
        return self._is_foo()

cpdef is more convinient here. Also note that final class couldn't
subclass types with cpdef methods.

-- 
vitja.

From vitja.makarov at gmail.com  Tue Aug 30 18:39:45 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Aug 2011 20:39:45 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E5D10F0.2010803@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E568180.5030104@behnel.de>
	<CAKGHGPTLthDn2zP==LoHy3EtjwZG_3VXk-g5RnownMWkWL0dow@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
	<4E5D10F0.2010803@behnel.de>
Message-ID: <CAKGHGPTwKd1xLh5S61J06Hepk8hyhde_m4C=Ssy+duWifewmMQ@mail.gmail.com>

2011/8/30 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 30.08.2011 18:18:
>>
>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>>
>>> What about final classes with cpdef methods?
>>>
>>> @cython.final
>>> class Foo:
>>> ? ?cpdef bar(self):
>>> ? ? ? ?pass
>>>
>>> Should that raise an error?
>>
>> That should be perfectly fine.
>
> Well, the 'final' decorator shouldn't work on normal Python classes.
>
> Regarding extension types, CPython has a way of declaring them 'final' with
> a type flag, which effectively prevents them from being subclassed in
> Python. So the above works as just fine for cdef classes.
>

Ok. So final class could have cpdef methods but non-final extension
type couldn't, am I right?



-- 
vitja.

From stefan_ml at behnel.de  Tue Aug 30 18:41:49 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 30 Aug 2011 18:41:49 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPSvGaxwYVv0u+JzpLwxWgvQW-V8LmDYwrr=KXXct_wzfA@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E569B3B.8060201@behnel.de>	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
	<CAKGHGPSvGaxwYVv0u+JzpLwxWgvQW-V8LmDYwrr=KXXct_wzfA@mail.gmail.com>
Message-ID: <4E5D12CD.5090603@behnel.de>

Vitja Makarov, 30.08.2011 18:36:
> 2011/8/30 Robert Bradshaw:
>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>> What about final classes with cpdef methods?
>>>
>>> @cython.final
>>> class Foo:
>>>     cpdef bar(self):
>>>         pass
>>>
>>> Should that raise an error?
>>
>> That should be perfectly fine.

Robert meant to say that the above code is fine and should *not* raise an 
error.


> I'm not sure here. This would make users write code like this:
>
> @final
> cdef class Foo:
>      cdef _is_foo(self):
>          return self._foo
>
>      def is_foo(self):
>          return self._is_foo()
>
> cpdef is more convinient here. Also note that final class couldn't
> subclass types with cpdef methods.

Right.

Stefan

From stefan_ml at behnel.de  Tue Aug 30 18:51:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 30 Aug 2011 18:51:45 +0200
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPTwKd1xLh5S61J06Hepk8hyhde_m4C=Ssy+duWifewmMQ@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>	<4E569B3B.8060201@behnel.de>	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>	<4E5D10F0.2010803@behnel.de>
	<CAKGHGPTwKd1xLh5S61J06Hepk8hyhde_m4C=Ssy+duWifewmMQ@mail.gmail.com>
Message-ID: <4E5D1521.6050704@behnel.de>

Vitja Makarov, 30.08.2011 18:39:
> 2011/8/30 Stefan Behnel:
>> Robert Bradshaw, 30.08.2011 18:18:
>>>
>>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>>>
>>>> What about final classes with cpdef methods?
>>>>
>>>> @cython.final
>>>> class Foo:
>>>>     cpdef bar(self):
>>>>         pass
>>>>
>>>> Should that raise an error?
>>>
>>> That should be perfectly fine.
>>
>> Well, the 'final' decorator shouldn't work on normal Python classes.
>>
>> Regarding extension types, CPython has a way of declaring them 'final' with
>> a type flag, which effectively prevents them from being subclassed in
>> Python. So the above works as just fine for cdef classes.
>>
>
> Ok. So final class could have cpdef methods but non-final extension
> type couldn't, am I right?

All extension types can have cpdef methods, be they final or not. For final 
classes, cpdef methods simply mean that they have a Python wrapper and will 
otherwise be called directly when called from Cython.

Actually, for cpdef methods in final classes, we can even drop the override 
check in the DefNode wrapper. I don't think that currently happens.

Stefan

From vitja.makarov at gmail.com  Tue Aug 30 19:19:03 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Aug 2011 21:19:03 +0400
Subject: [Cython] non-virtual methods
In-Reply-To: <4E5D1521.6050704@behnel.de>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
	<4E5D10F0.2010803@behnel.de>
	<CAKGHGPTwKd1xLh5S61J06Hepk8hyhde_m4C=Ssy+duWifewmMQ@mail.gmail.com>
	<4E5D1521.6050704@behnel.de>
Message-ID: <CAKGHGPQcD=0g3PghxgwHO325dZO9nCF4aDC53p9j+7-NcqCN4g@mail.gmail.com>

2011/8/30 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 30.08.2011 18:39:
>>
>> 2011/8/30 Stefan Behnel:
>>>
>>> Robert Bradshaw, 30.08.2011 18:18:
>>>>
>>>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>>>>
>>>>> What about final classes with cpdef methods?
>>>>>
>>>>> @cython.final
>>>>> class Foo:
>>>>> ? ?cpdef bar(self):
>>>>> ? ? ? ?pass
>>>>>
>>>>> Should that raise an error?
>>>>
>>>> That should be perfectly fine.
>>>
>>> Well, the 'final' decorator shouldn't work on normal Python classes.
>>>
>>> Regarding extension types, CPython has a way of declaring them 'final'
>>> with
>>> a type flag, which effectively prevents them from being subclassed in
>>> Python. So the above works as just fine for cdef classes.
>>>
>>
>> Ok. So final class could have cpdef methods but non-final extension
>> type couldn't, am I right?
>
> All extension types can have cpdef methods, be they final or not. For final
> classes, cpdef methods simply mean that they have a Python wrapper and will
> otherwise be called directly when called from Cython.
>
> Actually, for cpdef methods in final classes, we can even drop the override
> check in the DefNode wrapper. I don't think that currently happens.
>

Currently I drop OverrideCheckNode for all final cpdef methods, see
https://github.com/cython/cython/pull/59/files#L2R1751

Final classes can have cpdef method that's ok.
Could non-final classes have final-cpdef methods?

Currently final class means that all its c[p]def methods are final too.

-- 
vitja.

From robertwb at math.washington.edu  Tue Aug 30 19:40:46 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Aug 2011 10:40:46 -0700
Subject: [Cython] non-virtual methods
In-Reply-To: <CAKGHGPQcD=0g3PghxgwHO325dZO9nCF4aDC53p9j+7-NcqCN4g@mail.gmail.com>
References: <CAKGHGPSwMckJuxvORiSMMTR=vZcKvcphiBQR2aeLvWFL9Urn=w@mail.gmail.com>
	<4E569B3B.8060201@behnel.de>
	<CAKGHGPRq2GHAHjuH-gFy+N6sgfjvuP1JA6wGbQ05=03oBe4OYQ@mail.gmail.com>
	<CAKGHGPQvVMC9P=PJSoDdgu1EU8E9PKmPw4wBRCe-5j3MmOHJSQ@mail.gmail.com>
	<CAKGHGPQzmWRj=1O6dRSdYQ6d80HY7jhunr8vuxBuzLdc4qyHJg@mail.gmail.com>
	<CAKGHGPR2vCHoUg4qW0boBQ+hPFvPhF-hDM5A_kENDp8fmSh=cw@mail.gmail.com>
	<CAKGHGPTAsTzB4Y7B_=cgQH1SmXfxzLY8vMhS-tUk_9CUf=52LA@mail.gmail.com>
	<CAKGHGPTkJ3pEdX5hUt6NXv=ZqT=GMn3Jb7s9zOGaPk5MpcOM+g@mail.gmail.com>
	<CADiQ+QAdah5QFTdDdfg5fu4ibwo+g-U_UqTHe6BZAMVN==EZoQ@mail.gmail.com>
	<CAKGHGPQDbmmKmW6aiwEL3YGo+0EdJHb0B7iEeL1nQFrj8tfF1w@mail.gmail.com>
	<CADiQ+QCKW6Qv072LV_heDU6VQv5micq3mjQd9DSwsnnJ+7JFyg@mail.gmail.com>
	<CAKGHGPR+CeAtrqKcmDAg=cycLM4nWUsuR9izOt=fVbDfWDX1vQ@mail.gmail.com>
	<CADiQ+QDZtYE4kopOjYYLywHhE6v8NT-dE+m8K4K+F5=07NesNQ@mail.gmail.com>
	<4E5D10F0.2010803@behnel.de>
	<CAKGHGPTwKd1xLh5S61J06Hepk8hyhde_m4C=Ssy+duWifewmMQ@mail.gmail.com>
	<4E5D1521.6050704@behnel.de>
	<CAKGHGPQcD=0g3PghxgwHO325dZO9nCF4aDC53p9j+7-NcqCN4g@mail.gmail.com>
Message-ID: <CADiQ+QBXjkVpz-ZiotUdKr=FkC2ewtOJeTCO4sgBT5G96bsoiA@mail.gmail.com>

On Tue, Aug 30, 2011 at 10:19 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/8/30 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 30.08.2011 18:39:
>>>
>>> 2011/8/30 Stefan Behnel:
>>>>
>>>> Robert Bradshaw, 30.08.2011 18:18:
>>>>>
>>>>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>>>>>
>>>>>> What about final classes with cpdef methods?
>>>>>>
>>>>>> @cython.final
>>>>>> class Foo:
>>>>>> ? ?cpdef bar(self):
>>>>>> ? ? ? ?pass
>>>>>>
>>>>>> Should that raise an error?
>>>>>
>>>>> That should be perfectly fine.
>>>>
>>>> Well, the 'final' decorator shouldn't work on normal Python classes.
>>>>
>>>> Regarding extension types, CPython has a way of declaring them 'final'
>>>> with
>>>> a type flag, which effectively prevents them from being subclassed in
>>>> Python. So the above works as just fine for cdef classes.
>>>>
>>>
>>> Ok. So final class could have cpdef methods but non-final extension
>>> type couldn't, am I right?
>>
>> All extension types can have cpdef methods, be they final or not. For final
>> classes, cpdef methods simply mean that they have a Python wrapper and will
>> otherwise be called directly when called from Cython.
>>
>> Actually, for cpdef methods in final classes, we can even drop the override
>> check in the DefNode wrapper. I don't think that currently happens.
>>
>
> Currently I drop OverrideCheckNode for all final cpdef methods, see
> https://github.com/cython/cython/pull/59/files#L2R1751
>
> Final classes can have cpdef method that's ok.
> Could non-final classes have final-cpdef methods?

No, as we can't prevent them from being overridden (violating the
semantics of final). We should raise an error when trying to declare
final cpdef methods in non-final classes.

> Currently final class means that all its c[p]def methods are final too.

Good point.

- Robert

From vitja.makarov at gmail.com  Wed Aug 31 15:04:04 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 31 Aug 2011 17:04:04 +0400
Subject: [Cython] Bug in CPython?
Message-ID: <CAKGHGPTmbZ28qWU4VPYTa2tUi_R92ZhwVfeeDouZ3Pz_hum=zQ@mail.gmail.com>

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/795/

It seems that crash was caused by change in cpython repo.

-- 
vitja.

From stefan_ml at behnel.de  Wed Aug 31 16:05:35 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 31 Aug 2011 16:05:35 +0200
Subject: [Cython] Bug in CPython?
In-Reply-To: <CAKGHGPTmbZ28qWU4VPYTa2tUi_R92ZhwVfeeDouZ3Pz_hum=zQ@mail.gmail.com>
References: <CAKGHGPTmbZ28qWU4VPYTa2tUi_R92ZhwVfeeDouZ3Pz_hum=zQ@mail.gmail.com>
Message-ID: <4E5E3FAF.8080406@behnel.de>

Vitja Makarov, 31.08.2011 15:04:
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/795/
>
> It seems that crash was caused by change in cpython repo.

Without having looked into this any deeper, that was my impression as well 
when I saw it.

Stefan

From vitja.makarov at gmail.com  Wed Aug 31 16:11:52 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 31 Aug 2011 18:11:52 +0400
Subject: [Cython] Bug in CPython?
In-Reply-To: <4E5E3FAF.8080406@behnel.de>
References: <CAKGHGPTmbZ28qWU4VPYTa2tUi_R92ZhwVfeeDouZ3Pz_hum=zQ@mail.gmail.com>
	<4E5E3FAF.8080406@behnel.de>
Message-ID: <CAKGHGPRFdNDTW-EjTb_1XWtjMLgZr1hCmzKNokYEzV=gjv1zRA@mail.gmail.com>

2011/8/31 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 31.08.2011 15:04:
>>
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/795/
>>
>> It seems that crash was caused by change in cpython repo.
>
> Without having looked into this any deeper, that was my impression as well
> when I saw it.
>
> Stefan

It seems to be fixed here
http://hg.python.org/cpython/rev/4aa00f465b4f/

How often does jenkins update py27-hg target?

-- 
vitja.

From stefan_ml at behnel.de  Wed Aug 31 16:26:07 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 31 Aug 2011 16:26:07 +0200
Subject: [Cython] Bug in CPython?
In-Reply-To: <CAKGHGPRFdNDTW-EjTb_1XWtjMLgZr1hCmzKNokYEzV=gjv1zRA@mail.gmail.com>
References: <CAKGHGPTmbZ28qWU4VPYTa2tUi_R92ZhwVfeeDouZ3Pz_hum=zQ@mail.gmail.com>	<4E5E3FAF.8080406@behnel.de>
	<CAKGHGPRFdNDTW-EjTb_1XWtjMLgZr1hCmzKNokYEzV=gjv1zRA@mail.gmail.com>
Message-ID: <4E5E447F.4090502@behnel.de>

Vitja Makarov, 31.08.2011 16:11:
> 2011/8/31 Stefan Behnel:
>> Vitja Makarov, 31.08.2011 15:04:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/795/
>>>
>>> It seems that crash was caused by change in cpython repo.
>>
>> Without having looked into this any deeper, that was my impression as well
>> when I saw it.
>
> It seems to be fixed here
> http://hg.python.org/cpython/rev/4aa00f465b4f/
>
> How often does jenkins update py27-hg target?

The CPython builds are updated once a day. You can also see that from the 
build history of those jobs.

I prefer keeping the CPython build rates low to avoid unnecessary builds. 
CPython tends to have a much higher churn rate than Cython, simply due to 
the size of the code base and the number of people working on it.

Since we only build once in 24h, it happens from time to time that the 
build jobs pick up a broken revision for a day or two, but it rarely takes 
longer than that to fix on their side, at least for something as obvious as 
a crash in the test suite.

Stefan