From mansourmoufid at gmail.com  Tue Jan  3 01:31:56 2012
From: mansourmoufid at gmail.com (Mansour Moufid)
Date: Mon, 2 Jan 2012 19:31:56 -0500
Subject: [Cython] Fix integer width constant names in stdint.pxd
Message-ID: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>

Hello,

Attached is a quick fix for some typos in stdint.pxd.

Tested with Cython version 0.15.1.

Mansour
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-integer-width-constant-names-in-stdint.pxd.patch
Type: text/x-patch
Size: 3155 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120102/4f5696c8/attachment.bin>

From robertwb at math.washington.edu  Tue Jan  3 01:56:13 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jan 2012 16:56:13 -0800
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
Message-ID: <CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>

Thanks.

On Mon, Jan 2, 2012 at 4:31 PM, Mansour Moufid <mansourmoufid at gmail.com> wrote:
> Hello,
>
> Attached is a quick fix for some typos in stdint.pxd.
>
> Tested with Cython version 0.15.1.
>
> Mansour
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From mansourmoufid at gmail.com  Tue Jan  3 02:37:34 2012
From: mansourmoufid at gmail.com (Mansour Moufid)
Date: Mon, 2 Jan 2012 20:37:34 -0500
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
Message-ID: <CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>

Now my issue is as follows.

(I CCed the cython-users list if this question is more appropriate there.)

I have a simple file, int.pyx:

from libc.stdint cimport *
print long(UINT8_MAX)
print long(UINT16_MAX)
print long(UINT32_MAX)
print long(UINT64_MAX)

with the usual setup.py stuff. Compiling and running:

$ python setup.py build_ext --inplace
...
int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
...
$ python -c 'import int'
255
65535
-1
-1

So obviously there are overflows here. Checking int.c, I see:

  /* "int.pyx":2
 * from libc.stdint cimport *
 * print long(UINT8_MAX)             # <<<<<<<<<<<<<<
 * print long(UINT16_MAX)
 * print long(UINT32_MAX)
 */
  __pyx_t_1 = PyInt_FromLong(UINT8_MAX);

and so on...

PyInt_FromLong is used for all these constants, regardless of
signedness or width, so any argument larger than LONG_MAX overflows,
*before* being converted to the arbitrary-size Python integer type.

I don't know if this is a bug, or if I'm overlooking something. Is
there a way for me to use these constants with Python's arbitrary-size
integers?

Thanks,
Mansour

From dalcinl at gmail.com  Tue Jan  3 02:48:40 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Jan 2012 22:48:40 -0300
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
	<CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>
Message-ID: <CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>

On 2 January 2012 22:37, Mansour Moufid <mansourmoufid at gmail.com> wrote:
> Now my issue is as follows.
>
> (I CCed the cython-users list if this question is more appropriate there.)
>
> I have a simple file, int.pyx:
>
> from libc.stdint cimport *
> print long(UINT8_MAX)
> print long(UINT16_MAX)
> print long(UINT32_MAX)
> print long(UINT64_MAX)
>
> with the usual setup.py stuff. Compiling and running:
>
> $ python setup.py build_ext --inplace
> ...
> int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
> ...
> $ python -c 'import int'
> 255
> 65535
> -1
> -1
>
> So obviously there are overflows here. Checking int.c, I see:
>
> ?/* "int.pyx":2
> ?* from libc.stdint cimport *
> ?* print long(UINT8_MAX) ? ? ? ? ? ? # <<<<<<<<<<<<<<
> ?* print long(UINT16_MAX)
> ?* print long(UINT32_MAX)
> ?*/
> ?__pyx_t_1 = PyInt_FromLong(UINT8_MAX);
>
> and so on...
>
> PyInt_FromLong is used for all these constants, regardless of
> signedness or width, so any argument larger than LONG_MAX overflows,
> *before* being converted to the arbitrary-size Python integer type.
>
> I don't know if this is a bug, or if I'm overlooking something. Is
> there a way for me to use these constants with Python's arbitrary-size
> integers?
>

All these constants are declared as "enum", so Cython promotes them to
"int". Once again, Cython should have something like a "const" type
qualifier to poperly declare these compile-time constants.

As workaround, you could explicitly cast the constants like this
"print long(<uint8_t>UINT8_MAX)"


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From mansourmoufid at gmail.com  Tue Jan  3 02:53:54 2012
From: mansourmoufid at gmail.com (Mansour Moufid)
Date: Mon, 2 Jan 2012 20:53:54 -0500
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
	<CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>
	<CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
Message-ID: <CALogXGUfzJNgzUM2acMwe2gxZmvja9xS1TgB2qm=Q+5moFdECA@mail.gmail.com>

On Mon, Jan 2, 2012 at 8:48 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 2 January 2012 22:37, Mansour Moufid <mansourmoufid at gmail.com> wrote:
>> Now my issue is as follows.
>>
>> (I CCed the cython-users list if this question is more appropriate there.)
>>
>> I have a simple file, int.pyx:
>>
>> from libc.stdint cimport *
>> print long(UINT8_MAX)
>> print long(UINT16_MAX)
>> print long(UINT32_MAX)
>> print long(UINT64_MAX)
>>
>> with the usual setup.py stuff. Compiling and running:
>>
>> $ python setup.py build_ext --inplace
>> ...
>> int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
>> ...
>> $ python -c 'import int'
>> 255
>> 65535
>> -1
>> -1
>>
>> So obviously there are overflows here. Checking int.c, I see:
>>
>> ?/* "int.pyx":2
>> ?* from libc.stdint cimport *
>> ?* print long(UINT8_MAX) ? ? ? ? ? ? # <<<<<<<<<<<<<<
>> ?* print long(UINT16_MAX)
>> ?* print long(UINT32_MAX)
>> ?*/
>> ?__pyx_t_1 = PyInt_FromLong(UINT8_MAX);
>>
>> and so on...
>>
>> PyInt_FromLong is used for all these constants, regardless of
>> signedness or width, so any argument larger than LONG_MAX overflows,
>> *before* being converted to the arbitrary-size Python integer type.
>>
>> I don't know if this is a bug, or if I'm overlooking something. Is
>> there a way for me to use these constants with Python's arbitrary-size
>> integers?
>>
>
> All these constants are declared as "enum", so Cython promotes them to
> "int". Once again, Cython should have something like a "const" type
> qualifier to poperly declare these compile-time constants.
>
> As workaround, you could explicitly cast the constants like this
> "print long(<uint8_t>UINT8_MAX)"

This works great. Exactly what I was looking for, thanks.

Mansour

From robertwb at math.washington.edu  Tue Jan  3 03:00:23 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jan 2012 18:00:23 -0800
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
	<CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>
	<CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
Message-ID: <CADiQ+QDcoTg0=-fFCH+zCqkXwHDivawNUm0qU_2SLYbXEezcHQ@mail.gmail.com>

On Mon, Jan 2, 2012 at 5:48 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 2 January 2012 22:37, Mansour Moufid <mansourmoufid at gmail.com> wrote:
>> Now my issue is as follows.
>>
>> (I CCed the cython-users list if this question is more appropriate there.)
>>
>> I have a simple file, int.pyx:
>>
>> from libc.stdint cimport *
>> print long(UINT8_MAX)
>> print long(UINT16_MAX)
>> print long(UINT32_MAX)
>> print long(UINT64_MAX)
>>
>> with the usual setup.py stuff. Compiling and running:
>>
>> $ python setup.py build_ext --inplace
>> ...
>> int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
>> ...
>> $ python -c 'import int'
>> 255
>> 65535
>> -1
>> -1
>>
>> So obviously there are overflows here. Checking int.c, I see:
>>
>> ?/* "int.pyx":2
>> ?* from libc.stdint cimport *
>> ?* print long(UINT8_MAX) ? ? ? ? ? ? # <<<<<<<<<<<<<<
>> ?* print long(UINT16_MAX)
>> ?* print long(UINT32_MAX)
>> ?*/
>> ?__pyx_t_1 = PyInt_FromLong(UINT8_MAX);
>>
>> and so on...
>>
>> PyInt_FromLong is used for all these constants, regardless of
>> signedness or width, so any argument larger than LONG_MAX overflows,
>> *before* being converted to the arbitrary-size Python integer type.
>>
>> I don't know if this is a bug, or if I'm overlooking something. Is
>> there a way for me to use these constants with Python's arbitrary-size
>> integers?
>>
>
> All these constants are declared as "enum", so Cython promotes them to
> "int". Once again, Cython should have something like a "const" type
> qualifier to poperly declare these compile-time constants.
>
> As workaround, you could explicitly cast the constants like this
> "print long(<uint8_t>UINT8_MAX)"

I'm leaning towards declaring them as being the proper type to begin
with; what's to be gained by declaring these extern values as enums
(=const)? At least with the larger types we should do this to avoid
patently incorrect behavior, and this way they would be consistant
with the actual C for arithmetic promotion, etc.

- Robert

From stefan_ml at behnel.de  Tue Jan  3 13:10:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jan 2012 13:10:07 +0100
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CADiQ+QDcoTg0=-fFCH+zCqkXwHDivawNUm0qU_2SLYbXEezcHQ@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>	<CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>	<CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
	<CADiQ+QDcoTg0=-fFCH+zCqkXwHDivawNUm0qU_2SLYbXEezcHQ@mail.gmail.com>
Message-ID: <4F02F01F.3090900@behnel.de>

Robert Bradshaw, 03.01.2012 03:00:
> On Mon, Jan 2, 2012 at 5:48 PM, Lisandro Dalcin wrote:
>> On 2 January 2012 22:37, Mansour Moufid wrote:
>>> Now my issue is as follows.
>>>
>>> (I CCed the cython-users list if this question is more appropriate there.)
>>>
>>> I have a simple file, int.pyx:
>>>
>>> from libc.stdint cimport *
>>> print long(UINT8_MAX)
>>> print long(UINT16_MAX)
>>> print long(UINT32_MAX)
>>> print long(UINT64_MAX)
>>>
>>> with the usual setup.py stuff. Compiling and running:
>>>
>>> $ python setup.py build_ext --inplace
>>> ...
>>> int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
>>> ...
>>> $ python -c 'import int'
>>> 255
>>> 65535
>>> -1
>>> -1
>>>
>>> So obviously there are overflows here. Checking int.c, I see:
>>>
>>>   /* "int.pyx":2
>>>   * from libc.stdint cimport *
>>>   * print long(UINT8_MAX)             #<<<<<<<<<<<<<<
>>>   * print long(UINT16_MAX)
>>>   * print long(UINT32_MAX)
>>>   */
>>>   __pyx_t_1 = PyInt_FromLong(UINT8_MAX);
>>>
>>> and so on...
>>>
>>> PyInt_FromLong is used for all these constants, regardless of
>>> signedness or width, so any argument larger than LONG_MAX overflows,
>>> *before* being converted to the arbitrary-size Python integer type.
>>>
>>> I don't know if this is a bug, or if I'm overlooking something. Is
>>> there a way for me to use these constants with Python's arbitrary-size
>>> integers?
>>>
>>
>> All these constants are declared as "enum", so Cython promotes them to
>> "int". Once again, Cython should have something like a "const" type
>> qualifier to poperly declare these compile-time constants.
>>
>> As workaround, you could explicitly cast the constants like this
>> "print long(<uint8_t>UINT8_MAX)"
>
> I'm leaning towards declaring them as being the proper type to begin
> with; what's to be gained by declaring these extern values as enums
> (=const)? At least with the larger types we should do this to avoid
> patently incorrect behavior, and this way they would be consistant
> with the actual C for arithmetic promotion, etc.

+1

Stefan

From dalcinl at gmail.com  Thu Jan  5 00:52:12 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Jan 2012 20:52:12 -0300
Subject: [Cython] Fix integer width constant names in stdint.pxd
In-Reply-To: <CADiQ+QDcoTg0=-fFCH+zCqkXwHDivawNUm0qU_2SLYbXEezcHQ@mail.gmail.com>
References: <CALogXGXNCBCWWLE6KNC2b3ZTWJhHTZ6TrmAgX9Duv7WTQ+Kqww@mail.gmail.com>
	<CADiQ+QA6+KvgONzk-RYXewKRiBfCeeoCkyY3+AtQP52BC-DqWw@mail.gmail.com>
	<CALogXGW=BsfEQripTUSPBNiOVwacj_7aUOPSaq0__mxXSwZjng@mail.gmail.com>
	<CAEcYPwBi+oXQXwiDoAMHWvzydUy0vA+6AN3HH8xKc2mCvSrs1w@mail.gmail.com>
	<CADiQ+QDcoTg0=-fFCH+zCqkXwHDivawNUm0qU_2SLYbXEezcHQ@mail.gmail.com>
Message-ID: <CAEcYPwDxLHkdqnMG_Eyq_CKQFWQtbsmdrPWJCf4CUOfobp9YBQ@mail.gmail.com>

On 2 January 2012 23:00, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Mon, Jan 2, 2012 at 5:48 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> On 2 January 2012 22:37, Mansour Moufid <mansourmoufid at gmail.com> wrote:
>>> Now my issue is as follows.
>>>
>>> (I CCed the cython-users list if this question is more appropriate there.)
>>>
>>> I have a simple file, int.pyx:
>>>
>>> from libc.stdint cimport *
>>> print long(UINT8_MAX)
>>> print long(UINT16_MAX)
>>> print long(UINT32_MAX)
>>> print long(UINT64_MAX)
>>>
>>> with the usual setup.py stuff. Compiling and running:
>>>
>>> $ python setup.py build_ext --inplace
>>> ...
>>> int.c:566:3: warning: overflow in implicit constant conversion [-Woverflow]
>>> ...
>>> $ python -c 'import int'
>>> 255
>>> 65535
>>> -1
>>> -1
>>>
>>> So obviously there are overflows here. Checking int.c, I see:
>>>
>>> ?/* "int.pyx":2
>>> ?* from libc.stdint cimport *
>>> ?* print long(UINT8_MAX) ? ? ? ? ? ? # <<<<<<<<<<<<<<
>>> ?* print long(UINT16_MAX)
>>> ?* print long(UINT32_MAX)
>>> ?*/
>>> ?__pyx_t_1 = PyInt_FromLong(UINT8_MAX);
>>>
>>> and so on...
>>>
>>> PyInt_FromLong is used for all these constants, regardless of
>>> signedness or width, so any argument larger than LONG_MAX overflows,
>>> *before* being converted to the arbitrary-size Python integer type.
>>>
>>> I don't know if this is a bug, or if I'm overlooking something. Is
>>> there a way for me to use these constants with Python's arbitrary-size
>>> integers?
>>>
>>
>> All these constants are declared as "enum", so Cython promotes them to
>> "int". Once again, Cython should have something like a "const" type
>> qualifier to poperly declare these compile-time constants.
>>
>> As workaround, you could explicitly cast the constants like this
>> "print long(<uint8_t>UINT8_MAX)"
>
> I'm leaning towards declaring them as being the proper type to begin
> with; what's to be gained by declaring these extern values as enums
> (=const)? At least with the larger types we should do this to avoid
> patently incorrect behavior, and this way they would be consistant
> with the actual C for arithmetic promotion, etc.
>

Not sure about recent Cython releases, but in older ones you cannot do:

cdef char buf[UINT8_MAX]

unless UINT8_MAX was declared in Cython as a compile time constant.

However, I do agree that for the case of stdint.h, using matching
types instead of "enum" is way better.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From vitja.makarov at gmail.com  Sat Jan 14 15:19:24 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 14 Jan 2012 18:19:24 +0400
Subject: [Cython] sage-tests failures
Message-ID: <CAKGHGPQe7hVqCVFp++iuvDEQjZZmNy__WAArJZ-1sktd76NTRw@mail.gmail.com>

I've recently merged my def-node-refactoring branch and found some
bugs, thanks to sage-build.

Then I've found that sage-tests has >100 failures.
So I'm wondering does anybody know what's wrong with sage-tests?

-- 
vitja.

From robertwb at math.washington.edu  Sat Jan 14 18:38:11 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 14 Jan 2012 09:38:11 -0800
Subject: [Cython] sage-tests failures
In-Reply-To: <CAKGHGPQe7hVqCVFp++iuvDEQjZZmNy__WAArJZ-1sktd76NTRw@mail.gmail.com>
References: <CAKGHGPQe7hVqCVFp++iuvDEQjZZmNy__WAArJZ-1sktd76NTRw@mail.gmail.com>
Message-ID: <CADiQ+QD78WHnRZ37MXjJsTN2LyNpMwRU4taT9ZH3aztNT8W0iA@mail.gmail.com>

On Sat, Jan 14, 2012 at 6:19 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> I've recently merged my def-node-refactoring branch and found some
> bugs, thanks to sage-build.
>
> Then I've found that sage-tests has >100 failures.
> So I'm wondering does anybody know what's wrong with sage-tests?

Yeah, sage-tests is a great stress-tester for Cython. There were a
couple of spurious failures before, but nothing this bad.

I blame https://github.com/cython/cython/commit/bce8b981a3e71378a164e8c9acca5f00bbbe32d7
which is causing variables to be undefined. Arguably, this is a bug in
Sage, but it was a backwards incompatible change.

- Robert

From robertwb at math.washington.edu  Sun Jan 15 01:08:39 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 14 Jan 2012 16:08:39 -0800
Subject: [Cython] sage-tests failures
In-Reply-To: <CADiQ+QD78WHnRZ37MXjJsTN2LyNpMwRU4taT9ZH3aztNT8W0iA@mail.gmail.com>
References: <CAKGHGPQe7hVqCVFp++iuvDEQjZZmNy__WAArJZ-1sktd76NTRw@mail.gmail.com>
	<CADiQ+QD78WHnRZ37MXjJsTN2LyNpMwRU4taT9ZH3aztNT8W0iA@mail.gmail.com>
Message-ID: <CADiQ+QD30YxhvcocSqB=dzvD2s_d3Dj8WJz3XfYKZU_iyT+s9g@mail.gmail.com>

OK, thinks are looking a lot better, but there's still quite a few
random segfaults:

https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/674/console

On Sat, Jan 14, 2012 at 9:38 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Sat, Jan 14, 2012 at 6:19 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> I've recently merged my def-node-refactoring branch and found some
>> bugs, thanks to sage-build.
>>
>> Then I've found that sage-tests has >100 failures.
>> So I'm wondering does anybody know what's wrong with sage-tests?
>
> Yeah, sage-tests is a great stress-tester for Cython. There were a
> couple of spurious failures before, but nothing this bad.
>
> I blame https://github.com/cython/cython/commit/bce8b981a3e71378a164e8c9acca5f00bbbe32d7
> which is causing variables to be undefined. Arguably, this is a bug in
> Sage, but it was a backwards incompatible change.
>
> - Robert

From vitja.makarov at gmail.com  Sun Jan 15 18:32:42 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 15 Jan 2012 21:32:42 +0400
Subject: [Cython] sage-tests failures
In-Reply-To: <CADiQ+QD30YxhvcocSqB=dzvD2s_d3Dj8WJz3XfYKZU_iyT+s9g@mail.gmail.com>
References: <CAKGHGPQe7hVqCVFp++iuvDEQjZZmNy__WAArJZ-1sktd76NTRw@mail.gmail.com>
	<CADiQ+QD78WHnRZ37MXjJsTN2LyNpMwRU4taT9ZH3aztNT8W0iA@mail.gmail.com>
	<CADiQ+QD30YxhvcocSqB=dzvD2s_d3Dj8WJz3XfYKZU_iyT+s9g@mail.gmail.com>
Message-ID: <CAKGHGPQJuS94DXvEBUqzptceVwjraztHeRn8FfpuNiWaGd1nwA@mail.gmail.com>

2012/1/15 Robert Bradshaw <robertwb at math.washington.edu>:
> OK, thinks are looking a lot better, but there's still quite a few
> random segfaults:
>
> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/674/console
>

Cool. Have you modified a private copy of sage?

I've tried to reproduce segfaults at home but I was unable to compile
sage due to incompatible changes in cython.


> On Sat, Jan 14, 2012 at 9:38 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Sat, Jan 14, 2012 at 6:19 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> I've recently merged my def-node-refactoring branch and found some
>>> bugs, thanks to sage-build.
>>>
>>> Then I've found that sage-tests has >100 failures.
>>> So I'm wondering does anybody know what's wrong with sage-tests?
>>
>> Yeah, sage-tests is a great stress-tester for Cython. There were a
>> couple of spurious failures before, but nothing this bad.
>>
>> I blame https://github.com/cython/cython/commit/bce8b981a3e71378a164e8c9acca5f00bbbe32d7
>> which is causing variables to be undefined. Arguably, this is a bug in
>> Sage, but it was a backwards incompatible change.
>>
>> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
vitja.

From dtcaciuc at gmail.com  Sun Jan 15 19:01:43 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Sun, 15 Jan 2012 10:01:43 -0800
Subject: [Cython] Cannot assign type 'set<size_t> &' to 'set<size_t>'
In-Reply-To: <CAL41sfSqN08+CWSqDknZiATdF7tUr32v7MVYg13ZwiqLjwSoDQ@mail.gmail.com>
References: <CAL41sfSqN08+CWSqDknZiATdF7tUr32v7MVYg13ZwiqLjwSoDQ@mail.gmail.com>
Message-ID: <CAL41sfRgET_3KmN6BAarmjQ88p3ajvDtq0ohxhKA4kqNykE6BQ@mail.gmail.com>

Hi folks,

Since the original question, I've created a pull request with a
failing test and tried to get some discussion going, but so far no
answer. I'm a tad discouraged, since obviously there's movement on
mail list and with pull requests. Is a pull request a proper way to do
this? I completely understand if you guys don't have enough available
time to deal with it right now, however at least some acknowledgement
and feedback would be much appreciated.

Thanks,


Dimitri.

On Sun, Dec 18, 2011 at 8:17 PM, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> Hello everyone,
>
> Here's a small test case I'm trying to compile. I'm trying to pass a
> STL set reference to a method in a template class.
>
> x.pyx:
>
> ? ?from libcpp.set cimport set as cpp_set
>
> ? ?cdef extern from "x.hh":
>
> ? ? ? ?cdef cppclass Foo [T]:
> ? ? ? ? ? ?Foo()
> ? ? ? ? ? ?void set_x(cpp_set[size_t] & x)
>
> ? ?cpdef func():
> ? ? ? ?cdef Foo[int] foo
>
> ? ? ? ?cdef cpp_set[size_t] x
> ? ? ? ?cdef cpp_set[size_t] & xref = x
>
> ? ? ? ?foo.set_x(xref)
>
> x.hh:
>
> ? ?#include <set>
>
> ? ?template <typename T>
> ? ?struct Foo {
> ? ? ? ?void set_x(const std::set<size_t> & x) { /* do nothing */ }
> ? ?};
>
> To compile,
>
> ? ?bash $ cython --cplus x.pyx
>
> Which results in
>
> ? ?foo.set_x(xref)
> ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
> x.pyx:15:18: Cannot assign type 'set<size_t> &' to 'set<size_t>'
>
>
> However, if I remove the template parameter from Foo, everything works.
>
>
> y.pyx:
>
> ? ?from libcpp.set cimport set as cpp_set
>
> ? ?cdef extern from "y.hh":
>
> ? ? ? ?cdef cppclass Foo:
> ? ? ? ? ? ?Foo()
> ? ? ? ? ? ?void set_x(cpp_set[size_t] & x)
>
> ? ?cpdef func():
> ? ? ? ?cdef Foo foo
>
> ? ? ? ?cdef cpp_set[size_t] x
> ? ? ? ?cdef cpp_set[size_t] & xref = x
>
> ? ? ? ?foo.set_x(xref)
>
> y.hh:
>
> ? ?#include <set>
>
> ? ?struct Foo {
> ? ? ? ?void set_x(const std::set<size_t> & x) { /* do nothing */ }
> ? ?};
>
>
> From what I can tell, the CppClassType instance the CReferenceType is
> pointing to has the correct name "set<size_t>", however it's a
> different class instance. The particular failing expression is in
> `ExprNode.coerce_to`
>
> ? ?if not (str(src.type) == str(dst_type) or
> dst_type.assignable_from(src_type))
>
>
> I wish I could suggest a patch, but unfortunately I'm a complete
> newbie to Cython internals. Perhaps someone could give a few pointers
> as to what should be done to fix this?
>
> Thanks,
>
>
> Dimitri

From markflorisson88 at gmail.com  Mon Jan 16 19:09:47 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 16 Jan 2012 18:09:47 +0000
Subject: [Cython] Cannot assign type 'set<size_t> &' to 'set<size_t>'
In-Reply-To: <CAL41sfRgET_3KmN6BAarmjQ88p3ajvDtq0ohxhKA4kqNykE6BQ@mail.gmail.com>
References: <CAL41sfSqN08+CWSqDknZiATdF7tUr32v7MVYg13ZwiqLjwSoDQ@mail.gmail.com>
	<CAL41sfRgET_3KmN6BAarmjQ88p3ajvDtq0ohxhKA4kqNykE6BQ@mail.gmail.com>
Message-ID: <CANg26EV1TW5pKYfzWtbqZnB7exr4dbqBeQyALJkdSsgppAbifg@mail.gmail.com>

Dear Dimitri,

Sorry for the delay, many developers are busy with their lives. Thanks
for the report, it think that this was broken in
464923673475879fedc103ef2ee0260ba88d1493, the culprit is
https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L603
, I think it should read 'if dst_type.is_reference and not
self.type.is_reference:'. If you want you could substitute that line
with this line and write a simple test and make a new pull request, we
will merge it.

Cheers,

Mark

On 15 January 2012 18:01, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> Hi folks,
>
> Since the original question, I've created a pull request with a
> failing test and tried to get some discussion going, but so far no
> answer. I'm a tad discouraged, since obviously there's movement on
> mail list and with pull requests. Is a pull request a proper way to do
> this? I completely understand if you guys don't have enough available
> time to deal with it right now, however at least some acknowledgement
> and feedback would be much appreciated.
>
> Thanks,
>
>
> Dimitri.
>
> On Sun, Dec 18, 2011 at 8:17 PM, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
>> Hello everyone,
>>
>> Here's a small test case I'm trying to compile. I'm trying to pass a
>> STL set reference to a method in a template class.
>>
>> x.pyx:
>>
>> ? ?from libcpp.set cimport set as cpp_set
>>
>> ? ?cdef extern from "x.hh":
>>
>> ? ? ? ?cdef cppclass Foo [T]:
>> ? ? ? ? ? ?Foo()
>> ? ? ? ? ? ?void set_x(cpp_set[size_t] & x)
>>
>> ? ?cpdef func():
>> ? ? ? ?cdef Foo[int] foo
>>
>> ? ? ? ?cdef cpp_set[size_t] x
>> ? ? ? ?cdef cpp_set[size_t] & xref = x
>>
>> ? ? ? ?foo.set_x(xref)
>>
>> x.hh:
>>
>> ? ?#include <set>
>>
>> ? ?template <typename T>
>> ? ?struct Foo {
>> ? ? ? ?void set_x(const std::set<size_t> & x) { /* do nothing */ }
>> ? ?};
>>
>> To compile,
>>
>> ? ?bash $ cython --cplus x.pyx
>>
>> Which results in
>>
>> ? ?foo.set_x(xref)
>> ? ? ? ? ? ? ? ? ^
>> ------------------------------------------------------------
>> x.pyx:15:18: Cannot assign type 'set<size_t> &' to 'set<size_t>'
>>
>>
>> However, if I remove the template parameter from Foo, everything works.
>>
>>
>> y.pyx:
>>
>> ? ?from libcpp.set cimport set as cpp_set
>>
>> ? ?cdef extern from "y.hh":
>>
>> ? ? ? ?cdef cppclass Foo:
>> ? ? ? ? ? ?Foo()
>> ? ? ? ? ? ?void set_x(cpp_set[size_t] & x)
>>
>> ? ?cpdef func():
>> ? ? ? ?cdef Foo foo
>>
>> ? ? ? ?cdef cpp_set[size_t] x
>> ? ? ? ?cdef cpp_set[size_t] & xref = x
>>
>> ? ? ? ?foo.set_x(xref)
>>
>> y.hh:
>>
>> ? ?#include <set>
>>
>> ? ?struct Foo {
>> ? ? ? ?void set_x(const std::set<size_t> & x) { /* do nothing */ }
>> ? ?};
>>
>>
>> From what I can tell, the CppClassType instance the CReferenceType is
>> pointing to has the correct name "set<size_t>", however it's a
>> different class instance. The particular failing expression is in
>> `ExprNode.coerce_to`
>>
>> ? ?if not (str(src.type) == str(dst_type) or
>> dst_type.assignable_from(src_type))
>>
>>
>> I wish I could suggest a patch, but unfortunately I'm a complete
>> newbie to Cython internals. Perhaps someone could give a few pointers
>> as to what should be done to fix this?
>>
>> Thanks,
>>
>>
>> Dimitri
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Wed Jan 18 21:30:43 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 19 Jan 2012 00:30:43 +0400
Subject: [Cython] Speedup module-level lookup
Message-ID: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>

I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.

In this example bar() is 1.6 time faster (500us against 842us):

C = 123
def foo(a):
? ? return C * adef bar():
? ? for i in range(10000):? ? ? ? foo(i)
Here is proof of
concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9

So the question is: does it worth it?

-- 
vitja.

From robertwb at math.washington.edu  Thu Jan 19 08:28:14 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 18 Jan 2012 23:28:14 -0800
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
Message-ID: <CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>

I think the right thing to do here is make all module-level globals
into "cdef public" attributes, i.e. C globals with getters and setters
for Python space. I'm not sure whether this would best be done by
creating a custom dict or module subclass, but it would probably be
cleaner and afford much more than a 1.6x speedup.

- Robert

On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>
> In this example bar() is 1.6 time faster (500us against 842us):
>
> C = 123
> def foo(a):
> ? ? return C * adef bar():
> ? ? for i in range(10000):? ? ? ? foo(i)
> Here is proof of
> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>
> So the question is: does it worth it?
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From sccolbert at gmail.com  Thu Jan 19 08:42:11 2012
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 19 Jan 2012 01:42:11 -0600
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
Message-ID: <CAPc_zRV2oyrT-s2U3=ALdCn7tX1QU_6qfPnXL9YdJhguxgOQbA@mail.gmail.com>

AFAIK, a module's dict is readonly, so I don't believe a dict subclass will
work there (I could be wrong) unless you hack up the module object from C.
You can do it with descriptors on a ModuleType however, which should be
plenty fast from Cython-land.

In [16]: class AGetter(object):
   ....:     def __get__(self, obj, cls):
   ....:         return obj.a
   ....:     def __set__(self, obj, val):
   ....:         obj.a = val
   ....:

In [17]: class MyMod(types.ModuleType):
   ....:     b = AGetter()
   ....:

In [18]: mmod = MyMod('my_mod')

In [20]: mmod.__dict__['a'] = 42

In [21]: mmod.a
Out[21]: 42

In [22]: mmod.b
Out[22]: 42

In [23]: mmod.b = 87

In [24]: mmod.a
Out[24]: 87
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120119/d6486493/attachment.html>

From vitja.makarov at gmail.com  Thu Jan 19 08:49:51 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 19 Jan 2012 11:49:51 +0400
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
Message-ID: <CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>

2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
> I think the right thing to do here is make all module-level globals
> into "cdef public" attributes, i.e. C globals with getters and setters
> for Python space. I'm not sure whether this would best be done by
> creating a custom dict or module subclass, but it would probably be
> cleaner and afford much more than a 1.6x speedup.
>
> - Robert
>
> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>
>> In this example bar() is 1.6 time faster (500us against 842us):
>>
>> C = 123
>> def foo(a):
>> ? ? return C * adef bar():
>> ? ? for i in range(10000):? ? ? ? foo(i)
>> Here is proof of
>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>
>> So the question is: does it worth it?
>>

Yes, nice idea.
It's possible to subclass PyModuleObject and I didn't find any use of
PyModule_CheckExact() in CPython's sources:

import types
import sys

global_foo = 1234

class CustomModule(types.ModuleType):
    def __init__(self, name):
        types.ModuleType.__init__(self, name)
        sys.modules[name] = self

    @property
    def foo(self):
        return global_foo

    @foo.setter
    def foo(self, value):
        global global_foo
        global_foo = value

CustomModule('foo')

import foo
print foo.foo


-- 
vitja.

From vitja.makarov at gmail.com  Thu Jan 19 08:53:23 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 19 Jan 2012 11:53:23 +0400
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
Message-ID: <CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>

2012/1/19 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
>> I think the right thing to do here is make all module-level globals
>> into "cdef public" attributes, i.e. C globals with getters and setters
>> for Python space. I'm not sure whether this would best be done by
>> creating a custom dict or module subclass, but it would probably be
>> cleaner and afford much more than a 1.6x speedup.
>>
>> - Robert
>>
>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>
>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>
>>> C = 123
>>> def foo(a):
>>> ? ? return C * adef bar():
>>> ? ? for i in range(10000):? ? ? ? foo(i)
>>> Here is proof of
>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>
>>> So the question is: does it worth it?
>>>
>
> Yes, nice idea.
> It's possible to subclass PyModuleObject and I didn't find any use of
> PyModule_CheckExact() in CPython's sources:
>
> import types
> import sys
>
> global_foo = 1234
>
> class CustomModule(types.ModuleType):
> ? ?def __init__(self, name):
> ? ? ? ?types.ModuleType.__init__(self, name)
> ? ? ? ?sys.modules[name] = self
>
> ? ?@property
> ? ?def foo(self):
> ? ? ? ?return global_foo
>
> ? ?@foo.setter
> ? ?def foo(self, value):
> ? ? ? ?global global_foo
> ? ? ? ?global_foo = value
>
> CustomModule('foo')
>
> import foo
> print foo.foo
>

But this seems to break globals().

-- 
vitja.

From robertwb at math.washington.edu  Thu Jan 19 09:00:51 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jan 2012 00:00:51 -0800
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
Message-ID: <CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>

On Wed, Jan 18, 2012 at 11:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/19 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
>>> I think the right thing to do here is make all module-level globals
>>> into "cdef public" attributes, i.e. C globals with getters and setters
>>> for Python space. I'm not sure whether this would best be done by
>>> creating a custom dict or module subclass, but it would probably be
>>> cleaner and afford much more than a 1.6x speedup.
>>>
>>> - Robert
>>>
>>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>>
>>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>>
>>>> C = 123
>>>> def foo(a):
>>>> ? ? return C * adef bar():
>>>> ? ? for i in range(10000):? ? ? ? foo(i)
>>>> Here is proof of
>>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>>
>>>> So the question is: does it worth it?
>>>>
>>
>> Yes, nice idea.
>> It's possible to subclass PyModuleObject and I didn't find any use of
>> PyModule_CheckExact() in CPython's sources:
>>
>> import types
>> import sys
>>
>> global_foo = 1234
>>
>> class CustomModule(types.ModuleType):
>> ? ?def __init__(self, name):
>> ? ? ? ?types.ModuleType.__init__(self, name)
>> ? ? ? ?sys.modules[name] = self
>>
>> ? ?@property
>> ? ?def foo(self):
>> ? ? ? ?return global_foo
>>
>> ? ?@foo.setter
>> ? ?def foo(self, value):
>> ? ? ? ?global global_foo
>> ? ? ? ?global_foo = value
>>
>> CustomModule('foo')
>>
>> import foo
>> print foo.foo
>>
>
> But this seems to break globals().

How so? We have to hack globals() to get it to work for a Cython
module anyways. (I wonder if this must return a dict, or would any
mapping (or subclass of dict) be sufficient...)

- Robert

From markflorisson88 at gmail.com  Thu Jan 19 09:04:30 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 19 Jan 2012 08:04:30 +0000
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
Message-ID: <CANg26EW8dTByT06VicXqZszrfY3X6UDgLLQf8xOLwSV3axDUUg@mail.gmail.com>

On 19 January 2012 08:00, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Jan 18, 2012 at 11:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/19 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
>>>> I think the right thing to do here is make all module-level globals
>>>> into "cdef public" attributes, i.e. C globals with getters and setters
>>>> for Python space. I'm not sure whether this would best be done by
>>>> creating a custom dict or module subclass, but it would probably be
>>>> cleaner and afford much more than a 1.6x speedup.
>>>>
>>>> - Robert
>>>>
>>>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>>>
>>>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>>>
>>>>> C = 123
>>>>> def foo(a):
>>>>> ? ? return C * adef bar():
>>>>> ? ? for i in range(10000):? ? ? ? foo(i)
>>>>> Here is proof of
>>>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>>>
>>>>> So the question is: does it worth it?
>>>>>
>>>
>>> Yes, nice idea.
>>> It's possible to subclass PyModuleObject and I didn't find any use of
>>> PyModule_CheckExact() in CPython's sources:
>>>
>>> import types
>>> import sys
>>>
>>> global_foo = 1234
>>>
>>> class CustomModule(types.ModuleType):
>>> ? ?def __init__(self, name):
>>> ? ? ? ?types.ModuleType.__init__(self, name)
>>> ? ? ? ?sys.modules[name] = self
>>>
>>> ? ?@property
>>> ? ?def foo(self):
>>> ? ? ? ?return global_foo
>>>
>>> ? ?@foo.setter
>>> ? ?def foo(self, value):
>>> ? ? ? ?global global_foo
>>> ? ? ? ?global_foo = value
>>>
>>> CustomModule('foo')
>>>
>>> import foo
>>> print foo.foo
>>>
>>
>> But this seems to break globals().
>
> How so? We have to hack globals() to get it to work for a Cython
> module anyways. (I wonder if this must return a dict, or would any
> mapping (or subclass of dict) be sufficient...)
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

You'd also want this to work from python space using module.__dict__
or vars(module). I think the custom dict could solve this. Or would
you make __dict__ a property as well? (I don't know if vars() would
still break).

From sccolbert at gmail.com  Thu Jan 19 09:18:21 2012
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 19 Jan 2012 02:18:21 -0600
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
Message-ID: <CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>

If it doesn't pass PyDict_CheckExact you won't be able to use it as the
globals to eval or exec. That's assuming you hack the module type so you
can change its __dict__.
On Jan 19, 2012 2:01 AM, "Robert Bradshaw" <robertwb at math.washington.edu>
wrote:

> On Wed, Jan 18, 2012 at 11:53 PM, Vitja Makarov <vitja.makarov at gmail.com>
> wrote:
> > 2012/1/19 Vitja Makarov <vitja.makarov at gmail.com>:
> >> 2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
> >>> I think the right thing to do here is make all module-level globals
> >>> into "cdef public" attributes, i.e. C globals with getters and setters
> >>> for Python space. I'm not sure whether this would best be done by
> >>> creating a custom dict or module subclass, but it would probably be
> >>> cleaner and afford much more than a 1.6x speedup.
> >>>
> >>> - Robert
> >>>
> >>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov <
> vitja.makarov at gmail.com> wrote:
> >>>> I tried to optimize module lookups (__pyx_m) by caching internal
> PyDict state.
> >>>>
> >>>> In this example bar() is 1.6 time faster (500us against 842us):
> >>>>
> >>>> C = 123
> >>>> def foo(a):
> >>>>     return C * adef bar():
> >>>>     for i in range(10000):        foo(i)
> >>>> Here is proof of
> >>>> concept:
> https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
> >>>>
> >>>> So the question is: does it worth it?
> >>>>
> >>
> >> Yes, nice idea.
> >> It's possible to subclass PyModuleObject and I didn't find any use of
> >> PyModule_CheckExact() in CPython's sources:
> >>
> >> import types
> >> import sys
> >>
> >> global_foo = 1234
> >>
> >> class CustomModule(types.ModuleType):
> >>    def __init__(self, name):
> >>        types.ModuleType.__init__(self, name)
> >>        sys.modules[name] = self
> >>
> >>    @property
> >>    def foo(self):
> >>        return global_foo
> >>
> >>    @foo.setter
> >>    def foo(self, value):
> >>        global global_foo
> >>        global_foo = value
> >>
> >> CustomModule('foo')
> >>
> >> import foo
> >> print foo.foo
> >>
> >
> > But this seems to break globals().
>
> How so? We have to hack globals() to get it to work for a Cython
> module anyways. (I wonder if this must return a dict, or would any
> mapping (or subclass of dict) be sufficient...)
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120119/77f72e95/attachment.html>

From robertwb at math.washington.edu  Thu Jan 19 09:53:49 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jan 2012 00:53:49 -0800
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
Message-ID: <CADiQ+QA5xHe20MzeSf8gdfQax4UDfJEbmc0yZ9o6tVvbZsBXrQ@mail.gmail.com>

On Thu, Jan 19, 2012 at 12:18 AM, Chris Colbert <sccolbert at gmail.com> wrote:
> If it doesn't pass PyDict_CheckExact you won't be able to use it as the
> globals to eval or exec.

:(. I wonder how many other places have similar restrictions, perhaps
even implicitly. In particular, this would mean that an eval statement
modifying globals() would be difficult to efficiently detect. Still,
if this can be done at all, the massive speedup for the common case
could make it worth it.

> That's assuming you hack the module type so you can
> change its __dict__.

I don't think that's near as big of a hurdle (from C).

> On Jan 19, 2012 2:01 AM, "Robert Bradshaw" <robertwb at math.washington.edu>
> wrote:
>>
>> On Wed, Jan 18, 2012 at 11:53 PM, Vitja Makarov <vitja.makarov at gmail.com>
>> wrote:
>> > 2012/1/19 Vitja Makarov <vitja.makarov at gmail.com>:
>> >> 2012/1/19 Robert Bradshaw <robertwb at math.washington.edu>:
>> >>> I think the right thing to do here is make all module-level globals
>> >>> into "cdef public" attributes, i.e. C globals with getters and setters
>> >>> for Python space. I'm not sure whether this would best be done by
>> >>> creating a custom dict or module subclass, but it would probably be
>> >>> cleaner and afford much more than a 1.6x speedup.
>> >>>
>> >>> - Robert
>> >>>
>> >>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov
>> >>> <vitja.makarov at gmail.com> wrote:
>> >>>> I tried to optimize module lookups (__pyx_m) by caching internal
>> >>>> PyDict state.
>> >>>>
>> >>>> In this example bar() is 1.6 time faster (500us against 842us):
>> >>>>
>> >>>> C = 123
>> >>>> def foo(a):
>> >>>> ? ? return C * adef bar():
>> >>>> ? ? for i in range(10000):? ? ? ? foo(i)
>> >>>> Here is proof of
>> >>>>
>> >>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>> >>>>
>> >>>> So the question is: does it worth it?
>> >>>>
>> >>
>> >> Yes, nice idea.
>> >> It's possible to subclass PyModuleObject and I didn't find any use of
>> >> PyModule_CheckExact() in CPython's sources:
>> >>
>> >> import types
>> >> import sys
>> >>
>> >> global_foo = 1234
>> >>
>> >> class CustomModule(types.ModuleType):
>> >> ? ?def __init__(self, name):
>> >> ? ? ? ?types.ModuleType.__init__(self, name)
>> >> ? ? ? ?sys.modules[name] = self
>> >>
>> >> ? ?@property
>> >> ? ?def foo(self):
>> >> ? ? ? ?return global_foo
>> >>
>> >> ? ?@foo.setter
>> >> ? ?def foo(self, value):
>> >> ? ? ? ?global global_foo
>> >> ? ? ? ?global_foo = value
>> >>
>> >> CustomModule('foo')
>> >>
>> >> import foo
>> >> print foo.foo
>> >>
>> >
>> > But this seems to break globals().
>>
>> How so? We have to hack globals() to get it to work for a Cython
>> module anyways. (I wonder if this must return a dict, or would any
>> mapping (or subclass of dict) be sufficient...)
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From anders at embl.de  Thu Jan 19 19:26:27 2012
From: anders at embl.de (Simon Anders)
Date: Thu, 19 Jan 2012 19:26:27 +0100
Subject: [Cython] Cython crash: C++ class with missing default constructor
Message-ID: <4F186053.1000706@embl.de>

Hi,

the following very short Cython code crashes the Cython compiler (v0.15.1):

---8<---
cdef extern from "foo.h":
    cdef cppclass foo:
       pass

foo()
---8<---

The stack trace is attached.

Best regards
   Simon

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: traceback
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120119/a29f417c/attachment.ksh>

From anders at embl.de  Thu Jan 19 21:10:47 2012
From: anders at embl.de (Simon Anders)
Date: Thu, 19 Jan 2012 21:10:47 +0100
Subject: [Cython] non-template nested C++ classes
Message-ID: <4F1878C7.1060705@embl.de>

Hi,

I'm currently experimenting with Cython's support for nested C++ classes 
and might havce encountered a bug.

I have attempted to strip down the example from the documentation to its 
bare minimum. This here works fine:

---8<---
cdef extern from "foo":
     cdef cppclass outer[T]:
         cppclass inner:
            pass

cdef outer[int].inner foo
---8<---

Next, I remove the template parameter as well. After all, not every 
outer class containing an inner class is a template class.

---8<---
cdef extern from "foo":
     cdef cppclass outer:
         cppclass inner:
            pass

cdef outer.inner foo
---8<---

Now, I get this error message:
    'outer' is not a cimported module

It seems that without the square brackets, Cython no longer recognizes 
'outer' as a class name and thinks it must be a module because it is 
followed by a dot. I suppose this is not what should happen, right?

Best regards
   Simon

From anders at embl.de  Thu Jan 19 22:46:48 2012
From: anders at embl.de (Simon Anders)
Date: Thu, 19 Jan 2012 22:46:48 +0100
Subject: [Cython] nested C++ classes, the third: default constructor
Message-ID: <4F188F48.2080704@embl.de>

Hi,

sorry for spreading these issues into three mails, but I cannot quite 
figure out whether they are related or not. So, here is the third 
installment of my adventures with nested classes.

Consider the following code which compiles fine:

---8<---
cdef extern from "foo":
    cdef cppclass outer[T]:
       outer( )
       cppclass inner:
          pass

cdef outer[int].inner bar
---8<---

If I change 'outer' to lose its default constructor, the last line, 
which instantiates 'inner', causes an error:

---8<---
cdef extern from "foo":
    cdef cppclass outer[T]:
       outer( int )
       cppclass inner:
          pass

cdef outer[int].inner bar
---8<---

The only change is that the constructor to 'outer' now takes an 
argument. Now, the last line causes this error:
    "C++ class must have a default constructor to be stack allocated".

However, 'inner' does have a default constructor, only 'outer' does not.

Could it be that Cython looks for the constructor for the outer class 
where it should look for the constructor for the inner class?

Cheers
   Simon

From mansourmoufid at gmail.com  Fri Jan 20 20:13:28 2012
From: mansourmoufid at gmail.com (Mansour Moufid)
Date: Fri, 20 Jan 2012 14:13:28 -0500
Subject: [Cython] More typed constants (was: Fix integer width constant
	names in stdint.pxd)
Message-ID: <CALogXGV1xwJ7Cptu+Be_to58YdUT1r3swnZF9aYMYZiE1aUAkQ@mail.gmail.com>

Hello again,

Attached is a patch that continues with the idea of declaring
constants using their corresponding type.

Great work on Cython, by the way. It's very useful.

Mansour
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Continue-defining-constants-using-corresponding-type.patch
Type: text/x-patch
Size: 3920 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120120/edace46d/attachment.bin>

From stefan_ml at behnel.de  Sat Jan 21 06:58:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jan 2012 06:58:14 +0100
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
Message-ID: <4F1A53F6.2050906@behnel.de>

Chris Colbert, 19.01.2012 09:18:
> If it doesn't pass PyDict_CheckExact you won't be able to use it as the
> globals to eval or exec.

What makes you say that? I tried and it worked for me, all the way back to
Python 2.4:

--------------------
Python 2.4.6 (#2, Jan 21 2010, 23:45:25)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class MyDict(dict): pass
>>> eval('1+1', MyDict())
2
>>> exec '1+1' in MyDict()
>>>
--------------------

I only see a couple of calls to PyDict_CheckExact() in CPython's sources
and they usually seem to be related to special casing for performance
reasons. Nothing that should impact a module's globals.

Besides, Cython controls its own language usages of eval and exec.

Stefan

From stefan_ml at behnel.de  Sat Jan 21 07:00:00 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jan 2012 07:00:00 +0100
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
Message-ID: <4F1A5460.6090908@behnel.de>

Vitja Makarov, 19.01.2012 08:49:
> 2012/1/19 Robert Bradshaw:
>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov wrote:
>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>
>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>
>>> C = 123
>>> def foo(a):
>>>     return C * adef bar():
>>>     for i in range(10000):        foo(i)
>>> Here is proof of
>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>
>>> So the question is: does it worth it?
>>
>> I think the right thing to do here is make all module-level globals
>> into "cdef public" attributes, i.e. C globals with getters and setters
>> for Python space. I'm not sure whether this would best be done by
>> creating a custom dict or module subclass, but it would probably be
>> cleaner and afford much more than a 1.6x speedup.
> 
> Yes, nice idea.
> It's possible to subclass PyModuleObject and I didn't find any use of
> PyModule_CheckExact() in CPython's sources:
> 
> import types
> import sys
> 
> global_foo = 1234
> 
> class CustomModule(types.ModuleType):
>     def __init__(self, name):
>         types.ModuleType.__init__(self, name)
>         sys.modules[name] = self
> 
>     @property
>     def foo(self):
>         return global_foo
> 
>     @foo.setter
>     def foo(self, value):
>         global global_foo
>         global_foo = value
> 
> CustomModule('foo')
> 
> import foo
> print foo.foo

The one thing I don't currently see is how to get the module subtype
instantiated in a safe and portable way.

The normal way to create the module in Python 2.x is a call to
Py_InitModule*(), which internally does a PyImport_AddModule(). We may get
away with creating and registering the module object before calling into
Py_InitModule*(), so that PyImport_AddModule() finds it there. At least,
the internal checks on modules seem to use PyModule_Check() and not
PyModule_CheckExact(), so someone seems to have already thought about this.

In Python 3.x, the situation is different. There is no lookup involved and
the module is always newly instantiated. That may mean that we have to copy
the module creation code into Cython. But that doesn't look like a huge
drawback (except for compatibility to potential future changes), because we
already do most of the module initialisation ourselves anyway, especially
now that we have CyFunction.

I start feeling a bit like Linus Torvalds when he broke his minix
installation and went: "ok, what else do I need to add to this terminal
emulator in order to make it an operating system?"

Stefan


From sccolbert at gmail.com  Sat Jan 21 07:09:49 2012
From: sccolbert at gmail.com (Chris Colbert)
Date: Sat, 21 Jan 2012 00:09:49 -0600
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <4F1A53F6.2050906@behnel.de>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
	<4F1A53F6.2050906@behnel.de>
Message-ID: <CAPc_zRXrnXi+AuZcekKJMEZ8XagtuTbyRutxaFEVCayKCjr-3Q@mail.gmail.com>

On Fri, Jan 20, 2012 at 11:58 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Chris Colbert, 19.01.2012 09:18:
> > If it doesn't pass PyDict_CheckExact you won't be able to use it as the
> > globals to eval or exec.
>
> What makes you say that? I tried and it worked for me, all the way back to
> Python 2.4:
>
>
Ah, you're right. I was mixing up issues I'd dealt with recently. The issue
with the eval/exec is that the globals must be a dict (or dict subclass)
whereas the locals can be a mapping. However, if you subclass dict for your
globals, any overridden __getitem__ or __setitem__ will not be called.
There is this line for eval/exec in ceval.c

if (!PyDict_Check(globals)) {
        PyErr_SetString(PyExc_TypeError,
            "exec: arg 2 must be a dictionary or None");
        return -1;

}

This allows the eval loop to use PyDict_GetItem on the globals instead of
PyObject_GetItem, so no subclass method overrides will be called.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120121/2822a9c8/attachment.html>

From robertwb at math.washington.edu  Sat Jan 21 07:21:37 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Jan 2012 22:21:37 -0800
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <4F1A5460.6090908@behnel.de>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<4F1A5460.6090908@behnel.de>
Message-ID: <CADiQ+QCOXcDcVU8Uy6LHakWG9nNw=6aWR4xiwCJNAS4k=D9sTw@mail.gmail.com>

On Fri, Jan 20, 2012 at 10:00 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 19.01.2012 08:49:
>> 2012/1/19 Robert Bradshaw:
>>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov wrote:
>>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>>
>>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>>
>>>> C = 123
>>>> def foo(a):
>>>> ? ? return C * adef bar():
>>>> ? ? for i in range(10000): ? ? ? ?foo(i)
>>>> Here is proof of
>>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>>
>>>> So the question is: does it worth it?
>>>
>>> I think the right thing to do here is make all module-level globals
>>> into "cdef public" attributes, i.e. C globals with getters and setters
>>> for Python space. I'm not sure whether this would best be done by
>>> creating a custom dict or module subclass, but it would probably be
>>> cleaner and afford much more than a 1.6x speedup.
>>
>> Yes, nice idea.
>> It's possible to subclass PyModuleObject and I didn't find any use of
>> PyModule_CheckExact() in CPython's sources:
>>
>> import types
>> import sys
>>
>> global_foo = 1234
>>
>> class CustomModule(types.ModuleType):
>> ? ? def __init__(self, name):
>> ? ? ? ? types.ModuleType.__init__(self, name)
>> ? ? ? ? sys.modules[name] = self
>>
>> ? ? @property
>> ? ? def foo(self):
>> ? ? ? ? return global_foo
>>
>> ? ? @foo.setter
>> ? ? def foo(self, value):
>> ? ? ? ? global global_foo
>> ? ? ? ? global_foo = value
>>
>> CustomModule('foo')
>>
>> import foo
>> print foo.foo
>
> The one thing I don't currently see is how to get the module subtype
> instantiated in a safe and portable way.
>
> The normal way to create the module in Python 2.x is a call to
> Py_InitModule*(), which internally does a PyImport_AddModule(). We may get
> away with creating and registering the module object before calling into
> Py_InitModule*(), so that PyImport_AddModule() finds it there. At least,
> the internal checks on modules seem to use PyModule_Check() and not
> PyModule_CheckExact(), so someone seems to have already thought about this.
>
> In Python 3.x, the situation is different. There is no lookup involved and
> the module is always newly instantiated. That may mean that we have to copy
> the module creation code into Cython. But that doesn't look like a huge
> drawback (except for compatibility to potential future changes), because we
> already do most of the module initialisation ourselves anyway, especially
> now that we have CyFunction.

Or swap out its ob_type pointer after it's created... It's going to be
messy unless we can directly add hooks into its __dict__ though.

> I start feeling a bit like Linus Torvalds when he broke his minix
> installation and went: "ok, what else do I need to add to this terminal
> emulator in order to make it an operating system?"
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Sat Jan 21 09:35:44 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 21 Jan 2012 12:35:44 +0400
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <4F1A53F6.2050906@behnel.de>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
	<4F1A53F6.2050906@behnel.de>
Message-ID: <CAKGHGPR-B6z5P=nFRD2yqonuyxhyXwsGAfrxYKFLTAKGdCCboQ@mail.gmail.com>

2012/1/21 Stefan Behnel <stefan_ml at behnel.de>:
> Chris Colbert, 19.01.2012 09:18:
>> If it doesn't pass PyDict_CheckExact you won't be able to use it as the
>> globals to eval or exec.
>
> What makes you say that? I tried and it worked for me, all the way back to
> Python 2.4:
>
> --------------------
> Python 2.4.6 (#2, Jan 21 2010, 23:45:25)
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> class MyDict(dict): pass
>>>> eval('1+1', MyDict())
> 2
>>>> exec '1+1' in MyDict()
>>>>
> --------------------
>
> I only see a couple of calls to PyDict_CheckExact() in CPython's sources
> and they usually seem to be related to special casing for performance
> reasons. Nothing that should impact a module's globals.
>
> Besides, Cython controls its own language usages of eval and exec.
>

Cool!
It seems that python internally uses PyObject_GetItem() for module
level lookups and not PyDict_GetItem().
Btw we use __Pyx_GetName() that calls PyObject_GetAttr() that isn't
exactly the same for module lookups:

# Works in Cython and doesn't work in Python
print __class__

So we can override __getitem__() and __setitem__():
class MyDict(dict):
    def __init__(self):
        self._dict = {}

    def __getitem__(self, key):
        print '__getitem__', key
        return self._dict[key]

    def __setitem__(self, key, value):
        print '__setitem__', key, value
        self._dict[key] = value

    def __getattr__(self, key):
        print '__getattr__'

d = MyDict()
exec('x = 1; print x', d)
eval('x', d)
$ python foo.py
__setitem__ x 1
__getitem__ x
1
__getitem__ x


So we can make globals() return special dict with custom
__setitem__()/__getitem__(). But it seems that we'll have to override
many dict's standard methods like values(), update() and so on. That
would be hard.


-- 
vitja.

From vitja.makarov at gmail.com  Sat Jan 21 09:40:21 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 21 Jan 2012 12:40:21 +0400
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <4F1A5460.6090908@behnel.de>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<4F1A5460.6090908@behnel.de>
Message-ID: <CAKGHGPSZTA9HYcbHrNjUNYTz41MhtcfkpLGaS2X8_Xoo01hwaw@mail.gmail.com>

2012/1/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 19.01.2012 08:49:
>> 2012/1/19 Robert Bradshaw:
>>> On Wed, Jan 18, 2012 at 12:30 PM, Vitja Makarov wrote:
>>>> I tried to optimize module lookups (__pyx_m) by caching internal PyDict state.
>>>>
>>>> In this example bar() is 1.6 time faster (500us against 842us):
>>>>
>>>> C = 123
>>>> def foo(a):
>>>> ? ? return C * adef bar():
>>>> ? ? for i in range(10000): ? ? ? ?foo(i)
>>>> Here is proof of
>>>> concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db499680b2a8d9
>>>>
>>>> So the question is: does it worth it?
>>>
>>> I think the right thing to do here is make all module-level globals
>>> into "cdef public" attributes, i.e. C globals with getters and setters
>>> for Python space. I'm not sure whether this would best be done by
>>> creating a custom dict or module subclass, but it would probably be
>>> cleaner and afford much more than a 1.6x speedup.
>>
>> Yes, nice idea.
>> It's possible to subclass PyModuleObject and I didn't find any use of
>> PyModule_CheckExact() in CPython's sources:
>>
>> import types
>> import sys
>>
>> global_foo = 1234
>>
>> class CustomModule(types.ModuleType):
>> ? ? def __init__(self, name):
>> ? ? ? ? types.ModuleType.__init__(self, name)
>> ? ? ? ? sys.modules[name] = self
>>
>> ? ? @property
>> ? ? def foo(self):
>> ? ? ? ? return global_foo
>>
>> ? ? @foo.setter
>> ? ? def foo(self, value):
>> ? ? ? ? global global_foo
>> ? ? ? ? global_foo = value
>>
>> CustomModule('foo')
>>
>> import foo
>> print foo.foo
>
> The one thing I don't currently see is how to get the module subtype
> instantiated in a safe and portable way.
>

We can do the same as types module:

ModuleType = type(sys)

or type(__builtins__) since we already got it (__pyx_b)

> The normal way to create the module in Python 2.x is a call to
> Py_InitModule*(), which internally does a PyImport_AddModule(). We may get
> away with creating and registering the module object before calling into
> Py_InitModule*(), so that PyImport_AddModule() finds it there. At least,
> the internal checks on modules seem to use PyModule_Check() and not
> PyModule_CheckExact(), so someone seems to have already thought about this.
>
> In Python 3.x, the situation is different. There is no lookup involved and
> the module is always newly instantiated. That may mean that we have to copy
> the module creation code into Cython. But that doesn't look like a huge
> drawback (except for compatibility to potential future changes), because we
> already do most of the module initialisation ourselves anyway, especially
> now that we have CyFunction.
>
> I start feeling a bit like Linus Torvalds when he broke his minix
> installation and went: "ok, what else do I need to add to this terminal
> emulator in order to make it an operating system?"
>



-- 
vitja.

From sccolbert at gmail.com  Sat Jan 21 19:08:26 2012
From: sccolbert at gmail.com (Chris Colbert)
Date: Sat, 21 Jan 2012 12:08:26 -0600
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAKGHGPR-B6z5P=nFRD2yqonuyxhyXwsGAfrxYKFLTAKGdCCboQ@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
	<4F1A53F6.2050906@behnel.de>
	<CAKGHGPR-B6z5P=nFRD2yqonuyxhyXwsGAfrxYKFLTAKGdCCboQ@mail.gmail.com>
Message-ID: <CAPc_zRW9ihiMft9ex3gOhH-5tJsE1jfhhACTBmHfBCFNg49s5A@mail.gmail.com>

On Sat, Jan 21, 2012 at 2:35 AM, Vitja Makarov <vitja.makarov at gmail.com>wrote:

> 2012/1/21 Stefan Behnel <stefan_ml at behnel.de>:
> > Chris Colbert, 19.01.2012 09:18:
> >> If it doesn't pass PyDict_CheckExact you won't be able to use it as the
> >> globals to eval or exec.
> >
> > What makes you say that? I tried and it worked for me, all the way back
> to
> > Python 2.4:
> >
> > --------------------
> > Python 2.4.6 (#2, Jan 21 2010, 23:45:25)
> > [GCC 4.4.1] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> class MyDict(dict): pass
> >>>> eval('1+1', MyDict())
> > 2
> >>>> exec '1+1' in MyDict()
> >>>>
> > --------------------
> >
> > I only see a couple of calls to PyDict_CheckExact() in CPython's sources
> > and they usually seem to be related to special casing for performance
> > reasons. Nothing that should impact a module's globals.
> >
> > Besides, Cython controls its own language usages of eval and exec.
> >
>
> Cool!
> It seems that python internally uses PyObject_GetItem() for module
> level lookups and not PyDict_GetItem().
> Btw we use __Pyx_GetName() that calls PyObject_GetAttr() that isn't
> exactly the same for module lookups:
>
> # Works in Cython and doesn't work in Python
> print __class__
>
> So we can override __getitem__() and __setitem__():
> class MyDict(dict):
>    def __init__(self):
>        self._dict = {}
>
>    def __getitem__(self, key):
>        print '__getitem__', key
>        return self._dict[key]
>
>    def __setitem__(self, key, value):
>        print '__setitem__', key, value
>        self._dict[key] = value
>
>    def __getattr__(self, key):
>        print '__getattr__'
>
> d = MyDict()
> exec('x = 1; print x', d)
> eval('x', d)
> $ python foo.py
> __setitem__ x 1
> __getitem__ x
> 1
> __getitem__ x
>
>
> So we can make globals() return special dict with custom
> __setitem__()/__getitem__(). But it seems that we'll have to override
> many dict's standard methods like values(), update() and so on. That
> would be hard.
>
>
>
Be careful. That only works because your dict subclass is being used as the
locals as well. The LOAD_NAME opcode does a PyDict_CheckExact on the locals
and will call PyDict_GetItem if true, PyObject_GetItem if False:

case LOAD_NAME:
            w = GETITEM(names, oparg);
            if ((v = f->f_locals) == NULL) {
                PyErr_Format(PyExc_SystemError,
                             "no locals when loading %s",
                             PyObject_REPR(w));
                why = WHY_EXCEPTION;
                break;
            }
            if (PyDict_CheckExact(v)) {
                x = PyDict_GetItem(v, w);
                Py_XINCREF(x);
            }
            else {
                x = PyObject_GetItem(v, w);
                if (x == NULL && PyErr_Occurred()) {
                    if (!PyErr_ExceptionMatches(
                                    PyExc_KeyError))
                        break;
                    PyErr_Clear();
                }

}


You can see that the dict subclassing breaks down when you pass an empty
dict as the locals:

In [1]: class Foo(dict): ...: def __getitem__(self, name): ...: print
'get', name ...: return super(Foo, self).__getitem__(name) ...: In [2]: f =
Foo(a=42) In [3]: eval('a', f) get a Out[3]: 42 In [4]: eval('a', f, {})
Out[4]: 42
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120121/464d77fe/attachment.html>

From vitja.makarov at gmail.com  Sat Jan 21 19:43:28 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 21 Jan 2012 22:43:28 +0400
Subject: [Cython] Speedup module-level lookup
In-Reply-To: <CAPc_zRW9ihiMft9ex3gOhH-5tJsE1jfhhACTBmHfBCFNg49s5A@mail.gmail.com>
References: <CAKGHGPSvqCuUsmgMjWzqW1KNGEbjqJvsnPOYo5QtZbF9=8VxAQ@mail.gmail.com>
	<CADiQ+QDKySW_vQFiGXuGxFUXiW9te=arOfsQL9CjMgoxOnS5xw@mail.gmail.com>
	<CAKGHGPR4QYN=ucMdZAXFDUUuOzQf+bMth2PVxz12WGyx6GhWSQ@mail.gmail.com>
	<CAKGHGPTzhzmH=a0ouxXEo3GR4x90txa-rWYScXBwZUwYfpj_Pg@mail.gmail.com>
	<CADiQ+QB35SLRsHhva2KU9Q5Op2T9PN_17t5pkUWSVXRTc0e1bg@mail.gmail.com>
	<CAPc_zRUeaiy+QbQGFQLxivDeyBv609_+Kop9h05YzuLKLzBPSQ@mail.gmail.com>
	<4F1A53F6.2050906@behnel.de>
	<CAKGHGPR-B6z5P=nFRD2yqonuyxhyXwsGAfrxYKFLTAKGdCCboQ@mail.gmail.com>
	<CAPc_zRW9ihiMft9ex3gOhH-5tJsE1jfhhACTBmHfBCFNg49s5A@mail.gmail.com>
Message-ID: <CAKGHGPSRCu6=hbLsSJPSWhe+OhCkgQDKhWCLDHOy2xcexNd5gA@mail.gmail.com>

2012/1/21 Chris Colbert <sccolbert at gmail.com>:
>
>
> On Sat, Jan 21, 2012 at 2:35 AM, Vitja Makarov <vitja.makarov at gmail.com>
> wrote:
>>
>> 2012/1/21 Stefan Behnel <stefan_ml at behnel.de>:
>> > Chris Colbert, 19.01.2012 09:18:
>> >> If it doesn't pass PyDict_CheckExact you won't be able to use it as the
>> >> globals to eval or exec.
>> >
>> > What makes you say that? I tried and it worked for me, all the way back
>> > to
>> > Python 2.4:
>> >
>> > --------------------
>> > Python 2.4.6 (#2, Jan 21 2010, 23:45:25)
>> > [GCC 4.4.1] on linux2
>> > Type "help", "copyright", "credits" or "license" for more information.
>> >>>> class MyDict(dict): pass
>> >>>> eval('1+1', MyDict())
>> > 2
>> >>>> exec '1+1' in MyDict()
>> >>>>
>> > --------------------
>> >
>> > I only see a couple of calls to PyDict_CheckExact() in CPython's sources
>> > and they usually seem to be related to special casing for performance
>> > reasons. Nothing that should impact a module's globals.
>> >
>> > Besides, Cython controls its own language usages of eval and exec.
>> >
>>
>> Cool!
>> It seems that python internally uses PyObject_GetItem() for module
>> level lookups and not PyDict_GetItem().
>> Btw we use __Pyx_GetName() that calls PyObject_GetAttr() that isn't
>> exactly the same for module lookups:
>>
>> # Works in Cython and doesn't work in Python
>> print __class__
>>
>> So we can override __getitem__() and __setitem__():
>> class MyDict(dict):
>> ? ?def __init__(self):
>> ? ? ? ?self._dict = {}
>>
>> ? ?def __getitem__(self, key):
>> ? ? ? ?print '__getitem__', key
>> ? ? ? ?return self._dict[key]
>>
>> ? ?def __setitem__(self, key, value):
>> ? ? ? ?print '__setitem__', key, value
>> ? ? ? ?self._dict[key] = value
>>
>> ? ?def __getattr__(self, key):
>> ? ? ? ?print '__getattr__'
>>
>> d = MyDict()
>> exec('x = 1; print x', d)
>> eval('x', d)
>> $ python foo.py
>> __setitem__ x 1
>> __getitem__ x
>> 1
>> __getitem__ x
>>
>>
>> So we can make globals() return special dict with custom
>> __setitem__()/__getitem__(). But it seems that we'll have to override
>> many dict's standard methods like values(), update() and so on. That
>> would be hard.
>>
>>
>
> Be careful. That only works because your dict subclass is being used as the
> locals as well. The LOAD_NAME opcode does a PyDict_CheckExact on the locals
> and will call PyDict_GetItem if true, PyObject_GetItem if False:
>
> case LOAD_NAME:
>             w = GETITEM(names, oparg);
>             if ((v = f->f_locals) == NULL) {
>                 PyErr_Format(PyExc_SystemError,
>                              "no locals when loading %s",
>                              PyObject_REPR(w));
>                 why = WHY_EXCEPTION;
>                 break;
>             }
>             if (PyDict_CheckExact(v)) {
>                 x = PyDict_GetItem(v, w);
>                 Py_XINCREF(x);
>             }
>             else {
>                 x = PyObject_GetItem(v, w);
>                 if (x == NULL && PyErr_Occurred()) {
>                     if (!PyErr_ExceptionMatches(
>                                     PyExc_KeyError))
>                         break;
>                     PyErr_Clear();
>                 }
>
> }
>
>
> You can see that the dict subclassing breaks down when you pass an empty
> dict as the locals:
>
> In [1]: class Foo(dict): ...: def __getitem__(self, name): ...: print 'get',
> name ...: return super(Foo, self).__getitem__(name) ...: In [2]: f =
> Foo(a=42) In [3]: eval('a', f) get a Out[3]: 42 In [4]: eval('a', f, {})
> Out[4]: 42
>
>

Nice catch! It seems that globals MUST be a real dict.

>>> help(eval)
eval(...)
    eval(source[, globals[, locals]]) -> value

    Evaluate the source in the context of globals and locals.
    The source may be a string representing a Python expression
    or a code object as returned by compile().
    The globals must be a dictionary and locals can be any mapping,
    defaulting to the current globals and locals.
    If only globals is given, locals defaults to it.


-- 
vitja.

From stefan_ml at behnel.de  Sat Jan 21 19:50:42 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jan 2012 19:50:42 +0100
Subject: [Cython] AddTraceback() slows down generators
Message-ID: <4F1B0902.1050903@behnel.de>

Hi,

I did some callgrind profiling on Cython's generators and was surprised to
find that AddTraceback() represents a serious performance penalty for short
running generators.

I profiled a compiled Python implementation of itertools.groupby(), which
yields (key, group) tuples where the group is an iterator again. I ran this
code in Python for benchmarking:

"""
L = sorted(range(1000)*5)

all(list(g) for k,g in groupby(L))
"""

Groups tend to be rather short in real code, often just one or a couple of
items, so unpacking the group iterator into a list will usually be a quick
loop and then the generator raises StopIteration on termination and builds
a traceback for it. According to callgrind (which, I should note, tends to
overestimate the amount of time spent in memory allocation), the iteration
during the group unpacking takes about 30% of the overall runtime of the
all() loop, and the AddTraceback() call at the end of each group traversal
takes up to 25% (!) on my side. That means that more than 80% of the group
unpacking time goes into raising StopIteration from the generators. I
attached the call graph with the relative timings.

About half of the exception raising time is eaten by PyString_FromFormat()
that builds the function-name + line-position string (which, I may note, is
basically a convenience feature). This string is a constant for a
generator's StopIteration exception, at least for each final return point
in a generator, but here it is being recreated over and over again, for
each exception that gets raised.

Even if we keep creating a new frame instance each time (which should be ok
because CPython has a frame instance cache already and we'd only create one
during the generator lifetime), the whole code object could actually be
cached after the first creation, preferably bound to the lifetime of the
generator creator function/method. Or, more generally, one code object per
generator termination point, which will be a single point in the majority
of cases. For the specific code above, that should shave off almost 20% of
the overall runtime of the all() loop.

I think that's totally worth doing.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: callgraph2.png
Type: image/png
Size: 22440 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120121/1a2a600f/attachment-0001.png>

From d.s.seljebotn at astro.uio.no  Sat Jan 21 22:16:20 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jan 2012 22:16:20 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F1B0902.1050903@behnel.de>
References: <4F1B0902.1050903@behnel.de>
Message-ID: <4F1B2B24.8040302@astro.uio.no>

On 01/21/2012 07:50 PM, Stefan Behnel wrote:
> Hi,
>
> I did some callgrind profiling on Cython's generators and was surprised to
> find that AddTraceback() represents a serious performance penalty for short
> running generators.
>
> I profiled a compiled Python implementation of itertools.groupby(), which
> yields (key, group) tuples where the group is an iterator again. I ran this
> code in Python for benchmarking:
>
> """
> L = sorted(range(1000)*5)
>
> all(list(g) for k,g in groupby(L))
> """
>
> Groups tend to be rather short in real code, often just one or a couple of
> items, so unpacking the group iterator into a list will usually be a quick
> loop and then the generator raises StopIteration on termination and builds
> a traceback for it. According to callgrind (which, I should note, tends to
> overestimate the amount of time spent in memory allocation), the iteration
> during the group unpacking takes about 30% of the overall runtime of the
> all() loop, and the AddTraceback() call at the end of each group traversal
> takes up to 25% (!) on my side. That means that more than 80% of the group
> unpacking time goes into raising StopIteration from the generators. I
> attached the call graph with the relative timings.

OT: Since you complain that callgrind is inaccurate; are you aware of 
sampling profilers, such as Google perftools? (I don't have experience 
with callgrind myself)

http://google-perftools.googlecode.com/svn/trunk/doc/cpuprofile.html

http://pypi.python.org/pypi/yep

Dag

>
> About half of the exception raising time is eaten by PyString_FromFormat()
> that builds the function-name + line-position string (which, I may note, is
> basically a convenience feature). This string is a constant for a
> generator's StopIteration exception, at least for each final return point
> in a generator, but here it is being recreated over and over again, for
> each exception that gets raised.
>
> Even if we keep creating a new frame instance each time (which should be ok
> because CPython has a frame instance cache already and we'd only create one
> during the generator lifetime), the whole code object could actually be
> cached after the first creation, preferably bound to the lifetime of the
> generator creator function/method. Or, more generally, one code object per
> generator termination point, which will be a single point in the majority
> of cases. For the specific code above, that should shave off almost 20% of
> the overall runtime of the all() loop.
>
> I think that's totally worth doing.
>
> Stefan
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From robertwb at math.washington.edu  Sat Jan 21 23:09:40 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 21 Jan 2012 14:09:40 -0800
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F1B0902.1050903@behnel.de>
References: <4F1B0902.1050903@behnel.de>
Message-ID: <CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>

On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I did some callgrind profiling on Cython's generators and was surprised to
> find that AddTraceback() represents a serious performance penalty for short
> running generators.
>
> I profiled a compiled Python implementation of itertools.groupby(), which
> yields (key, group) tuples where the group is an iterator again. I ran this
> code in Python for benchmarking:
>
> """
> L = sorted(range(1000)*5)
>
> all(list(g) for k,g in groupby(L))
> """
>
> Groups tend to be rather short in real code, often just one or a couple of
> items, so unpacking the group iterator into a list will usually be a quick
> loop and then the generator raises StopIteration on termination and builds
> a traceback for it. According to callgrind (which, I should note, tends to
> overestimate the amount of time spent in memory allocation), the iteration
> during the group unpacking takes about 30% of the overall runtime of the
> all() loop, and the AddTraceback() call at the end of each group traversal
> takes up to 25% (!) on my side. That means that more than 80% of the group
> unpacking time goes into raising StopIteration from the generators. I
> attached the call graph with the relative timings.
>
> About half of the exception raising time is eaten by PyString_FromFormat()
> that builds the function-name + line-position string (which, I may note, is
> basically a convenience feature). This string is a constant for a
> generator's StopIteration exception, at least for each final return point
> in a generator, but here it is being recreated over and over again, for
> each exception that gets raised.
>
> Even if we keep creating a new frame instance each time (which should be ok
> because CPython has a frame instance cache already and we'd only create one
> during the generator lifetime), the whole code object could actually be
> cached after the first creation, preferably bound to the lifetime of the
> generator creator function/method. Or, more generally, one code object per
> generator termination point, which will be a single point in the majority
> of cases. For the specific code above, that should shave off almost 20% of
> the overall runtime of the all() loop.
>
> I think that's totally worth doing.

Makes sense to me. I did some caching like this for profiling.

- Robert

From markflorisson88 at gmail.com  Mon Jan 23 11:27:12 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 23 Jan 2012 10:27:12 +0000
Subject: [Cython] 0.16 release
Message-ID: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>

Hey,

It's been almost three months since we talked about a 0.16 release, I
think it's quite ready. It would already be a big release, it would be
good to see how people like it, and to catch any issues etc before we
pile on more features.

Mark

From konrad.hinsen at fastmail.net  Tue Jan 24 12:37:50 2012
From: konrad.hinsen at fastmail.net (Konrad Hinsen)
Date: Tue, 24 Jan 2012 12:37:50 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
Message-ID: <1327405070.15017.140661027320813@webmail.messagingengine.com>

Compiling the attached Cython file produced the attached C file which
has errors in lines 532-534:

  __pyx_v_self->xx = None;
  __pyx_v_self->yy = None;
  __pyx_v_self->zz = None;

There is no C symbol "None", so this doesn't compile.

I first noticed the bug in Cython 0.15, but it's still in the latest
revision from Github.

Konrad.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug.pyx
Type: application/octet-stream
Size: 147 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120124/dfc2c327/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug.c.gz
Type: application/x-gzip
Size: 10310 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120124/dfc2c327/attachment.bin>

From markflorisson88 at gmail.com  Tue Jan 24 14:53:20 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 24 Jan 2012 13:53:20 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <1327405070.15017.140661027320813@webmail.messagingengine.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
Message-ID: <CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>

On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
> Compiling the attached Cython file produced the attached C file which
> has errors in lines 532-534:
>
> ?__pyx_v_self->xx = None;
> ?__pyx_v_self->yy = None;
> ?__pyx_v_self->zz = None;
>
> There is no C symbol "None", so this doesn't compile.
>
> I first noticed the bug in Cython 0.15, but it's still in the latest
> revision from Github.
>
> Konrad.
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Hm, it seems the problem is that the call to the builtin float results
in SimpleCallNode being replaced with PythonCApiNode, which then
generates the result code, but the list of coerced nodes are
CloneNodes of the original rhs, and CloneNode does not generate the
result code of the original rhs (i.e. allocate and assign to a temp),
which results in a None result.

Maybe CascadedAssignmentNode should replace CloneNode.arg with the
latest self.rhs in generate_assignment_code? I'm not entirely sure.

From vitja.makarov at gmail.com  Tue Jan 24 15:09:04 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jan 2012 18:09:04 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
Message-ID: <CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>

2012/1/24 mark florisson <markflorisson88 at gmail.com>:
> On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
>> Compiling the attached Cython file produced the attached C file which
>> has errors in lines 532-534:
>>
>> ?__pyx_v_self->xx = None;
>> ?__pyx_v_self->yy = None;
>> ?__pyx_v_self->zz = None;
>>
>> There is no C symbol "None", so this doesn't compile.
>>
>> I first noticed the bug in Cython 0.15, but it's still in the latest
>> revision from Github.
>>
>> Konrad.
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> Hm, it seems the problem is that the call to the builtin float results
> in SimpleCallNode being replaced with PythonCApiNode, which then
> generates the result code, but the list of coerced nodes are
> CloneNodes of the original rhs, and CloneNode does not generate the
> result code of the original rhs (i.e. allocate and assign to a temp),
> which results in a None result.
>
> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
> latest self.rhs in generate_assignment_code? I'm not entirely sure.


May be it's better to run OptimizeBuiltinCalls before
AnalyseExpressionsTransform?

I have a patch that initializes NameNode's entry at ControlFlowAnalysis stage.


-- 
vitja.

From robertwb at math.washington.edu  Tue Jan 24 18:36:31 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jan 2012 09:36:31 -0800
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
Message-ID: <CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>

On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/24 mark florisson <markflorisson88 at gmail.com>:
>> On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
>>> Compiling the attached Cython file produced the attached C file which
>>> has errors in lines 532-534:
>>>
>>> ?__pyx_v_self->xx = None;
>>> ?__pyx_v_self->yy = None;
>>> ?__pyx_v_self->zz = None;
>>>
>>> There is no C symbol "None", so this doesn't compile.
>>>
>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>> revision from Github.
>>>
>>> Konrad.
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> Hm, it seems the problem is that the call to the builtin float results
>> in SimpleCallNode being replaced with PythonCApiNode, which then
>> generates the result code, but the list of coerced nodes are
>> CloneNodes of the original rhs, and CloneNode does not generate the
>> result code of the original rhs (i.e. allocate and assign to a temp),
>> which results in a None result.
>>
>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>
>
> May be it's better to run OptimizeBuiltinCalls before
> AnalyseExpressionsTransform?

Doesn't OptimizeBuiltinCalls take advantage of type information?

From vitja.makarov at gmail.com  Tue Jan 24 19:30:43 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jan 2012 22:30:43 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
Message-ID: <CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>

2012/1/24 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/24 mark florisson <markflorisson88 at gmail.com>:
>>> On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
>>>> Compiling the attached Cython file produced the attached C file which
>>>> has errors in lines 532-534:
>>>>
>>>> ?__pyx_v_self->xx = None;
>>>> ?__pyx_v_self->yy = None;
>>>> ?__pyx_v_self->zz = None;
>>>>
>>>> There is no C symbol "None", so this doesn't compile.
>>>>
>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>> revision from Github.
>>>>
>>>> Konrad.
>>>>
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>
>>> Hm, it seems the problem is that the call to the builtin float results
>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>> generates the result code, but the list of coerced nodes are
>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>> which results in a None result.
>>>
>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.

Seems like a hack to me.

>>
>>
>> May be it's better to run OptimizeBuiltinCalls before
>> AnalyseExpressionsTransform?
>
> Doesn't OptimizeBuiltinCalls take advantage of type information?

Yes, it does :(

So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
is created before rhs is updated.



-- 
vitja.

From markflorisson88 at gmail.com  Tue Jan 24 19:51:37 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 24 Jan 2012 18:51:37 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
Message-ID: <CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>

On 24 January 2012 18:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/24 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/24 mark florisson <markflorisson88 at gmail.com>:
>>>> On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
>>>>> Compiling the attached Cython file produced the attached C file which
>>>>> has errors in lines 532-534:
>>>>>
>>>>> ?__pyx_v_self->xx = None;
>>>>> ?__pyx_v_self->yy = None;
>>>>> ?__pyx_v_self->zz = None;
>>>>>
>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>
>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>> revision from Github.
>>>>>
>>>>> Konrad.
>>>>>
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>
>>>> Hm, it seems the problem is that the call to the builtin float results
>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>> generates the result code, but the list of coerced nodes are
>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>> which results in a None result.
>>>>
>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>
> Seems like a hack to me.
>
>>>
>>>
>>> May be it's better to run OptimizeBuiltinCalls before
>>> AnalyseExpressionsTransform?
>>
>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>
> Yes, it does :(
>
> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
> is created before rhs is updated.
>

I think deferring the CloneNode creation to code generation time works
(are there any known problem with doing type coercions at code
generation time?). E.g. save 'env' during analyse_types and in
generate_assignment_code do

    rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
    rhs.generate_evaluation_code(code)
    lhs.generate_assignment_code(rhs, code)

Seems to work.

>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Tue Jan 24 20:05:48 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jan 2012 23:05:48 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
Message-ID: <CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>

2012/1/24 mark florisson <markflorisson88 at gmail.com>:
> On 24 January 2012 18:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/24 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/1/24 mark florisson <markflorisson88 at gmail.com>:
>>>>> On 24 January 2012 11:37, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
>>>>>> Compiling the attached Cython file produced the attached C file which
>>>>>> has errors in lines 532-534:
>>>>>>
>>>>>> ?__pyx_v_self->xx = None;
>>>>>> ?__pyx_v_self->yy = None;
>>>>>> ?__pyx_v_self->zz = None;
>>>>>>
>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>
>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>> revision from Github.
>>>>>>
>>>>>> Konrad.
>>>>>>
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>
>>>>>
>>>>> Hm, it seems the problem is that the call to the builtin float results
>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>> generates the result code, but the list of coerced nodes are
>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>> which results in a None result.
>>>>>
>>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>>
>> Seems like a hack to me.
>>
>>>>
>>>>
>>>> May be it's better to run OptimizeBuiltinCalls before
>>>> AnalyseExpressionsTransform?
>>>
>>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>>
>> Yes, it does :(
>>
>> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
>> is created before rhs is updated.
>>
>
> I think deferring the CloneNode creation to code generation time works
> (are there any known problem with doing type coercions at code
> generation time?).

Coercion errors at code generation time?

> E.g. save 'env' during analyse_types and in
> generate_assignment_code do
>
> ? ?rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
> ? ?rhs.generate_evaluation_code(code)
> ? ?lhs.generate_assignment_code(rhs, code)
>
> Seems to work.
>

Yeah, that's better.


-- 
vitja.

From d.s.seljebotn at astro.uio.no  Tue Jan 24 20:18:06 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 24 Jan 2012 20:18:06 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
	<CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>
Message-ID: <4F1F03EE.5000504@astro.uio.no>

On 01/24/2012 08:05 PM, Vitja Makarov wrote:
> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>> On 24 January 2012 18:30, Vitja Makarov<vitja.makarov at gmail.com>  wrote:
>>> 2012/1/24 Robert Bradshaw<robertwb at math.washington.edu>:
>>>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov<vitja.makarov at gmail.com>  wrote:
>>>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>>> On 24 January 2012 11:37, Konrad Hinsen<konrad.hinsen at fastmail.net>  wrote:
>>>>>>> Compiling the attached Cython file produced the attached C file which
>>>>>>> has errors in lines 532-534:
>>>>>>>
>>>>>>>   __pyx_v_self->xx = None;
>>>>>>>   __pyx_v_self->yy = None;
>>>>>>>   __pyx_v_self->zz = None;
>>>>>>>
>>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>>
>>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>>> revision from Github.
>>>>>>>
>>>>>>> Konrad.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>
>>>>>>
>>>>>> Hm, it seems the problem is that the call to the builtin float results
>>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>>> generates the result code, but the list of coerced nodes are
>>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>>> which results in a None result.
>>>>>>
>>>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>>>
>>> Seems like a hack to me.
>>>
>>>>>
>>>>>
>>>>> May be it's better to run OptimizeBuiltinCalls before
>>>>> AnalyseExpressionsTransform?
>>>>
>>>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>>>
>>> Yes, it does :(
>>>
>>> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
>>> is created before rhs is updated.
>>>
>>
>> I think deferring the CloneNode creation to code generation time works
>> (are there any known problem with doing type coercions at code
>> generation time?).
>
> Coercion errors at code generation time?

Apologies up front for raising my voice, as my knowledge of the 
internals are getting so rusty...take this with a grain of salt.

I'm +1 on working towards having the code generation phase be pure code 
generation. I did some refactorings to take mini-steps towards that once 
upon a time, moving some error conditions to before code generation.

My preferred approach would be to do away with CascadedAssignmentNode at 
the parse tree stage:

a = b = c = expr

goes to

tmp = expr
c = tmp
b = tmp
a = tmp

and so on. Of course it gets messier;

(expr1)[expr2] = (expr3).attr = expr4

But apart from getting the time of evaluating each expression right the 
transform should be straightforward. One of the tempnodes/"let"-nodes (I 
forgot which one, or if they've been consolidated) should be able to fix 
this.

Takes some more work though than a quick hack though...

Dag

>
>> E.g. save 'env' during analyse_types and in
>> generate_assignment_code do
>>
>>     rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
>>     rhs.generate_evaluation_code(code)
>>     lhs.generate_assignment_code(rhs, code)
>>
>> Seems to work.
>>
>
> Yeah, that's better.
>
>


From markflorisson88 at gmail.com  Tue Jan 24 21:28:54 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 24 Jan 2012 20:28:54 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <4F1F03EE.5000504@astro.uio.no>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
	<CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>
	<4F1F03EE.5000504@astro.uio.no>
Message-ID: <CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>

On 24 January 2012 19:18, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 01/24/2012 08:05 PM, Vitja Makarov wrote:
>>
>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>
>>> On 24 January 2012 18:30, Vitja Makarov<vitja.makarov at gmail.com> ?wrote:
>>>>
>>>> 2012/1/24 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>
>>>>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov<vitja.makarov at gmail.com>
>>>>> ?wrote:
>>>>>>
>>>>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>>>>
>>>>>>> On 24 January 2012 11:37, Konrad Hinsen<konrad.hinsen at fastmail.net>
>>>>>>> ?wrote:
>>>>>>>>
>>>>>>>> Compiling the attached Cython file produced the attached C file
>>>>>>>> which
>>>>>>>> has errors in lines 532-534:
>>>>>>>>
>>>>>>>> ?__pyx_v_self->xx = None;
>>>>>>>> ?__pyx_v_self->yy = None;
>>>>>>>> ?__pyx_v_self->zz = None;
>>>>>>>>
>>>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>>>
>>>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>>>> revision from Github.
>>>>>>>>
>>>>>>>> Konrad.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> cython-devel mailing list
>>>>>>>> cython-devel at python.org
>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>
>>>>>>>
>>>>>>> Hm, it seems the problem is that the call to the builtin float
>>>>>>> results
>>>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>>>> generates the result code, but the list of coerced nodes are
>>>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>>>> which results in a None result.
>>>>>>>
>>>>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>>>>
>>>>
>>>> Seems like a hack to me.
>>>>
>>>>>>
>>>>>>
>>>>>> May be it's better to run OptimizeBuiltinCalls before
>>>>>> AnalyseExpressionsTransform?
>>>>>
>>>>>
>>>>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>>>>
>>>>
>>>> Yes, it does :(
>>>>
>>>> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
>>>> is created before rhs is updated.
>>>>
>>>
>>> I think deferring the CloneNode creation to code generation time works
>>> (are there any known problem with doing type coercions at code
>>> generation time?).
>>
>>
>> Coercion errors at code generation time?
>
>
> Apologies up front for raising my voice, as my knowledge of the internals
> are getting so rusty...take this with a grain of salt.
>
> I'm +1 on working towards having the code generation phase be pure code
> generation. I did some refactorings to take mini-steps towards that once
> upon a time, moving some error conditions to before code generation.
>
> My preferred approach would be to do away with CascadedAssignmentNode at the
> parse tree stage:
>
> a = b = c = expr
>
> goes to
>
> tmp = expr
> c = tmp
> b = tmp
> a = tmp
>
> and so on. Of course it gets messier;
>
> (expr1)[expr2] = (expr3).attr = expr4
>
> But apart from getting the time of evaluating each expression right the
> transform should be straightforward. One of the tempnodes/"let"-nodes (I
> forgot which one, or if they've been consolidated) should be able to fix
> this.
>
> Takes some more work though than a quick hack though...
>
> Dag
>

In principle it was doing the same thing, apart from the actual
rewrite. I suppose the replacement problem can also be circumvented by
manually wrapping self.rhs in a CoerceToTempNode. The problem with
coerce_to_temp is that it does not create this node if the result is
already in a temp. Creating it manually does mean an extra useless
assignment, but it is an easy fix which happens at analyse_types time.
 Instead we could also use another node that just proxies a few things
like generate_result_code and the result method.

I like the idea though, it would be nice to only handle things in
SingleAssignmentNode. I recently added broadcasting (inserting leading
dimensions) and scalar assignment to memoryviews, and you can only
catch that at the assignment point. Currently it only supports single
assignments as the functionality is only in SingleAssignmentNode.

I must say though, the following would look a bit weird:

    a = b[:] = c[:, :] = d

as you always expect a kind of "cascade", e.g. you expect c[:, :] to
be assignable to b[:], or 'a', but none of that may be true at all. So
I'm fine with disallowing that, I think people should only use
cascaded assignment for variables.

>>
>>> E.g. save 'env' during analyse_types and in
>>> generate_assignment_code do
>>>
>>> ? ?rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
>>> ? ?rhs.generate_evaluation_code(code)
>>> ? ?lhs.generate_assignment_code(rhs, code)
>>>
>>> Seems to work.
>>>
>>
>> Yeah, that's better.
>>
>>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Jan 24 21:58:13 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 24 Jan 2012 20:58:13 +0000
Subject: [Cython] inline defnode calls
Message-ID: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>

I just noticed the inline defnode call code. When I try to compile
with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
following code:

def foo(x): print foo
foo(10)

I get

Error compiling Cython file:
------------------------------------------------------------
...
def foo(x):
    print x

foo(10)
  ^
------------------------------------------------------------

test.pyx:4:3: Compiler crash in InlineDefNodeCalls

ModuleNode.body = StatListNode(test.pyx:1:0)
StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
    result = handler_method(child)
  File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
visit_SimpleCallNode
    if not function_name.cf_state.is_single:
AttributeError: 'NoneType' object has no attribute 'is_single'

From robertwb at math.washington.edu  Wed Jan 25 02:27:42 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jan 2012 17:27:42 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
Message-ID: <CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>

On Mon, Jan 23, 2012 at 2:27 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Hey,
>
> It's been almost three months since we talked about a 0.16 release, I
> think it's quite ready. It would already be a big release, it would be
> good to see how people like it, and to catch any issues etc before we
> pile on more features.

I would love to do a release soon. Last time this came up, I think the
big issue was (compilation) performance regression. Has this been
adequately addressed? The other issue is that there are a couple of
doctest failures with Sage. One source of problems is decorators due
to the (ugly) disallowing of function re-declarations, I'll try look
into this one. There are also a huge number of segfaults (see the
bottom of https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/lastSuccessfulBuild/artifact/log.txt
) which we need to get to the bottom of.

- Robert

From vitja.makarov at gmail.com  Wed Jan 25 07:49:52 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 25 Jan 2012 10:49:52 +0400
Subject: [Cython] inline defnode calls
In-Reply-To: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
Message-ID: <CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>

2012/1/25 mark florisson <markflorisson88 at gmail.com>:
> I just noticed the inline defnode call code. When I try to compile
> with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
> following code:
>
> def foo(x): print foo
> foo(10)
>
> I get
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> def foo(x):
> ? ?print x
>
> foo(10)
> ?^
> ------------------------------------------------------------
>
> test.pyx:4:3: Compiler crash in InlineDefNodeCalls
>
> ModuleNode.body = StatListNode(test.pyx:1:0)
> StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
> ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
> ? ?result_is_used = True,
> ? ?use_managed_ref = True)
>
> Compiler crash traceback from this point on:
> ?File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
> ? ?result = handler_method(child)
> ?File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
> visit_SimpleCallNode
> ? ?if not function_name.cf_state.is_single:
> AttributeError: 'NoneType' object has no attribute 'is_single'


Thanks for the report! The feature is still experimental and by
default is disabled.
Anyway it wouldn't work for your example. It works when we know what
exactly function is referred by the name so it's closure case:

def foo():
    def bar():
        pass
    bar()

-- 
vitja.

From vitja.makarov at gmail.com  Wed Jan 25 07:59:43 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 25 Jan 2012 10:59:43 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
	<CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>
	<4F1F03EE.5000504@astro.uio.no>
	<CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>
Message-ID: <CAKGHGPQcn3hqxZqjb+STTkNwrM9JBjBXg5wh6D6CjULKHXuaDg@mail.gmail.com>

2012/1/25 mark florisson <markflorisson88 at gmail.com>:
> On 24 January 2012 19:18, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 01/24/2012 08:05 PM, Vitja Makarov wrote:
>>>
>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>
>>>> On 24 January 2012 18:30, Vitja Makarov<vitja.makarov at gmail.com> ?wrote:
>>>>>
>>>>> 2012/1/24 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>
>>>>>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov<vitja.makarov at gmail.com>
>>>>>> ?wrote:
>>>>>>>
>>>>>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>>>>>
>>>>>>>> On 24 January 2012 11:37, Konrad Hinsen<konrad.hinsen at fastmail.net>
>>>>>>>> ?wrote:
>>>>>>>>>
>>>>>>>>> Compiling the attached Cython file produced the attached C file
>>>>>>>>> which
>>>>>>>>> has errors in lines 532-534:
>>>>>>>>>
>>>>>>>>> ?__pyx_v_self->xx = None;
>>>>>>>>> ?__pyx_v_self->yy = None;
>>>>>>>>> ?__pyx_v_self->zz = None;
>>>>>>>>>
>>>>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>>>>
>>>>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>>>>> revision from Github.
>>>>>>>>>
>>>>>>>>> Konrad.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> cython-devel mailing list
>>>>>>>>> cython-devel at python.org
>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hm, it seems the problem is that the call to the builtin float
>>>>>>>> results
>>>>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>>>>> generates the result code, but the list of coerced nodes are
>>>>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>>>>> which results in a None result.
>>>>>>>>
>>>>>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>>>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>>>>>
>>>>>
>>>>> Seems like a hack to me.
>>>>>
>>>>>>>
>>>>>>>
>>>>>>> May be it's better to run OptimizeBuiltinCalls before
>>>>>>> AnalyseExpressionsTransform?
>>>>>>
>>>>>>
>>>>>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>>>>>
>>>>>
>>>>> Yes, it does :(
>>>>>
>>>>> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
>>>>> is created before rhs is updated.
>>>>>
>>>>
>>>> I think deferring the CloneNode creation to code generation time works
>>>> (are there any known problem with doing type coercions at code
>>>> generation time?).
>>>
>>>
>>> Coercion errors at code generation time?
>>
>>
>> Apologies up front for raising my voice, as my knowledge of the internals
>> are getting so rusty...take this with a grain of salt.
>>
>> I'm +1 on working towards having the code generation phase be pure code
>> generation. I did some refactorings to take mini-steps towards that once
>> upon a time, moving some error conditions to before code generation.
>>
>> My preferred approach would be to do away with CascadedAssignmentNode at the
>> parse tree stage:
>>
>> a = b = c = expr
>>
>> goes to
>>
>> tmp = expr
>> c = tmp
>> b = tmp
>> a = tmp
>>
>> and so on. Of course it gets messier;
>>
>> (expr1)[expr2] = (expr3).attr = expr4
>>
>> But apart from getting the time of evaluating each expression right the
>> transform should be straightforward. One of the tempnodes/"let"-nodes (I
>> forgot which one, or if they've been consolidated) should be able to fix
>> this.
>>
>> Takes some more work though than a quick hack though...
>>
>> Dag
>>
>
> In principle it was doing the same thing, apart from the actual
> rewrite. I suppose the replacement problem can also be circumvented by
> manually wrapping self.rhs in a CoerceToTempNode. The problem with
> coerce_to_temp is that it does not create this node if the result is
> already in a temp. Creating it manually does mean an extra useless
> assignment, but it is an easy fix which happens at analyse_types time.
> ?Instead we could also use another node that just proxies a few things
> like generate_result_code and the result method.
>
> I like the idea though, it would be nice to only handle things in
> SingleAssignmentNode. I recently added broadcasting (inserting leading
> dimensions) and scalar assignment to memoryviews, and you can only
> catch that at the assignment point. Currently it only supports single
> assignments as the functionality is only in SingleAssignmentNode.
>
> I must say though, the following would look a bit weird:
>
> ? ?a = b[:] = c[:, :] = d
>
> as you always expect a kind of "cascade", e.g. you expect c[:, :] to
> be assignable to b[:], or 'a', but none of that may be true at all. So
> I'm fine with disallowing that, I think people should only use
> cascaded assignment for variables.
>
>>>
>>>> E.g. save 'env' during analyse_types and in
>>>> generate_assignment_code do
>>>>
>>>> ? ?rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
>>>> ? ?rhs.generate_evaluation_code(code)
>>>> ? ?lhs.generate_assignment_code(rhs, code)
>>>>
>>>> Seems to work.
>>>>
>>>
>>> Yeah, that's better.
>>>
>>>
>>

I don't like idea of transforming cascade assignment into N single
assignment since we might break some optimizations and loose CF info.

I thought about playing with properties. We can make CloneNode.arg a
property, e.g.:

CloneNode(arg_getter=lambda:self.rhs)


-- 
vitja.

From stefan_ml at behnel.de  Wed Jan 25 08:41:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jan 2012 08:41:14 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
Message-ID: <4F1FB21A.9080407@behnel.de>

mark florisson, 24.01.2012 14:53:
> On 24 January 2012 11:37, Konrad Hinsen wrote:
>> Compiling the attached Cython file produced the attached C file which
>> has errors in lines 532-534:
>>
>>  __pyx_v_self->xx = None;
>>  __pyx_v_self->yy = None;
>>  __pyx_v_self->zz = None;
>>
>> There is no C symbol "None", so this doesn't compile.
>>
>> I first noticed the bug in Cython 0.15, but it's still in the latest
>> revision from Github.
> 
> Hm, it seems the problem is that the call to the builtin float results
> in SimpleCallNode being replaced with PythonCApiNode, which then
> generates the result code, but the list of coerced nodes are
> CloneNodes of the original rhs, and CloneNode does not generate the
> result code of the original rhs (i.e. allocate and assign to a temp),
> which results in a None result.

Back to the old idea of separating the type analysis into 1) a basic
typing, inference and entry creation step and 2) a proper type analysis,
coercion, etc. step.

The type driven optimisations would then run in between the two. That would
simplify the optimisations (which would no longer have to unpack wrapped
nodes) and improve the type analysis because it could work with the
optimised types, e.g. return types of optimised builtin functions.

I'm not entirely sure where the type inference should run. It may make more
sense to move it after the tree optimisations to make use of optimised
function calls.

While we're at it, we should also replace the current type inference
mechanism with a control flow based one.

Sounds like a good topic for a Cython hacking workshop.

Stefan

From d.s.seljebotn at astro.uio.no  Wed Jan 25 09:00:40 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 25 Jan 2012 09:00:40 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPQcn3hqxZqjb+STTkNwrM9JBjBXg5wh6D6CjULKHXuaDg@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>	<CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>	<4F1F03EE.5000504@astro.uio.no>	<CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>
	<CAKGHGPQcn3hqxZqjb+STTkNwrM9JBjBXg5wh6D6CjULKHXuaDg@mail.gmail.com>
Message-ID: <4F1FB6A8.8080103@astro.uio.no>

On 01/25/2012 07:59 AM, Vitja Makarov wrote:
> 2012/1/25 mark florisson<markflorisson88 at gmail.com>:
>> On 24 January 2012 19:18, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no>  wrote:
>>> On 01/24/2012 08:05 PM, Vitja Makarov wrote:
>>>>
>>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>>
>>>>> On 24 January 2012 18:30, Vitja Makarov<vitja.makarov at gmail.com>    wrote:
>>>>>>
>>>>>> 2012/1/24 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>>
>>>>>>> On Tue, Jan 24, 2012 at 6:09 AM, Vitja Makarov<vitja.makarov at gmail.com>
>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> 2012/1/24 mark florisson<markflorisson88 at gmail.com>:
>>>>>>>>>
>>>>>>>>> On 24 January 2012 11:37, Konrad Hinsen<konrad.hinsen at fastmail.net>
>>>>>>>>>   wrote:
>>>>>>>>>>
>>>>>>>>>> Compiling the attached Cython file produced the attached C file
>>>>>>>>>> which
>>>>>>>>>> has errors in lines 532-534:
>>>>>>>>>>
>>>>>>>>>>   __pyx_v_self->xx = None;
>>>>>>>>>>   __pyx_v_self->yy = None;
>>>>>>>>>>   __pyx_v_self->zz = None;
>>>>>>>>>>
>>>>>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>>>>>
>>>>>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>>>>>> revision from Github.
>>>>>>>>>>
>>>>>>>>>> Konrad.
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> cython-devel mailing list
>>>>>>>>>> cython-devel at python.org
>>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hm, it seems the problem is that the call to the builtin float
>>>>>>>>> results
>>>>>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>>>>>> generates the result code, but the list of coerced nodes are
>>>>>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>>>>>> which results in a None result.
>>>>>>>>>
>>>>>>>>> Maybe CascadedAssignmentNode should replace CloneNode.arg with the
>>>>>>>>> latest self.rhs in generate_assignment_code? I'm not entirely sure.
>>>>>>
>>>>>>
>>>>>> Seems like a hack to me.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> May be it's better to run OptimizeBuiltinCalls before
>>>>>>>> AnalyseExpressionsTransform?
>>>>>>>
>>>>>>>
>>>>>>> Doesn't OptimizeBuiltinCalls take advantage of type information?
>>>>>>
>>>>>>
>>>>>> Yes, it does :(
>>>>>>
>>>>>> So as Mark said the problem is CascadedAssignmentNode.coerced_rhs_list
>>>>>> is created before rhs is updated.
>>>>>>
>>>>>
>>>>> I think deferring the CloneNode creation to code generation time works
>>>>> (are there any known problem with doing type coercions at code
>>>>> generation time?).
>>>>
>>>>
>>>> Coercion errors at code generation time?
>>>
>>>
>>> Apologies up front for raising my voice, as my knowledge of the internals
>>> are getting so rusty...take this with a grain of salt.
>>>
>>> I'm +1 on working towards having the code generation phase be pure code
>>> generation. I did some refactorings to take mini-steps towards that once
>>> upon a time, moving some error conditions to before code generation.
>>>
>>> My preferred approach would be to do away with CascadedAssignmentNode at the
>>> parse tree stage:
>>>
>>> a = b = c = expr
>>>
>>> goes to
>>>
>>> tmp = expr
>>> c = tmp
>>> b = tmp
>>> a = tmp
>>>
>>> and so on. Of course it gets messier;
>>>
>>> (expr1)[expr2] = (expr3).attr = expr4
>>>
>>> But apart from getting the time of evaluating each expression right the
>>> transform should be straightforward. One of the tempnodes/"let"-nodes (I
>>> forgot which one, or if they've been consolidated) should be able to fix
>>> this.
>>>
>>> Takes some more work though than a quick hack though...
>>>
>>> Dag
>>>
>>
>> In principle it was doing the same thing, apart from the actual
>> rewrite. I suppose the replacement problem can also be circumvented by
>> manually wrapping self.rhs in a CoerceToTempNode. The problem with
>> coerce_to_temp is that it does not create this node if the result is
>> already in a temp. Creating it manually does mean an extra useless
>> assignment, but it is an easy fix which happens at analyse_types time.
>>   Instead we could also use another node that just proxies a few things
>> like generate_result_code and the result method.
>>
>> I like the idea though, it would be nice to only handle things in
>> SingleAssignmentNode. I recently added broadcasting (inserting leading
>> dimensions) and scalar assignment to memoryviews, and you can only
>> catch that at the assignment point. Currently it only supports single
>> assignments as the functionality is only in SingleAssignmentNode.
>>
>> I must say though, the following would look a bit weird:
>>
>>     a = b[:] = c[:, :] = d
>>
>> as you always expect a kind of "cascade", e.g. you expect c[:, :] to
>> be assignable to b[:], or 'a', but none of that may be true at all. So
>> I'm fine with disallowing that, I think people should only use
>> cascaded assignment for variables.

I don't think that is a problem myself; but that's perhaps just because 
I'm so used to it (and to "a = b.x = y" not invoking b.__getattr__, and 
so on). After all, that is what you get with Python and NumPy! This is, 
in a sense Python being a bit strange and us just following Python.

So I'm +1 for supporting this if we can do it "by accident".

>>
>>>>
>>>>> E.g. save 'env' during analyse_types and in
>>>>> generate_assignment_code do
>>>>>
>>>>>     rhs = CloneNode(self.rhs).coerce_to(lhs.type, self.env)
>>>>>     rhs.generate_evaluation_code(code)
>>>>>     lhs.generate_assignment_code(rhs, code)
>>>>>
>>>>> Seems to work.
>>>>>
>>>>
>>>> Yeah, that's better.
>>>>
>>>>
>>>
>
> I don't like idea of transforming cascade assignment into N single
> assignment since we might break some optimizations and loose CF info.

But what if the user decides to write

tmp = expr
a = tmp
b = tmp

manually? Shouldn't the same optimizations apply then?

Consider that if we can get down to a single assignment node, we can 
then split it into SingleAssignmentNode into "AssignLocalNode", 
"AssignPythonModuleVarNode", "AssignAttributeNode", "AssignTypedVarNode" 
and so on, if we want to -- that should clean up some code...


Dag Sverre

From stefan_ml at behnel.de  Wed Jan 25 09:04:02 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jan 2012 09:04:02 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<CAKGHGPQwSoNefFEBBYDOEXOKLnW7L3rEFtQptqknQ+SbUaNsjw@mail.gmail.com>
	<CADiQ+QDw+rLo85ko2qs-OUtBQ_K3DWbK_FT46saP4j7RTfMLww@mail.gmail.com>
	<CAKGHGPQZqqMadOfimARyKi2Djr0FMBosVGu-e4_j38LTrfxtxg@mail.gmail.com>
	<CANg26EUQdkpOzuy=CKEq2=iN750KPYD=G6ApkCGcYQmKBhfcrA@mail.gmail.com>
	<CAKGHGPQS4+52M9ih__phZ3cSOuO6JGnYsCT9n8ZXKCbUAg309A@mail.gmail.com>
	<4F1F03EE.5000504@astro.uio.no>
	<CANg26EUWHpMK1dW-FuEArATC-9mf2u9AxDVttNAjE90YmNQ=JQ@mail.gmail.com>
Message-ID: <4F1FB772.1030906@behnel.de>

mark florisson, 24.01.2012 21:28:
> I must say though, the following would look a bit weird:
> 
>     a = b[:] = c[:, :] = d
> 
> as you always expect a kind of "cascade", e.g. you expect c[:, :] to
> be assignable to b[:], or 'a', but none of that may be true at all.

That's normal for a typed language that has type auto-coercion. I consider
this a major feature. It certainly makes the internals tricky, but when
working on the assignment code, I always tried to keep the coercions and
the eventual assignment code independent, even in the face of efficient
tuple unpacking, because I considered it the expected behaviour.

Stefan

From markflorisson88 at gmail.com  Wed Jan 25 11:43:35 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jan 2012 10:43:35 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
Message-ID: <CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>

On 25 January 2012 01:27, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Mon, Jan 23, 2012 at 2:27 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> Hey,
>>
>> It's been almost three months since we talked about a 0.16 release, I
>> think it's quite ready. It would already be a big release, it would be
>> good to see how people like it, and to catch any issues etc before we
>> pile on more features.
>
> I would love to do a release soon. Last time this came up, I think the
> big issue was (compilation) performance regression. Has this been
> adequately addressed?

Sort of. Basically if you don't use memoryviews it will be as fast as
it used to be, otherwise there is about a 3 second constant time
overhead (on my machine).

> The other issue is that there are a couple of
> doctest failures with Sage. One source of problems is decorators due
> to the (ugly) disallowing of function re-declarations, I'll try look
> into this one. There are also a huge number of segfaults (see the
> bottom of https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/lastSuccessfulBuild/artifact/log.txt
> ) which we need to get to the bottom of.

Oh I see. I suppose to try it out under a debugger one would have to
compile the whole of sage from source?

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Wed Jan 25 11:44:40 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jan 2012 10:44:40 +0000
Subject: [Cython] inline defnode calls
In-Reply-To: <CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
	<CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
Message-ID: <CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>

On 25 January 2012 06:49, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>> I just noticed the inline defnode call code. When I try to compile
>> with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
>> following code:
>>
>> def foo(x): print foo
>> foo(10)
>>
>> I get
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> def foo(x):
>> ? ?print x
>>
>> foo(10)
>> ?^
>> ------------------------------------------------------------
>>
>> test.pyx:4:3: Compiler crash in InlineDefNodeCalls
>>
>> ModuleNode.body = StatListNode(test.pyx:1:0)
>> StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
>> ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
>> ? ?result_is_used = True,
>> ? ?use_managed_ref = True)
>>
>> Compiler crash traceback from this point on:
>> ?File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
>> ? ?result = handler_method(child)
>> ?File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
>> visit_SimpleCallNode
>> ? ?if not function_name.cf_state.is_single:
>> AttributeError: 'NoneType' object has no attribute 'is_single'
>
>
> Thanks for the report! The feature is still experimental and by
> default is disabled.
> Anyway it wouldn't work for your example. It works when we know what
> exactly function is referred by the name so it's closure case:
>
> def foo():
> ? ?def bar():
> ? ? ? ?pass
> ? ?bar()
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Ah, neat. I thought it was perhaps also defying monkeypatching.

From vitja.makarov at gmail.com  Wed Jan 25 12:24:08 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 25 Jan 2012 15:24:08 +0400
Subject: [Cython] inline defnode calls
In-Reply-To: <CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
	<CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
	<CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>
Message-ID: <CAKGHGPToVU+zZnmBexfd4p5TJUX4P+vt4MpNSen3D1xq-B3jzQ@mail.gmail.com>

2012/1/25 mark florisson <markflorisson88 at gmail.com>:
> On 25 January 2012 06:49, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>>> I just noticed the inline defnode call code. When I try to compile
>>> with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
>>> following code:
>>>
>>> def foo(x): print foo
>>> foo(10)
>>>
>>> I get
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>> def foo(x):
>>> ? ?print x
>>>
>>> foo(10)
>>> ?^
>>> ------------------------------------------------------------
>>>
>>> test.pyx:4:3: Compiler crash in InlineDefNodeCalls
>>>
>>> ModuleNode.body = StatListNode(test.pyx:1:0)
>>> StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
>>> ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
>>> ? ?result_is_used = True,
>>> ? ?use_managed_ref = True)
>>>
>>> Compiler crash traceback from this point on:
>>> ?File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
>>> ? ?result = handler_method(child)
>>> ?File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
>>> visit_SimpleCallNode
>>> ? ?if not function_name.cf_state.is_single:
>>> AttributeError: 'NoneType' object has no attribute 'is_single'
>>
>>
>> Thanks for the report! The feature is still experimental and by
>> default is disabled.
>> Anyway it wouldn't work for your example. It works when we know what
>> exactly function is referred by the name so it's closure case:
>>
>> def foo():
>> ? ?def bar():
>> ? ? ? ?pass
>> ? ?bar()
>>
>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Ah, neat. I thought it was perhaps also defying monkeypatching.


I'm thinking about implementing  "conditional inlining": depending on
what function actually is it'll make direct call to C function or
PyObject_Call().


-- 
vitja.

From markflorisson88 at gmail.com  Wed Jan 25 12:32:34 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jan 2012 11:32:34 +0000
Subject: [Cython] inline defnode calls
In-Reply-To: <CAKGHGPToVU+zZnmBexfd4p5TJUX4P+vt4MpNSen3D1xq-B3jzQ@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
	<CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
	<CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>
	<CAKGHGPToVU+zZnmBexfd4p5TJUX4P+vt4MpNSen3D1xq-B3jzQ@mail.gmail.com>
Message-ID: <CANg26EW2J1yWXJgmPOinCHjGCz2-7k1h2Q4tgFE6-tWWfF=TLQ@mail.gmail.com>

On 25 January 2012 11:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>> On 25 January 2012 06:49, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>>>> I just noticed the inline defnode call code. When I try to compile
>>>> with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
>>>> following code:
>>>>
>>>> def foo(x): print foo
>>>> foo(10)
>>>>
>>>> I get
>>>>
>>>> Error compiling Cython file:
>>>> ------------------------------------------------------------
>>>> ...
>>>> def foo(x):
>>>> ? ?print x
>>>>
>>>> foo(10)
>>>> ?^
>>>> ------------------------------------------------------------
>>>>
>>>> test.pyx:4:3: Compiler crash in InlineDefNodeCalls
>>>>
>>>> ModuleNode.body = StatListNode(test.pyx:1:0)
>>>> StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
>>>> ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
>>>> ? ?result_is_used = True,
>>>> ? ?use_managed_ref = True)
>>>>
>>>> Compiler crash traceback from this point on:
>>>> ?File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
>>>> ? ?result = handler_method(child)
>>>> ?File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
>>>> visit_SimpleCallNode
>>>> ? ?if not function_name.cf_state.is_single:
>>>> AttributeError: 'NoneType' object has no attribute 'is_single'
>>>
>>>
>>> Thanks for the report! The feature is still experimental and by
>>> default is disabled.
>>> Anyway it wouldn't work for your example. It works when we know what
>>> exactly function is referred by the name so it's closure case:
>>>
>>> def foo():
>>> ? ?def bar():
>>> ? ? ? ?pass
>>> ? ?bar()
>>>
>>> --
>>> vitja.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Ah, neat. I thought it was perhaps also defying monkeypatching.
>
>
> I'm thinking about implementing ?"conditional inlining": depending on
> what function actually is it'll make direct call to C function or
> PyObject_Call().
>

Sounds like a good idea. Any idea how much faster that can be?

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Wed Jan 25 12:36:03 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jan 2012 11:36:03 +0000
Subject: [Cython] inline defnode calls
In-Reply-To: <CANg26EW2J1yWXJgmPOinCHjGCz2-7k1h2Q4tgFE6-tWWfF=TLQ@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
	<CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
	<CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>
	<CAKGHGPToVU+zZnmBexfd4p5TJUX4P+vt4MpNSen3D1xq-B3jzQ@mail.gmail.com>
	<CANg26EW2J1yWXJgmPOinCHjGCz2-7k1h2Q4tgFE6-tWWfF=TLQ@mail.gmail.com>
Message-ID: <CANg26EU2sesoYomMhUFiLWKVp0Rffjw5e_Bq=7NULBOwgCf=8g@mail.gmail.com>

On 25 January 2012 11:32, mark florisson <markflorisson88 at gmail.com> wrote:
> On 25 January 2012 11:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>>> On 25 January 2012 06:49, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/1/25 mark florisson <markflorisson88 at gmail.com>:
>>>>> I just noticed the inline defnode call code. When I try to compile
>>>>> with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the
>>>>> following code:
>>>>>
>>>>> def foo(x): print foo
>>>>> foo(10)
>>>>>
>>>>> I get
>>>>>
>>>>> Error compiling Cython file:
>>>>> ------------------------------------------------------------
>>>>> ...
>>>>> def foo(x):
>>>>> ? ?print x
>>>>>
>>>>> foo(10)
>>>>> ?^
>>>>> ------------------------------------------------------------
>>>>>
>>>>> test.pyx:4:3: Compiler crash in InlineDefNodeCalls
>>>>>
>>>>> ModuleNode.body = StatListNode(test.pyx:1:0)
>>>>> StatListNode.stats[2] = ExprStatNode(test.pyx:4:3)
>>>>> ExprStatNode.expr = SimpleCallNode(test.pyx:4:3,
>>>>> ? ?result_is_used = True,
>>>>> ? ?use_managed_ref = True)
>>>>>
>>>>> Compiler crash traceback from this point on:
>>>>> ?File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild
>>>>> ? ?result = handler_method(child)
>>>>> ?File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in
>>>>> visit_SimpleCallNode
>>>>> ? ?if not function_name.cf_state.is_single:
>>>>> AttributeError: 'NoneType' object has no attribute 'is_single'
>>>>
>>>>
>>>> Thanks for the report! The feature is still experimental and by
>>>> default is disabled.
>>>> Anyway it wouldn't work for your example. It works when we know what
>>>> exactly function is referred by the name so it's closure case:
>>>>
>>>> def foo():
>>>> ? ?def bar():
>>>> ? ? ? ?pass
>>>> ? ?bar()
>>>>
>>>> --
>>>> vitja.
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Ah, neat. I thought it was perhaps also defying monkeypatching.
>>
>>
>> I'm thinking about implementing ?"conditional inlining": depending on
>> what function actually is it'll make direct call to C function or
>> PyObject_Call().
>>
>
> Sounds like a good idea. Any idea how much faster that can be?

Hm, probably about an order of magnitude for a noop function (simple
test of cdef vs def call).

>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Wed Jan 25 13:00:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jan 2012 13:00:46 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
Message-ID: <4F1FEEEE.2060605@behnel.de>

mark florisson, 25.01.2012 11:43:
> On 25 January 2012 01:27, Robert Bradshaw wrote:
>> On Mon, Jan 23, 2012 at 2:27 AM, mark florisson wrote:
>>> It's been almost three months since we talked about a 0.16 release, I
>>> think it's quite ready. It would already be a big release, it would be
>>> good to see how people like it, and to catch any issues etc before we
>>> pile on more features.
>>
>> I would love to do a release soon. Last time this came up, I think the
>> big issue was (compilation) performance regression. Has this been
>> adequately addressed?
> 
> Sort of. Basically if you don't use memoryviews it will be as fast as
> it used to be, otherwise there is about a 3 second constant time
> overhead (on my machine).
> 
>> The other issue is that there are a couple of
>> doctest failures with Sage. One source of problems is decorators due
>> to the (ugly) disallowing of function re-declarations, I'll try look
>> into this one. There are also a huge number of segfaults (see the
>> bottom of https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/lastSuccessfulBuild/artifact/log.txt
>> ) which we need to get to the bottom of.
> 
> Oh I see. I suppose to try it out under a debugger one would have to
> compile the whole of sage from source?

It might be easier to log into sage.math, go to the sage build directory
that Jenkins uses and do some experiments there. It's in

/levi/scratch/robertwb/hudson/sage-4.8/

Stefan

From stefan_ml at behnel.de  Wed Jan 25 13:10:02 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jan 2012 13:10:02 +0100
Subject: [Cython] inline defnode calls
In-Reply-To: <CANg26EU2sesoYomMhUFiLWKVp0Rffjw5e_Bq=7NULBOwgCf=8g@mail.gmail.com>
References: <CANg26EUChLS3SZTJMecTwAuHoRB_UKmGqq_KtmE3g=NbbvhkWw@mail.gmail.com>
	<CAKGHGPRpj2EP-cFzQN4_Ha1q_Qs=0QnJF5zvGuyBoogmu2Nxnw@mail.gmail.com>
	<CANg26EVPrV2AVv1tBSRQm6niZ7mxaoffng_HeYKK7D0roJwXtw@mail.gmail.com>
	<CAKGHGPToVU+zZnmBexfd4p5TJUX4P+vt4MpNSen3D1xq-B3jzQ@mail.gmail.com>
	<CANg26EW2J1yWXJgmPOinCHjGCz2-7k1h2Q4tgFE6-tWWfF=TLQ@mail.gmail.com>
	<CANg26EU2sesoYomMhUFiLWKVp0Rffjw5e_Bq=7NULBOwgCf=8g@mail.gmail.com>
Message-ID: <4F1FF11A.8090309@behnel.de>

mark florisson, 25.01.2012 12:36:
> On 25 January 2012 11:32, mark florisson wrote:
>> On 25 January 2012 11:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> I'm thinking about implementing  "conditional inlining": depending on
>>> what function actually is it'll make direct call to C function or
>>> PyObject_Call().
>>
>> Sounds like a good idea. Any idea how much faster that can be?
> 
> Hm, probably about an order of magnitude for a noop function (simple
> test of cdef vs def call).

Easily, yes. It avoids all the argument type conversion and
packing/unpacking. That's a huge overhead compared to a straight
conditional C function call which the C compiler could even inline, or the
CPU could at least keep in its branch prediction cache and optimise its
pipeline for.

Stefan

From markflorisson88 at gmail.com  Wed Jan 25 13:17:18 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jan 2012 12:17:18 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <4F1FEEEE.2060605@behnel.de>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
Message-ID: <CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>

On 25 January 2012 12:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 25.01.2012 11:43:
>> On 25 January 2012 01:27, Robert Bradshaw wrote:
>>> On Mon, Jan 23, 2012 at 2:27 AM, mark florisson wrote:
>>>> It's been almost three months since we talked about a 0.16 release, I
>>>> think it's quite ready. It would already be a big release, it would be
>>>> good to see how people like it, and to catch any issues etc before we
>>>> pile on more features.
>>>
>>> I would love to do a release soon. Last time this came up, I think the
>>> big issue was (compilation) performance regression. Has this been
>>> adequately addressed?
>>
>> Sort of. Basically if you don't use memoryviews it will be as fast as
>> it used to be, otherwise there is about a 3 second constant time
>> overhead (on my machine).
>>
>>> The other issue is that there are a couple of
>>> doctest failures with Sage. One source of problems is decorators due
>>> to the (ugly) disallowing of function re-declarations, I'll try look
>>> into this one. There are also a huge number of segfaults (see the
>>> bottom of https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/lastSuccessfulBuild/artifact/log.txt
>>> ) which we need to get to the bottom of.
>>
>> Oh I see. I suppose to try it out under a debugger one would have to
>> compile the whole of sage from source?
>
> It might be easier to log into sage.math, go to the sage build directory
> that Jenkins uses and do some experiments there. It's in
>
> /levi/scratch/robertwb/hudson/sage-4.8/
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Ah, neat, thanks.

From robertwb at math.washington.edu  Wed Jan 25 18:39:17 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 25 Jan 2012 09:39:17 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
Message-ID: <CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>

On Wed, Jan 25, 2012 at 4:17 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 25 January 2012 12:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 25.01.2012 11:43:
>>> On 25 January 2012 01:27, Robert Bradshaw wrote:
>>>> On Mon, Jan 23, 2012 at 2:27 AM, mark florisson wrote:
>>>>> It's been almost three months since we talked about a 0.16 release, I
>>>>> think it's quite ready. It would already be a big release, it would be
>>>>> good to see how people like it, and to catch any issues etc before we
>>>>> pile on more features.
>>>>
>>>> I would love to do a release soon. Last time this came up, I think the
>>>> big issue was (compilation) performance regression. Has this been
>>>> adequately addressed?
>>>
>>> Sort of. Basically if you don't use memoryviews it will be as fast as
>>> it used to be, otherwise there is about a 3 second constant time
>>> overhead (on my machine).
>>>
>>>> The other issue is that there are a couple of
>>>> doctest failures with Sage. One source of problems is decorators due
>>>> to the (ugly) disallowing of function re-declarations, I'll try look
>>>> into this one. There are also a huge number of segfaults (see the
>>>> bottom of https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-tests/lastSuccessfulBuild/artifact/log.txt
>>>> ) which we need to get to the bottom of.
>>>
>>> Oh I see. I suppose to try it out under a debugger one would have to
>>> compile the whole of sage from source?
>>
>> It might be easier to log into sage.math, go to the sage build directory
>> that Jenkins uses and do some experiments there. It's in
>>
>> /levi/scratch/robertwb/hudson/sage-4.8/

And compiling Sage from scratch isn't actually that hard: type "make"
and wait a couple of hours. I've updated the description at
https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/
to explain how to build a cython-devel sage locally as others have
asked for this as well, in summary you apply the patch at
http://sage.math.washington.edu/home/robertwb/hudson-sage/sage-4.8/devel/sage-main/.hg/patches/0.16
to the repo in $SAGE_ROOT/devel/sage-main/ , install
https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
by downloading it and running "sage -i cython-devel.spkg" and then do
"sage -ba" to re-build all Cython files.

sage -gdb and sage -t -gdb /path/to/file are useful to know as well.

- Robert

From jason-sage at creativetrax.com  Wed Jan 25 23:35:28 2012
From: jason-sage at creativetrax.com (Jason Grout)
Date: Wed, 25 Jan 2012 16:35:28 -0600
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
Message-ID: <4F2083B0.9020209@creativetrax.com>

On 1/25/12 11:39 AM, Robert Bradshaw wrote:
> install
> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
> by downloading it and running "sage -i cython-devel.spkg"


In fact, you could just do

sage -i 
https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg

and Sage will (at least, should) download it for you, so that's even one 
less step!

Jason


From vitja.makarov at gmail.com  Thu Jan 26 07:39:28 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 26 Jan 2012 10:39:28 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <4F1FB21A.9080407@behnel.de>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
Message-ID: <CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>

2012/1/25 Stefan Behnel <stefan_ml at behnel.de>:
> mark florisson, 24.01.2012 14:53:
>> On 24 January 2012 11:37, Konrad Hinsen wrote:
>>> Compiling the attached Cython file produced the attached C file which
>>> has errors in lines 532-534:
>>>
>>> ?__pyx_v_self->xx = None;
>>> ?__pyx_v_self->yy = None;
>>> ?__pyx_v_self->zz = None;
>>>
>>> There is no C symbol "None", so this doesn't compile.
>>>
>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>> revision from Github.
>>
>> Hm, it seems the problem is that the call to the builtin float results
>> in SimpleCallNode being replaced with PythonCApiNode, which then
>> generates the result code, but the list of coerced nodes are
>> CloneNodes of the original rhs, and CloneNode does not generate the
>> result code of the original rhs (i.e. allocate and assign to a temp),
>> which results in a None result.
>
> Back to the old idea of separating the type analysis into 1) a basic
> typing, inference and entry creation step and 2) a proper type analysis,
> coercion, etc. step.
>

Yeah! I think the issue must be fixed before release. We can start
moving slowly in this direction and split
CascadedAssignmentNode.analyse_types into parts:
 - normal analyse_types()/expressions()
 - create clone nodes at some late stage

> The type driven optimisations would then run in between the two. That would
> simplify the optimisations (which would no longer have to unpack wrapped
> nodes) and improve the type analysis because it could work with the
> optimised types, e.g. return types of optimised builtin functions.
>
> I'm not entirely sure where the type inference should run. It may make more
> sense to move it after the tree optimisations to make use of optimised
> function calls.
>
> While we're at it, we should also replace the current type inference
> mechanism with a control flow based one.
>
> Sounds like a good topic for a Cython hacking workshop.
>

Nice. Any news on that?


-- 
vitja.

From markflorisson88 at gmail.com  Thu Jan 26 16:20:46 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 15:20:46 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
Message-ID: <CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>

On 26 January 2012 06:39, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/25 Stefan Behnel <stefan_ml at behnel.de>:
>> mark florisson, 24.01.2012 14:53:
>>> On 24 January 2012 11:37, Konrad Hinsen wrote:
>>>> Compiling the attached Cython file produced the attached C file which
>>>> has errors in lines 532-534:
>>>>
>>>> ?__pyx_v_self->xx = None;
>>>> ?__pyx_v_self->yy = None;
>>>> ?__pyx_v_self->zz = None;
>>>>
>>>> There is no C symbol "None", so this doesn't compile.
>>>>
>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>> revision from Github.
>>>
>>> Hm, it seems the problem is that the call to the builtin float results
>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>> generates the result code, but the list of coerced nodes are
>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>> which results in a None result.
>>
>> Back to the old idea of separating the type analysis into 1) a basic
>> typing, inference and entry creation step and 2) a proper type analysis,
>> coercion, etc. step.
>>
>
> Yeah! I think the issue must be fixed before release. We can start
> moving slowly in this direction and split
> CascadedAssignmentNode.analyse_types into parts:
> ?- normal analyse_types()/expressions()
> ?- create clone nodes at some late stage

At what stage would the stage 2) proper type analysis take place?
Basically nodes may be replaced at any point, and I'm not sure you
want to wait until just before code generation to do the coercions
(e.g.  GILCheck won't catch coercions to object, although assignment
nodes seem to check manually).

I think this problem can trivially be solved by creating a ProxyNode
that should never be replaced by any transform, but it's argument may
be replaced. So you wrap self.rhs in a ProxyNode and use that to
create your CloneNodes.

>> The type driven optimisations would then run in between the two. That would
>> simplify the optimisations (which would no longer have to unpack wrapped
>> nodes) and improve the type analysis because it could work with the
>> optimised types, e.g. return types of optimised builtin functions.
>>
>> I'm not entirely sure where the type inference should run. It may make more
>> sense to move it after the tree optimisations to make use of optimised
>> function calls.
>>
>> While we're at it, we should also replace the current type inference
>> mechanism with a control flow based one.
>>
>> Sounds like a good topic for a Cython hacking workshop.
>>
>
> Nice. Any news on that?
>
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From wesmckinn at gmail.com  Thu Jan 26 18:56:31 2012
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 26 Jan 2012 12:56:31 -0500
Subject: [Cython] Slow traceback reconstruction in IPython between 0.15.1
	and master
Message-ID: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>

Just wanted to bring this issue to your guys' attention in case you
knew what was responsible for this:

https://github.com/ipython/ipython/issues/1317#issuecomment-3652550

I traced down the problem (with git bisect) to a seemingly innocuous
commit referenced in that GitHub thread. The issue seemed to only
present itself in IPython, so likely there was some problem with
inspecting the Cython frames for giving context around the full
traceback.

best,
Wes

From markflorisson88 at gmail.com  Thu Jan 26 19:36:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 18:36:22 +0000
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
Message-ID: <CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>

On 26 January 2012 17:56, Wes McKinney <wesmckinn at gmail.com> wrote:
> Just wanted to bring this issue to your guys' attention in case you
> knew what was responsible for this:
>
> https://github.com/ipython/ipython/issues/1317#issuecomment-3652550
>
> I traced down the problem (with git bisect) to a seemingly innocuous
> commit referenced in that GitHub thread. The issue seemed to only
> present itself in IPython, so likely there was some problem with
> inspecting the Cython frames for giving context around the full
> traceback.
>
> best,
> Wes
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

So commit 0e579823bd34de5d1d9b4aeac2c8d727415cba2d fixed the problem?
I don't see how it could do that. Anyway, when I try to run the code
from the example I don't get any traceback at all. Could you perhaps
paste the exact code that produces the behaviour?

From markflorisson88 at gmail.com  Thu Jan 26 19:37:56 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 18:37:56 +0000
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
Message-ID: <CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>

On 26 January 2012 18:36, mark florisson <markflorisson88 at gmail.com> wrote:
> On 26 January 2012 17:56, Wes McKinney <wesmckinn at gmail.com> wrote:
>> Just wanted to bring this issue to your guys' attention in case you
>> knew what was responsible for this:
>>
>> https://github.com/ipython/ipython/issues/1317#issuecomment-3652550
>>
>> I traced down the problem (with git bisect) to a seemingly innocuous
>> commit referenced in that GitHub thread. The issue seemed to only
>> present itself in IPython, so likely there was some problem with
>> inspecting the Cython frames for giving context around the full
>> traceback.
>>
>> best,
>> Wes
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> So commit 0e579823bd34de5d1d9b4aeac2c8d727415cba2d fixed the problem?
> I don't see how it could do that. Anyway, when I try to run the code
> from the example I don't get any traceback at all. Could you perhaps
> paste the exact code that produces the behaviour?

On a side note, ipython is not something I usually trust to test
things out, as it gets things wrong sometimes which can seriously make
you question your own sanity.

From vitja.makarov at gmail.com  Thu Jan 26 19:51:03 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 26 Jan 2012 22:51:03 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
Message-ID: <CAKGHGPRGhJB6Fw6SsWPBy9B=YdfzOK2e5JnD+5cTLrR_9g7OJA@mail.gmail.com>

2012/1/26 mark florisson <markflorisson88 at gmail.com>:
> On 26 January 2012 06:39, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/25 Stefan Behnel <stefan_ml at behnel.de>:
>>> mark florisson, 24.01.2012 14:53:
>>>> On 24 January 2012 11:37, Konrad Hinsen wrote:
>>>>> Compiling the attached Cython file produced the attached C file which
>>>>> has errors in lines 532-534:
>>>>>
>>>>> ?__pyx_v_self->xx = None;
>>>>> ?__pyx_v_self->yy = None;
>>>>> ?__pyx_v_self->zz = None;
>>>>>
>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>
>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>> revision from Github.
>>>>
>>>> Hm, it seems the problem is that the call to the builtin float results
>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>> generates the result code, but the list of coerced nodes are
>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>> which results in a None result.
>>>
>>> Back to the old idea of separating the type analysis into 1) a basic
>>> typing, inference and entry creation step and 2) a proper type analysis,
>>> coercion, etc. step.
>>>
>>
>> Yeah! I think the issue must be fixed before release. We can start
>> moving slowly in this direction and split
>> CascadedAssignmentNode.analyse_types into parts:
>> ?- normal analyse_types()/expressions()
>> ?- create clone nodes at some late stage
>
> At what stage would the stage 2) proper type analysis take place?
> Basically nodes may be replaced at any point, and I'm not sure you
> want to wait until just before code generation to do the coercions
> (e.g. ?GILCheck won't catch coercions to object, although assignment
> nodes seem to check manually).
>

That must be run before GilCheck. Stage 2 is "I believe the tree won't
change much later"


> I think this problem can trivially be solved by creating a ProxyNode
> that should never be replaced by any transform, but it's argument may
> be replaced. So you wrap self.rhs in a ProxyNode and use that to
> create your CloneNodes.
>

Do you mean proxy node to be something like this:

class ProxyNode(object):
    def __init__(self, obj):
        object.__setattr__(self, '_obj', obj)

    def __getattr__(self, key):
        return getattr(self._obj, key)

    def __setattr__(self, key, value):
        setattr(self._obj, key, value)

That might help but I'm not sure how evil that is.

It also will require TreeVisitor.find_handler() modification. Node
replacement could be avoided by introducing ProxyProperty()

I see another problem with proxies: CloneNode or its owner may depend
on content of the argument so when it's changed things can be messed
up.

-- 
vitja.

From stefan_ml at behnel.de  Thu Jan 26 19:53:06 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Jan 2012 19:53:06 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
Message-ID: <4F21A112.30803@behnel.de>

mark florisson, 26.01.2012 16:20:
> On 26 January 2012 06:39, Vitja Makarov wrote:
>> 2012/1/25 Stefan Behnel:
>>> Back to the old idea of separating the type analysis into 1) a basic
>>> typing, inference and entry creation step and 2) a proper type analysis,
>>> coercion, etc. step.
>>>
>>
>> Yeah! I think the issue must be fixed before release. We can start
>> moving slowly in this direction and split
>> CascadedAssignmentNode.analyse_types into parts:
>>  - normal analyse_types()/expressions()
>>  - create clone nodes at some late stage
> 
> At what stage would the stage 2) proper type analysis take place?

After the structural optimisation phase and before any optimisations or
other transforms that require complete type information but do not change
types anymore. I don't see it being moved to the end of the pipeline, the
results will be needed way before that. Even some optimisations may not be
possible without complete type analysis.


> Basically nodes may be replaced at any point, and I'm not sure you
> want to wait until just before code generation to do the coercions
> (e.g.  GILCheck won't catch coercions to object, although assignment
> nodes seem to check manually).

There's a large grey area in between. It'll need some refactoring and
rebalancing, just as before. But it should be easier than it is now because
the grey area will have more anchors in it.


> I think this problem can trivially be solved by creating a ProxyNode
> that should never be replaced by any transform, but it's argument may
> be replaced. So you wrap self.rhs in a ProxyNode and use that to
> create your CloneNodes.

I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.

Stefan

From fperez.net at gmail.com  Thu Jan 26 20:10:08 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 26 Jan 2012 11:10:08 -0800
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
	<CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
Message-ID: <CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>

On Thu, Jan 26, 2012 at 10:37 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On a side note, ipython is not something I usually trust to test
> things out, as it gets things wrong sometimes which can seriously make
> you question your own sanity.

I should note that we'd love to know specifics about problems that
severe.  Without a concrete report it's impossible for us to fix a
problem, unfortunately, and a vague statement like this only serves to
spread the notion "ipython is bad, for reasons unspecified".

Also, since I know some core Cython devs are also heavy sage users,
and the sage command-line is actually ipython, having an issue at the
intersection of cython/ipython is problematic for all sage users
working at the terminal.  So we'd really like to make the
cython/ipython combo as robust as possible.

I certainly don't expect everyone to use ipython, and obviously when I
see something that looks really bizarre, I double-check with plain
Python first, as ultimately that is our reference and regressions from
plain python are automatically bugs for us.

But I have to say that it's been a very long time since I have
encountered a situation where ipython produces something completely
absurd, and if it happens to you, by all means please let us know, so
we can try to fix it.

Best,

f

From markflorisson88 at gmail.com  Thu Jan 26 20:12:52 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 19:12:52 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPRGhJB6Fw6SsWPBy9B=YdfzOK2e5JnD+5cTLrR_9g7OJA@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<CAKGHGPRGhJB6Fw6SsWPBy9B=YdfzOK2e5JnD+5cTLrR_9g7OJA@mail.gmail.com>
Message-ID: <CANg26EUesYLejDBefjMEVwd_=MX29FwOzYocSdhHrXbN1qqNsw@mail.gmail.com>

On 26 January 2012 18:51, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/26 mark florisson <markflorisson88 at gmail.com>:
>> On 26 January 2012 06:39, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/25 Stefan Behnel <stefan_ml at behnel.de>:
>>>> mark florisson, 24.01.2012 14:53:
>>>>> On 24 January 2012 11:37, Konrad Hinsen wrote:
>>>>>> Compiling the attached Cython file produced the attached C file which
>>>>>> has errors in lines 532-534:
>>>>>>
>>>>>> ?__pyx_v_self->xx = None;
>>>>>> ?__pyx_v_self->yy = None;
>>>>>> ?__pyx_v_self->zz = None;
>>>>>>
>>>>>> There is no C symbol "None", so this doesn't compile.
>>>>>>
>>>>>> I first noticed the bug in Cython 0.15, but it's still in the latest
>>>>>> revision from Github.
>>>>>
>>>>> Hm, it seems the problem is that the call to the builtin float results
>>>>> in SimpleCallNode being replaced with PythonCApiNode, which then
>>>>> generates the result code, but the list of coerced nodes are
>>>>> CloneNodes of the original rhs, and CloneNode does not generate the
>>>>> result code of the original rhs (i.e. allocate and assign to a temp),
>>>>> which results in a None result.
>>>>
>>>> Back to the old idea of separating the type analysis into 1) a basic
>>>> typing, inference and entry creation step and 2) a proper type analysis,
>>>> coercion, etc. step.
>>>>
>>>
>>> Yeah! I think the issue must be fixed before release. We can start
>>> moving slowly in this direction and split
>>> CascadedAssignmentNode.analyse_types into parts:
>>> ?- normal analyse_types()/expressions()
>>> ?- create clone nodes at some late stage
>>
>> At what stage would the stage 2) proper type analysis take place?
>> Basically nodes may be replaced at any point, and I'm not sure you
>> want to wait until just before code generation to do the coercions
>> (e.g. ?GILCheck won't catch coercions to object, although assignment
>> nodes seem to check manually).
>>
>
> That must be run before GilCheck. Stage 2 is "I believe the tree won't
> change much later"
>
>
>> I think this problem can trivially be solved by creating a ProxyNode
>> that should never be replaced by any transform, but it's argument may
>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>> create your CloneNodes.
>>
>
> Do you mean proxy node to be something like this:
>
> class ProxyNode(object):
> ? ?def __init__(self, obj):
> ? ? ? ?object.__setattr__(self, '_obj', obj)
>
> ? ?def __getattr__(self, key):
> ? ? ? ?return getattr(self._obj, key)
>
> ? ?def __setattr__(self, key, value):
> ? ? ? ?setattr(self._obj, key, value)
>
> That might help but I'm not sure how evil that is.
>
> It also will require TreeVisitor.find_handler() modification. Node
> replacement could be avoided by introducing ProxyProperty()
>
> I see another problem with proxies: CloneNode or its owner may depend
> on content of the argument so when it's changed things can be messed
> up.

Not quite like that. It should be a regular node that is ignored by
transforms but delegates a few methods. e.g.

class ProxyNode(object):
    child_attrs = ['arg']

    def __init__(self, arg):
        self.arg = arg

    def result(self):
        return self.arg.result()

    def generate_result_code(self, code):
        return self.arg.generate_result_code(code)

and that's pretty much it (untested). And CloneNode doesn't depend on
any specific node, nor should any code wrapping its nodes in
ProxyNodes (obviously).

The nodes sole purpose is to create an indirection, so that when
previously self.rhs got replaced, now self.rhs.arg will be replaced,
so self.rhs can be safely shared with other CloneNodes.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Thu Jan 26 20:15:44 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 19:15:44 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <4F21A112.30803@behnel.de>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
Message-ID: <CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>

On 26 January 2012 18:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 26.01.2012 16:20:
>> On 26 January 2012 06:39, Vitja Makarov wrote:
>>> 2012/1/25 Stefan Behnel:
>>>> Back to the old idea of separating the type analysis into 1) a basic
>>>> typing, inference and entry creation step and 2) a proper type analysis,
>>>> coercion, etc. step.
>>>>
>>>
>>> Yeah! I think the issue must be fixed before release. We can start
>>> moving slowly in this direction and split
>>> CascadedAssignmentNode.analyse_types into parts:
>>> ?- normal analyse_types()/expressions()
>>> ?- create clone nodes at some late stage
>>
>> At what stage would the stage 2) proper type analysis take place?
>
> After the structural optimisation phase and before any optimisations or
> other transforms that require complete type information but do not change
> types anymore. I don't see it being moved to the end of the pipeline, the
> results will be needed way before that. Even some optimisations may not be
> possible without complete type analysis.
>
>
>> Basically nodes may be replaced at any point, and I'm not sure you
>> want to wait until just before code generation to do the coercions
>> (e.g. ?GILCheck won't catch coercions to object, although assignment
>> nodes seem to check manually).
>
> There's a large grey area in between. It'll need some refactoring and
> rebalancing, just as before. But it should be easier than it is now because
> the grey area will have more anchors in it.
>

That's nice. It wouldn't solve the problem at hand though, and I think
nothing can unless it will be at the very end of the pipeline.

>> I think this problem can trivially be solved by creating a ProxyNode
>> that should never be replaced by any transform, but it's argument may
>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>> create your CloneNodes.
>
> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.

It wouldn't be a replacement, merely an addition (an extra indirection).

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Thu Jan 26 20:21:03 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 19:21:03 +0000
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
	<CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
	<CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>
Message-ID: <CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>

On 26 January 2012 19:10, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Jan 26, 2012 at 10:37 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On a side note, ipython is not something I usually trust to test
>> things out, as it gets things wrong sometimes which can seriously make
>> you question your own sanity.
>
> I should note that we'd love to know specifics about problems that
> severe. ?Without a concrete report it's impossible for us to fix a
> problem, unfortunately, and a vague statement like this only serves to
> spread the notion "ipython is bad, for reasons unspecified".

Apologies, it was indeed a rather vague comment. I had some issues
with pasting unicode characters a few years back that would get
incorrect codepoints in ipython but not regular python, but I'm afraid
I don't have any concrete report. As such I withdraw my comment, I
just wanted to mention that if something smells iffy it might be a
good idea to resort to regular python.

> Also, since I know some core Cython devs are also heavy sage users,
> and the sage command-line is actually ipython, having an issue at the
> intersection of cython/ipython is problematic for all sage users
> working at the terminal. ?So we'd really like to make the
> cython/ipython combo as robust as possible.
>
> I certainly don't expect everyone to use ipython, and obviously when I
> see something that looks really bizarre, I double-check with plain
> Python first, as ultimately that is our reference and regressions from
> plain python are automatically bugs for us.
>
> But I have to say that it's been a very long time since I have
> encountered a situation where ipython produces something completely
> absurd, and if it happens to you, by all means please let us know, so
> we can try to fix it.
>
> Best,
>
> f
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Thu Jan 26 20:27:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Jan 2012 20:27:43 +0100
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
Message-ID: <4F21A92F.6050401@behnel.de>

mark florisson, 26.01.2012 20:15:
> On 26 January 2012 18:53, Stefan Behnel wrote:
>> mark florisson, 26.01.2012 16:20:
>>> I think this problem can trivially be solved by creating a ProxyNode
>>> that should never be replaced by any transform, but it's argument may
>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>> create your CloneNodes.
>>
>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
> 
> It wouldn't be a replacement, merely an addition (an extra indirection).

What I was trying to say was that a ProxyNode would always be required by a
CloneNode, but I don't see where a ProxyNode would be needed outside of a
CloneNode. So it seems rather redundant and I don't know if we need a
separate node for it.

Stefan

From markflorisson88 at gmail.com  Thu Jan 26 20:41:03 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 19:41:03 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <4F21A92F.6050401@behnel.de>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
	<4F21A92F.6050401@behnel.de>
Message-ID: <CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>

On 26 January 2012 19:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 26.01.2012 20:15:
>> On 26 January 2012 18:53, Stefan Behnel wrote:
>>> mark florisson, 26.01.2012 16:20:
>>>> I think this problem can trivially be solved by creating a ProxyNode
>>>> that should never be replaced by any transform, but it's argument may
>>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>>> create your CloneNodes.
>>>
>>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
>>
>> It wouldn't be a replacement, merely an addition (an extra indirection).
>
> What I was trying to say was that a ProxyNode would always be required by a
> CloneNode, but I don't see where a ProxyNode would be needed outside of a
> CloneNode. So it seems rather redundant and I don't know if we need a
> separate node for it.

Yes it would be needed only for that, but I think the only real
alternative is to not use CloneNode at all, i.e. make the
transformation Dag mentioned, where you create new rhs (NameNode?)
references to the temporary result.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From wesmckinn at gmail.com  Thu Jan 26 20:40:41 2012
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 26 Jan 2012 14:40:41 -0500
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
	<CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
	<CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>
	<CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>
Message-ID: <CAJPUwMA0=BMEL=Vq2_vzhvmAvBfNCC7neDToPyn6fd=r9eVL6w@mail.gmail.com>

On Thu, Jan 26, 2012 at 2:21 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 26 January 2012 19:10, Fernando Perez <fperez.net at gmail.com> wrote:
>> On Thu, Jan 26, 2012 at 10:37 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On a side note, ipython is not something I usually trust to test
>>> things out, as it gets things wrong sometimes which can seriously make
>>> you question your own sanity.
>>
>> I should note that we'd love to know specifics about problems that
>> severe. ?Without a concrete report it's impossible for us to fix a
>> problem, unfortunately, and a vague statement like this only serves to
>> spread the notion "ipython is bad, for reasons unspecified".
>
> Apologies, it was indeed a rather vague comment. I had some issues
> with pasting unicode characters a few years back that would get
> incorrect codepoints in ipython but not regular python, but I'm afraid
> I don't have any concrete report. As such I withdraw my comment, I
> just wanted to mention that if something smells iffy it might be a
> good idea to resort to regular python.
>
>> Also, since I know some core Cython devs are also heavy sage users,
>> and the sage command-line is actually ipython, having an issue at the
>> intersection of cython/ipython is problematic for all sage users
>> working at the terminal. ?So we'd really like to make the
>> cython/ipython combo as robust as possible.
>>
>> I certainly don't expect everyone to use ipython, and obviously when I
>> see something that looks really bizarre, I double-check with plain
>> Python first, as ultimately that is our reference and regressions from
>> plain python are automatically bugs for us.
>>
>> But I have to say that it's been a very long time since I have
>> encountered a situation where ipython produces something completely
>> absurd, and if it happens to you, by all means please let us know, so
>> we can try to fix it.
>>
>> Best,
>>
>> f
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

To reproduce the problem you'll want to use the specific pandas SHA
(99e2eec) that I referenced there. The pandas bug in the issue has
been fixed since then.

From stefan_ml at behnel.de  Thu Jan 26 21:02:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Jan 2012 21:02:43 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
Message-ID: <4F21B163.2080006@behnel.de>

Robert Bradshaw, 21.01.2012 23:09:
> On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel wrote:
>> I did some callgrind profiling on Cython's generators and was surprised to
>> find that AddTraceback() represents a serious performance penalty for short
>> running generators.
>>
>> I profiled a compiled Python implementation of itertools.groupby(), which
>> yields (key, group) tuples where the group is an iterator again. I ran this
>> code in Python for benchmarking:
>>
>> """
>> L = sorted(range(1000)*5)
>>
>> all(list(g) for k,g in groupby(L))
>> """
>>
>> Groups tend to be rather short in real code, often just one or a couple of
>> items, so unpacking the group iterator into a list will usually be a quick
>> loop and then the generator raises StopIteration on termination and builds
>> a traceback for it. According to callgrind (which, I should note, tends to
>> overestimate the amount of time spent in memory allocation), the iteration
>> during the group unpacking takes about 30% of the overall runtime of the
>> all() loop, and the AddTraceback() call at the end of each group traversal
>> takes up to 25% (!) on my side. That means that more than 80% of the group
>> unpacking time goes into raising StopIteration from the generators. I
>> attached the call graph with the relative timings.
>>
>> About half of the exception raising time is eaten by PyString_FromFormat()
>> that builds the function-name + line-position string (which, I may note, is
>> basically a convenience feature). This string is a constant for a
>> generator's StopIteration exception, at least for each final return point
>> in a generator, but here it is being recreated over and over again, for
>> each exception that gets raised.
>>
>> Even if we keep creating a new frame instance each time (which should be ok
>> because CPython has a frame instance cache already and we'd only create one
>> during the generator lifetime), the whole code object could actually be
>> cached after the first creation, preferably bound to the lifetime of the
>> generator creator function/method. Or, more generally, one code object per
>> generator termination point, which will be a single point in the majority
>> of cases. For the specific code above, that should shave off almost 20% of
>> the overall runtime of the all() loop.
>>
>> I think that's totally worth doing.
> 
> Makes sense to me. I did some caching like this for profiling.

Here's a ticket for now:

http://trac.cython.org/cython_trac/ticket/760

Stefan

From markflorisson88 at gmail.com  Thu Jan 26 21:05:26 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 26 Jan 2012 20:05:26 +0000
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CAJPUwMA0=BMEL=Vq2_vzhvmAvBfNCC7neDToPyn6fd=r9eVL6w@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
	<CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
	<CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>
	<CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>
	<CAJPUwMA0=BMEL=Vq2_vzhvmAvBfNCC7neDToPyn6fd=r9eVL6w@mail.gmail.com>
Message-ID: <CANg26EWYyh05vGCprV_8w5ZqJt=zwxFHMRsU7P=kLz=Kkryiag@mail.gmail.com>

On 26 January 2012 19:40, Wes McKinney <wesmckinn at gmail.com> wrote:
> On Thu, Jan 26, 2012 at 2:21 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 26 January 2012 19:10, Fernando Perez <fperez.net at gmail.com> wrote:
>>> On Thu, Jan 26, 2012 at 10:37 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On a side note, ipython is not something I usually trust to test
>>>> things out, as it gets things wrong sometimes which can seriously make
>>>> you question your own sanity.
>>>
>>> I should note that we'd love to know specifics about problems that
>>> severe. ?Without a concrete report it's impossible for us to fix a
>>> problem, unfortunately, and a vague statement like this only serves to
>>> spread the notion "ipython is bad, for reasons unspecified".
>>
>> Apologies, it was indeed a rather vague comment. I had some issues
>> with pasting unicode characters a few years back that would get
>> incorrect codepoints in ipython but not regular python, but I'm afraid
>> I don't have any concrete report. As such I withdraw my comment, I
>> just wanted to mention that if something smells iffy it might be a
>> good idea to resort to regular python.
>>
>>> Also, since I know some core Cython devs are also heavy sage users,
>>> and the sage command-line is actually ipython, having an issue at the
>>> intersection of cython/ipython is problematic for all sage users
>>> working at the terminal. ?So we'd really like to make the
>>> cython/ipython combo as robust as possible.
>>>
>>> I certainly don't expect everyone to use ipython, and obviously when I
>>> see something that looks really bizarre, I double-check with plain
>>> Python first, as ultimately that is our reference and regressions from
>>> plain python are automatically bugs for us.
>>>
>>> But I have to say that it's been a very long time since I have
>>> encountered a situation where ipython produces something completely
>>> absurd, and if it happens to you, by all means please let us know, so
>>> we can try to fix it.
>>>
>>> Best,
>>>
>>> f
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> To reproduce the problem you'll want to use the specific pandas SHA
> (99e2eec) that I referenced there. The pandas bug in the issue has
> been fixed since then.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I get the same as minrk got: TypeError: unhashable type 'dict' .
Anyway, if it's working now (and if other tracebacks are fast (enough)
as they should be), I don't think we have anything to worry about.

From vitja.makarov at gmail.com  Thu Jan 26 21:19:27 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 27 Jan 2012 00:19:27 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F21B163.2080006@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
Message-ID: <CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>

2012/1/27 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 21.01.2012 23:09:
>> On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel wrote:
>>> I did some callgrind profiling on Cython's generators and was surprised to
>>> find that AddTraceback() represents a serious performance penalty for short
>>> running generators.
>>>
>>> I profiled a compiled Python implementation of itertools.groupby(), which
>>> yields (key, group) tuples where the group is an iterator again. I ran this
>>> code in Python for benchmarking:
>>>
>>> """
>>> L = sorted(range(1000)*5)
>>>
>>> all(list(g) for k,g in groupby(L))
>>> """
>>>
>>> Groups tend to be rather short in real code, often just one or a couple of
>>> items, so unpacking the group iterator into a list will usually be a quick
>>> loop and then the generator raises StopIteration on termination and builds
>>> a traceback for it. According to callgrind (which, I should note, tends to
>>> overestimate the amount of time spent in memory allocation), the iteration
>>> during the group unpacking takes about 30% of the overall runtime of the
>>> all() loop, and the AddTraceback() call at the end of each group traversal
>>> takes up to 25% (!) on my side. That means that more than 80% of the group
>>> unpacking time goes into raising StopIteration from the generators. I
>>> attached the call graph with the relative timings.
>>>
>>> About half of the exception raising time is eaten by PyString_FromFormat()
>>> that builds the function-name + line-position string (which, I may note, is
>>> basically a convenience feature). This string is a constant for a
>>> generator's StopIteration exception, at least for each final return point
>>> in a generator, but here it is being recreated over and over again, for
>>> each exception that gets raised.
>>>
>>> Even if we keep creating a new frame instance each time (which should be ok
>>> because CPython has a frame instance cache already and we'd only create one
>>> during the generator lifetime), the whole code object could actually be
>>> cached after the first creation, preferably bound to the lifetime of the
>>> generator creator function/method. Or, more generally, one code object per
>>> generator termination point, which will be a single point in the majority
>>> of cases. For the specific code above, that should shave off almost 20% of
>>> the overall runtime of the all() loop.
>>>
>>> I think that's totally worth doing.
>>
>> Makes sense to me. I did some caching like this for profiling.
>
> Here's a ticket for now:
>
> http://trac.cython.org/cython_trac/ticket/760
>

I think that could be easily fixed. CPython doesn't add any traceback
info for generator's ending.

https://github.com/vitek/cython/commit/63620bc2a29f3064bbdf7a49eefffaae4e3c369d

-- 
vitja.

From fperez.net at gmail.com  Thu Jan 26 21:42:06 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 26 Jan 2012 12:42:06 -0800
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<CANg26EUwCK69ow2_UX278eGChR1VjYRttYZEwOi_VPi_Zb8wsA@mail.gmail.com>
	<CANg26EW84NuYrf=hTO74FwEbR1YLoUgxJb7FrxOn6UhSXNRGfg@mail.gmail.com>
	<CAHAreOrJHZpizLcLhRtSnZkXuWC=_DXSxicLPbUW4S9F+bmc3Q@mail.gmail.com>
	<CANg26EWNCrBMNv0-8L_r4CuvENmGrqC84P7iL5ev-Ku3n1xJRg@mail.gmail.com>
Message-ID: <CAHAreOqcXbxPgM+Q-LW2S_CHTCBvXUMAeV-8uJjpW_h+mD5C9g@mail.gmail.com>

On Thu, Jan 26, 2012 at 11:21 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Apologies, it was indeed a rather vague comment. I had some issues

No worries.

> with pasting unicode characters a few years back that would get
> incorrect codepoints in ipython but not regular python, but I'm afraid

We had a long and distinguished history of unicode bugs (we even had a
specific tag for them, as they were so severe).  But the vast majority
of them have been fixed, the unicode-filtered list only shows three
open ones:

https://github.com/ipython/ipython/issues?labels=unicode&sort=created&direction=desc&state=open&page=1

So it's possible your problem has indeed been fixed, since we know
that at least the worst horrors are gone now, largely thanks to Thomas
Kluyver's insane persistence.

> I don't have any concrete report. As such I withdraw my comment, I
> just wanted to mention that if something smells iffy it might be a
> good idea to resort to regular python.

That is most certainly a good policy, and I use it myself.  IPython is
necessarily a big and complex beast, so if something looks really odd,
the first sanity check is to remove that layer from the problem.

Cheers,

f

From stefan_ml at behnel.de  Thu Jan 26 21:57:42 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Jan 2012 21:57:42 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
Message-ID: <4F21BE46.5050509@behnel.de>

Vitja Makarov, 26.01.2012 21:19:
> 2012/1/27 Stefan Behnel:
>> Robert Bradshaw, 21.01.2012 23:09:
>>> On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel wrote:
>>>> I did some callgrind profiling on Cython's generators and was surprised to
>>>> find that AddTraceback() represents a serious performance penalty for short
>>>> running generators.
>>>>
>>>> I profiled a compiled Python implementation of itertools.groupby(), which
>>>> yields (key, group) tuples where the group is an iterator again. I ran this
>>>> code in Python for benchmarking:
>>>>
>>>> """
>>>> L = sorted(range(1000)*5)
>>>>
>>>> all(list(g) for k,g in groupby(L))
>>>> """
>>>>
>>>> Groups tend to be rather short in real code, often just one or a couple of
>>>> items, so unpacking the group iterator into a list will usually be a quick
>>>> loop and then the generator raises StopIteration on termination and builds
>>>> a traceback for it. According to callgrind (which, I should note, tends to
>>>> overestimate the amount of time spent in memory allocation), the iteration
>>>> during the group unpacking takes about 30% of the overall runtime of the
>>>> all() loop, and the AddTraceback() call at the end of each group traversal
>>>> takes up to 25% (!) on my side. That means that more than 80% of the group
>>>> unpacking time goes into raising StopIteration from the generators. I
>>>> attached the call graph with the relative timings.
>>>>
>>>> About half of the exception raising time is eaten by PyString_FromFormat()
>>>> that builds the function-name + line-position string (which, I may note, is
>>>> basically a convenience feature). This string is a constant for a
>>>> generator's StopIteration exception, at least for each final return point
>>>> in a generator, but here it is being recreated over and over again, for
>>>> each exception that gets raised.
>>>>
>>>> Even if we keep creating a new frame instance each time (which should be ok
>>>> because CPython has a frame instance cache already and we'd only create one
>>>> during the generator lifetime), the whole code object could actually be
>>>> cached after the first creation, preferably bound to the lifetime of the
>>>> generator creator function/method. Or, more generally, one code object per
>>>> generator termination point, which will be a single point in the majority
>>>> of cases. For the specific code above, that should shave off almost 20% of
>>>> the overall runtime of the all() loop.
> 
> I think that could be easily fixed. CPython doesn't add any traceback
> info for generator's ending.
> 
> https://github.com/vitek/cython/commit/63620bc2a29f3064bbdf7a49eefffaae4e3c369d

Interesting. It doesn't solve the general problem of slow exceptions, but I
think that's a beautiful way of fixing this particular performance problem
for generators. Here are the timings. Your change made the above code twice
as fast!

Cython compiled Python implementation of itertools.groupby(), *without*
your change:

python2.7 -m timeit -s 'from cytertools import groupby; \
    L=sorted(range(1000)*5)' 'all(list(g) for k,g in groupby(L))'
100 loops, best of 3: 3.76 msec per loop

Cython compiled Python implementation of itertools.groupby(), *with* your
change:

python2.7 -m timeit -s 'from cytertools import groupby; \
    L=sorted(range(1000)*5)' 'all(list(g) for k,g in groupby(L))'
1000 loops, best of 3: 1.81 msec per loop

The real itertools, for comparison:

python2.7 -m timeit -s 'from itertools import groupby; \
    L=sorted(range(1000)*5)' 'all(list(g) for k,g in groupby(L))'
1000 loops, best of 3: 1.31 msec per loop

That's close.

Stefan

From stefan_ml at behnel.de  Fri Jan 27 08:40:57 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Jan 2012 08:40:57 +0100
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
Message-ID: <4F225509.3070605@behnel.de>

Wes McKinney, 26.01.2012 18:56:
> Just wanted to bring this issue to your guys' attention in case you
> knew what was responsible for this:
> 
> https://github.com/ipython/ipython/issues/1317#issuecomment-3652550
> 
> I traced down the problem (with git bisect) to a seemingly innocuous
> commit referenced in that GitHub thread. The issue seemed to only
> present itself in IPython, so likely there was some problem with
> inspecting the Cython frames for giving context around the full
> traceback.

That's not impossible. Traceback frames behave differently in Cython for
two reasons: a) because they are only being constructed after the fact in
an error handling case, not for normal functions, and b) because Cython
doesn't have real code objects for inspection and fakes them in a rather
ad-hoc way. For example, there is currently no function signature
information in them, and line number computation doesn't use the normal
CPython way that matches the byte code position with a compressed line
table. Instead, Cython creates a new code object for a given line on the
fly and just pretends that the function starts there. This usually works
well enough for a traceback, but this kind of differences makes it quite
possible that IPython makes assumptions about inspection here that Cython
doesn't meet.

In another thread ("AddTraceback() slows down generators"), I was proposing
to cache the code object for a given function. That would then imply
providing a (fake) byte code position map as well and would make it easier
to provide static signature information, thus improving the overall
compatibility with the code objects that CPython creates.

Note that it's unclear without further investigation if the problem you ran
into really has to do with these internal details. I'm just raising a
possible explanation here.

Stefan

From stefan_ml at behnel.de  Fri Jan 27 09:02:24 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Jan 2012 09:02:24 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F21BE46.5050509@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de>
Message-ID: <4F225A10.6020709@behnel.de>

Stefan Behnel, 26.01.2012 21:57:
> Vitja Makarov, 26.01.2012 21:19:
>> 2012/1/27 Stefan Behnel:
>>> Robert Bradshaw, 21.01.2012 23:09:
>>>> On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel wrote:
>>>>> I did some callgrind profiling on Cython's generators and was surprised to
>>>>> find that AddTraceback() represents a serious performance penalty for short
>>>>> running generators.
>>>>>
>>>>> I profiled a compiled Python implementation of itertools.groupby(), which
>>>>> yields (key, group) tuples where the group is an iterator again. I ran this
>>>>> code in Python for benchmarking:
>>>>>
>>>>> """
>>>>> L = sorted(range(1000)*5)
>>>>>
>>>>> all(list(g) for k,g in groupby(L))
>>>>> """
>>>>>
>>>>> Groups tend to be rather short in real code, often just one or a couple of
>>>>> items, so unpacking the group iterator into a list will usually be a quick
>>>>> loop and then the generator raises StopIteration on termination and builds
>>>>> a traceback for it. According to callgrind (which, I should note, tends to
>>>>> overestimate the amount of time spent in memory allocation), the iteration
>>>>> during the group unpacking takes about 30% of the overall runtime of the
>>>>> all() loop, and the AddTraceback() call at the end of each group traversal
>>>>> takes up to 25% (!) on my side. That means that more than 80% of the group
>>>>> unpacking time goes into raising StopIteration from the generators. I
>>>>> attached the call graph with the relative timings.
>>>>>
>>>>> About half of the exception raising time is eaten by PyString_FromFormat()
>>>>> that builds the function-name + line-position string (which, I may note, is
>>>>> basically a convenience feature). This string is a constant for a
>>>>> generator's StopIteration exception, at least for each final return point
>>>>> in a generator, but here it is being recreated over and over again, for
>>>>> each exception that gets raised.
>>>>>
>>>>> Even if we keep creating a new frame instance each time (which should be ok
>>>>> because CPython has a frame instance cache already and we'd only create one
>>>>> during the generator lifetime), the whole code object could actually be
>>>>> cached after the first creation, preferably bound to the lifetime of the
>>>>> generator creator function/method. Or, more generally, one code object per
>>>>> generator termination point, which will be a single point in the majority
>>>>> of cases. For the specific code above, that should shave off almost 20% of
>>>>> the overall runtime of the all() loop.
>>
>> I think that could be easily fixed. CPython doesn't add any traceback
>> info for generator's ending.
>>
>> https://github.com/vitek/cython/commit/63620bc2a29f3064bbdf7a49eefffaae4e3c369d
> 
> Interesting. It doesn't solve the general problem of slow exceptions, but I
> think that's a beautiful way of fixing this particular performance problem
> for generators.

One thing that it doesn't fix is when a generator gets its input from
another iterator and terminates automatically by passing on the
StopIteration that the iterator raises. I.e. any exception *propagation* is
still substantially slower than necessary, and that's a general issue.

Stefan

From vitja.makarov at gmail.com  Fri Jan 27 12:02:39 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 27 Jan 2012 15:02:39 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F225A10.6020709@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
Message-ID: <CAKGHGPQC1j+kUc89hBWtQmL9Dwf1uZPqnCP=pfdAuu-rBHFL9Q@mail.gmail.com>

2012/1/27 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 26.01.2012 21:57:
>> Vitja Makarov, 26.01.2012 21:19:
>>> 2012/1/27 Stefan Behnel:
>>>> Robert Bradshaw, 21.01.2012 23:09:
>>>>> On Sat, Jan 21, 2012 at 10:50 AM, Stefan Behnel wrote:
>>>>>> I did some callgrind profiling on Cython's generators and was surprised to
>>>>>> find that AddTraceback() represents a serious performance penalty for short
>>>>>> running generators.
>>>>>>
>>>>>> I profiled a compiled Python implementation of itertools.groupby(), which
>>>>>> yields (key, group) tuples where the group is an iterator again. I ran this
>>>>>> code in Python for benchmarking:
>>>>>>
>>>>>> """
>>>>>> L = sorted(range(1000)*5)
>>>>>>
>>>>>> all(list(g) for k,g in groupby(L))
>>>>>> """
>>>>>>
>>>>>> Groups tend to be rather short in real code, often just one or a couple of
>>>>>> items, so unpacking the group iterator into a list will usually be a quick
>>>>>> loop and then the generator raises StopIteration on termination and builds
>>>>>> a traceback for it. According to callgrind (which, I should note, tends to
>>>>>> overestimate the amount of time spent in memory allocation), the iteration
>>>>>> during the group unpacking takes about 30% of the overall runtime of the
>>>>>> all() loop, and the AddTraceback() call at the end of each group traversal
>>>>>> takes up to 25% (!) on my side. That means that more than 80% of the group
>>>>>> unpacking time goes into raising StopIteration from the generators. I
>>>>>> attached the call graph with the relative timings.
>>>>>>
>>>>>> About half of the exception raising time is eaten by PyString_FromFormat()
>>>>>> that builds the function-name + line-position string (which, I may note, is
>>>>>> basically a convenience feature). This string is a constant for a
>>>>>> generator's StopIteration exception, at least for each final return point
>>>>>> in a generator, but here it is being recreated over and over again, for
>>>>>> each exception that gets raised.
>>>>>>
>>>>>> Even if we keep creating a new frame instance each time (which should be ok
>>>>>> because CPython has a frame instance cache already and we'd only create one
>>>>>> during the generator lifetime), the whole code object could actually be
>>>>>> cached after the first creation, preferably bound to the lifetime of the
>>>>>> generator creator function/method. Or, more generally, one code object per
>>>>>> generator termination point, which will be a single point in the majority
>>>>>> of cases. For the specific code above, that should shave off almost 20% of
>>>>>> the overall runtime of the all() loop.
>>>
>>> I think that could be easily fixed. CPython doesn't add any traceback
>>> info for generator's ending.
>>>
>>> https://github.com/vitek/cython/commit/63620bc2a29f3064bbdf7a49eefffaae4e3c369d
>>
>> Interesting. It doesn't solve the general problem of slow exceptions, but I
>> think that's a beautiful way of fixing this particular performance problem
>> for generators.
>
> One thing that it doesn't fix is when a generator gets its input from
> another iterator and terminates automatically by passing on the
> StopIteration that the iterator raises. I.e. any exception *propagation* is
> still substantially slower than necessary, and that's a general issue.
>

I'll push my patch to upstream. One question: does it close the ticket or not?

Perhaps it's better to rename your ticket to something like
"AddTraceback() is slow" as itsn't related to generators.


-- 
vitja.

From stefan_ml at behnel.de  Fri Jan 27 14:57:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Jan 2012 14:57:27 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPQC1j+kUc89hBWtQmL9Dwf1uZPqnCP=pfdAuu-rBHFL9Q@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<CAKGHGPQC1j+kUc89hBWtQmL9Dwf1uZPqnCP=pfdAuu-rBHFL9Q@mail.gmail.com>
Message-ID: <4F22AD47.5020901@behnel.de>

Vitja Makarov, 27.01.2012 12:02:
> I'll push my patch to upstream.

Please do.


> One question: does it close the ticket or not?

No.


> Perhaps it's better to rename your ticket to something like
> "AddTraceback() is slow" as itsn't related to generators.

Ok, I'll do that and hang the time machine keys back to where I got them from.

Stefan

From markflorisson88 at gmail.com  Fri Jan 27 17:30:55 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 27 Jan 2012 16:30:55 +0000
Subject: [Cython] [cython-users] Re: How to find out where an
	AttributeError is ignored
In-Reply-To: <CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
Message-ID: <CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>

On 27 January 2012 16:22, mark florisson <markflorisson88 at gmail.com> wrote:
> On 27 January 2012 15:47, Simon King <simon.king at uni-jena.de> wrote:
>> Hi all,
>>
>> I am still *very* frustrated about the fact that Cython does not tell
>> where the error occurs. Since about one week, I am adding lots and
>> lots of lines into Sage that write a log into some file, so that I get
>> at least some idea where the error occurs. But still: Even these
>> extensive logs do not provide a hint on what exactly is happening.
>>
>> How can I patch Cython such that some more information on the location
>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>> R ignored .", but the code lines containing the word "ignored" did not
>> seem to be the lines that are responsible for printing the warning
>> message
>> ? Exception AttributeError: 'PolynomialRing_field_with_category'
>> object has no attribute '_modulus' in ?ignored
>>
>> Can you point me to the file in Sage's Cython spkg which is
>> responsible for printing the warning?
>>
>> Best regards,
>> Simon
>
> These messages are written by PyErr_WriteUnraisable, which is a
> CPython C API function that writes unraisable exceptions. There are
> typically two reasons for unraisable exceptions:
>
> ? ?1) as Robert mentioned, a function that does not allow propagation
> of exceptions, e.g.
>
> ? ? ? ?cdef int func():
> ? ? ? ? ? ?raise Exception
>
> ? ? ? ?Here there is no way to propagate the raised exception, so
> instead one should write something like
>
> ? ? ? ? ? ?cdef int func() except -1: ...
>
> ? ? ? ?Alternatively one may use 'except *' in case there is no error
> indicator and Cython should always check, or "except ? -1" which means
> "-1 may or may not indicate an error".
>
> ? ?2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>
> For functions the right thing is to add an except clause, for
> finalizers and destructors one could use the traceback module, e.g.
>
> ? ?try:
> ? ? ? ?...
> ? ?except:
> ? ? ? ?traceback.print_exc()
>
> If this all still doesn't help, try setting a (deferred) breakpoint on
> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.

Actually, I don't see why the default is to write unraisable
exceptions. Instead Cython could detect that exceptions may propagate
and have callers do the check (i.e. make it implicitly "except *").
Was this not implemented because Cython only knows whether functions
may propagate exceptions at code generation time by looking at the
presence of an error label?
Maybe it could keep code insertion points around for every call to
such a potential function and if the function uses the error label
have the caller perform the check? Although I do forsee problems for
external such functions... maybe Cython could have it's own
threadstate regardless of the GIL which would indicate whether an
error has occurred? e.g. CyErr_Occurred()?

From stefan_ml at behnel.de  Fri Jan 27 17:58:10 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Jan 2012 17:58:10 +0100
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
Message-ID: <4F22D7A2.1050806@behnel.de>

mark florisson, 27.01.2012 17:30:
> On 27 January 2012 16:22, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 27 January 2012 15:47, Simon King <simon.king at uni-jena.de> wrote:
>>> Hi all,
>>>
>>> I am still *very* frustrated about the fact that Cython does not tell
>>> where the error occurs. Since about one week, I am adding lots and
>>> lots of lines into Sage that write a log into some file, so that I get
>>> at least some idea where the error occurs. But still: Even these
>>> extensive logs do not provide a hint on what exactly is happening.
>>>
>>> How can I patch Cython such that some more information on the location
>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>> R ignored .", but the code lines containing the word "ignored" did not
>>> seem to be the lines that are responsible for printing the warning
>>> message
>>>   Exception AttributeError: 'PolynomialRing_field_with_category'
>>> object has no attribute '_modulus' in  ignored
>>>
>>> Can you point me to the file in Sage's Cython spkg which is
>>> responsible for printing the warning?
>>>
>>> Best regards,
>>> Simon
>>
>> These messages are written by PyErr_WriteUnraisable, which is a
>> CPython C API function that writes unraisable exceptions. There are
>> typically two reasons for unraisable exceptions:
>>
>>    1) as Robert mentioned, a function that does not allow propagation
>> of exceptions, e.g.
>>
>>        cdef int func():
>>            raise Exception
>>
>>        Here there is no way to propagate the raised exception, so
>> instead one should write something like
>>
>>            cdef int func() except -1: ...
>>
>>        Alternatively one may use 'except *' in case there is no error
>> indicator and Cython should always check, or "except ? -1" which means
>> "-1 may or may not indicate an error".
>>
>>    2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>
>> For functions the right thing is to add an except clause, for
>> finalizers and destructors one could use the traceback module, e.g.
>>
>>    try:
>>        ...
>>    except:
>>        traceback.print_exc()
>>
>> If this all still doesn't help, try setting a (deferred) breakpoint on
>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
> 
> Actually, I don't see why the default is to write unraisable
> exceptions. Instead Cython could detect that exceptions may propagate
> and have callers do the check (i.e. make it implicitly "except *").
> Was this not implemented because Cython only knows whether functions
> may propagate exceptions at code generation time by looking at the
> presence of an error label?
> Maybe it could keep code insertion points around for every call to
> such a potential function and if the function uses the error label
> have the caller perform the check? Although I do forsee problems for
> external such functions... maybe Cython could have it's own
> threadstate regardless of the GIL which would indicate whether an
> error has occurred? e.g. CyErr_Occurred()?

Yep, those are the kind of reasons why writing unraisable exceptions is the
default.

Stefan

From vitja.makarov at gmail.com  Fri Jan 27 20:17:05 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 27 Jan 2012 23:17:05 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F22AD47.5020901@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<CAKGHGPQC1j+kUc89hBWtQmL9Dwf1uZPqnCP=pfdAuu-rBHFL9Q@mail.gmail.com>
	<4F22AD47.5020901@behnel.de>
Message-ID: <CAKGHGPTSpJ728UHVVd-E7o5Vy3azaxnc+LRuJBbyf68G3N--RA@mail.gmail.com>

2012/1/27 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 27.01.2012 12:02:
>> I'll push my patch to upstream.
>
> Please do.
>

https://github.com/cython/cython/commit/7ae9d5b9a66bb586cd0d03b3aa137eb762602087

>
>> One question: does it close the ticket or not?
>
> No.
>
>
>> Perhaps it's better to rename your ticket to something like
>> "AddTraceback() is slow" as itsn't related to generators.
>
> Ok, I'll do that and hang the time machine keys back to where I got them from.
>

Sorry, I meant no harm.

-- 
vitja.

From d.s.seljebotn at astro.uio.no  Fri Jan 27 21:03:30 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 27 Jan 2012 21:03:30 +0100
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <4F22D7A2.1050806@behnel.de>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de>
Message-ID: <4F230312.9050506@astro.uio.no>

On 01/27/2012 05:58 PM, Stefan Behnel wrote:
> mark florisson, 27.01.2012 17:30:
>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com>  wrote:
>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de>  wrote:
>>>> Hi all,
>>>>
>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>> where the error occurs. Since about one week, I am adding lots and
>>>> lots of lines into Sage that write a log into some file, so that I get
>>>> at least some idea where the error occurs. But still: Even these
>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>
>>>> How can I patch Cython such that some more information on the location
>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>> seem to be the lines that are responsible for printing the warning
>>>> message
>>>>    Exception AttributeError: 'PolynomialRing_field_with_category'
>>>> object has no attribute '_modulus' in  ignored
>>>>
>>>> Can you point me to the file in Sage's Cython spkg which is
>>>> responsible for printing the warning?
>>>>
>>>> Best regards,
>>>> Simon
>>>
>>> These messages are written by PyErr_WriteUnraisable, which is a
>>> CPython C API function that writes unraisable exceptions. There are
>>> typically two reasons for unraisable exceptions:
>>>
>>>     1) as Robert mentioned, a function that does not allow propagation
>>> of exceptions, e.g.
>>>
>>>         cdef int func():
>>>             raise Exception
>>>
>>>         Here there is no way to propagate the raised exception, so
>>> instead one should write something like
>>>
>>>             cdef int func() except -1: ...
>>>
>>>         Alternatively one may use 'except *' in case there is no error
>>> indicator and Cython should always check, or "except ? -1" which means
>>> "-1 may or may not indicate an error".
>>>
>>>     2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>
>>> For functions the right thing is to add an except clause, for
>>> finalizers and destructors one could use the traceback module, e.g.
>>>
>>>     try:
>>>         ...
>>>     except:
>>>         traceback.print_exc()
>>>
>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>
>> Actually, I don't see why the default is to write unraisable
>> exceptions. Instead Cython could detect that exceptions may propagate
>> and have callers do the check (i.e. make it implicitly "except *").

As for speed, there's optimizations on this, e.g., "except? 32434623" if 
the return type is int, "except? 0xfffff..." if the return type is a 
pointer.

And for floating point, we could make our own NaN -- that's obscure 
enough that it could probably be made "except 
cython.cython_exception_nan" by default, not "except? 
cython.cython_exception_nan".

>> Was this not implemented because Cython only knows whether functions
>> may propagate exceptions at code generation time by looking at the
>> presence of an error label?
>> Maybe it could keep code insertion points around for every call to
>> such a potential function and if the function uses the error label
>> have the caller perform the check? Although I do forsee problems for
>> external such functions... maybe Cython could have it's own
>> threadstate regardless of the GIL which would indicate whether an
>> error has occurred? e.g. CyErr_Occurred()?
>
> Yep, those are the kind of reasons why writing unraisable exceptions is the
> default.

Still, the need to explicitly declare "except *" keeps coming up again 
and again, and is really a blemish on the usability of Cython. When 
teaching people Cython, then it's really irritating to have to follow 
"all you need to do is add some 'cdef' and some types" with "and then 
you need to remember to say "except *", or you're in deep trouble". 
Cython sort of looks very elegant until that point...

Long-term we should change CPython to make sure that PyErr_Occurred 
doesn't need the GIL :-) (there's really no reason it should need to go 
beyond checking a thread-local variable).

We could also change the Cython ABI/calling convention -- use the return 
code always for reporting error status (unless there's an "except" or it 
is "cdef extern") and report return value in a pointer out-argument. 
That'd generalize to support multiple typed return values as well. Of 
course, there's a lot of downsides to changing ABI...

Dag Sverre

From markflorisson88 at gmail.com  Fri Jan 27 21:46:08 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 27 Jan 2012 20:46:08 +0000
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <4F230312.9050506@astro.uio.no>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
Message-ID: <CANg26EVqdDt-+sh2=YvnLWeuYub1QPA2YBcDaamJXrmB4MOiQQ@mail.gmail.com>

On 27 January 2012 20:03, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>
>> mark florisson, 27.01.2012 17:30:
>>>
>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com>
>>> ?wrote:
>>>>
>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>> at least some idea where the error occurs. But still: Even these
>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>
>>>>> How can I patch Cython such that some more information on the location
>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>> seem to be the lines that are responsible for printing the warning
>>>>> message
>>>>> ? Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>> object has no attribute '_modulus' in ?ignored
>>>>>
>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>> responsible for printing the warning?
>>>>>
>>>>> Best regards,
>>>>> Simon
>>>>
>>>>
>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>> CPython C API function that writes unraisable exceptions. There are
>>>> typically two reasons for unraisable exceptions:
>>>>
>>>> ? ?1) as Robert mentioned, a function that does not allow propagation
>>>> of exceptions, e.g.
>>>>
>>>> ? ? ? ?cdef int func():
>>>> ? ? ? ? ? ?raise Exception
>>>>
>>>> ? ? ? ?Here there is no way to propagate the raised exception, so
>>>> instead one should write something like
>>>>
>>>> ? ? ? ? ? ?cdef int func() except -1: ...
>>>>
>>>> ? ? ? ?Alternatively one may use 'except *' in case there is no error
>>>> indicator and Cython should always check, or "except ? -1" which means
>>>> "-1 may or may not indicate an error".
>>>>
>>>> ? ?2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>
>>>> For functions the right thing is to add an except clause, for
>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>
>>>> ? ?try:
>>>> ? ? ? ?...
>>>> ? ?except:
>>>> ? ? ? ?traceback.print_exc()
>>>>
>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>
>>>
>>> Actually, I don't see why the default is to write unraisable
>>> exceptions. Instead Cython could detect that exceptions may propagate
>>> and have callers do the check (i.e. make it implicitly "except *").
>
>
> As for speed, there's optimizations on this, e.g., "except? 32434623" if the
> return type is int, "except? 0xfffff..." if the return type is a pointer.
>
> And for floating point, we could make our own NaN -- that's obscure enough
> that it could probably be made "except cython.cython_exception_nan" by
> default, not "except? cython.cython_exception_nan".
>
>
>>> Was this not implemented because Cython only knows whether functions
>>> may propagate exceptions at code generation time by looking at the
>>> presence of an error label?
>>> Maybe it could keep code insertion points around for every call to
>>> such a potential function and if the function uses the error label
>>> have the caller perform the check? Although I do forsee problems for
>>> external such functions... maybe Cython could have it's own
>>> threadstate regardless of the GIL which would indicate whether an
>>> error has occurred? e.g. CyErr_Occurred()?
>>
>>
>> Yep, those are the kind of reasons why writing unraisable exceptions is
>> the
>> default.
>
>
> Still, the need to explicitly declare "except *" keeps coming up again and
> again, and is really a blemish on the usability of Cython. When teaching
> people Cython, then it's really irritating to have to follow "all you need
> to do is add some 'cdef' and some types" with "and then you need to remember
> to say "except *", or you're in deep trouble". Cython sort of looks very
> elegant until that point...

I totally agree. The syntax is kind of bothersome, 'with gil' and
'nogil' are bad enough as it is.

> Long-term we should change CPython to make sure that PyErr_Occurred doesn't
> need the GIL :-) (there's really no reason it should need to go beyond
> checking a thread-local variable).

I think the problem is that thread states are not actually
thread-local data. They are saved on the stack when you release the
GIL and restored when you re-acquire it. As such I think it just keeps
one global C variable around. I have no idea how much overhead it
would be to make it a thread-local variable. But if needed we could
always use a cython-specific thread-local error indicator.

> We could also change the Cython ABI/calling convention -- use the return
> code always for reporting error status (unless there's an "except" or it is
> "cdef extern") and report return value in a pointer out-argument. That'd
> generalize to support multiple typed return values as well. Of course,
> there's a lot of downsides to changing ABI...

I think you can't change the ABI for a public/api function, as it
would need to be callable from C. Also, if the address of the function
is takes you can't do it. But those could be special cases, which
would be promoted to the "except badval" case or the "except *" case.
Or perhaps public and api functions without an except clause should
always use the write unraisable approach, as their callers might be
ignoring the exception entirely (although that could be viewed as a
bug as well).

In general I think a cython thread-local error indicator (in addition
to checking against bad values) would be robust. It wouldn't need to
add a lot of overhead as you can often use the bad value approach
anyway.

> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Jan 27 22:01:18 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Jan 2012 22:01:18 +0100
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <4F230312.9050506@astro.uio.no>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
Message-ID: <4F23109E.3030203@behnel.de>

Dag Sverre Seljebotn, 27.01.2012 21:03:
> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>> mark florisson, 27.01.2012 17:30:
>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com>  wrote:
>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de>  wrote:
>>>>> Hi all,
>>>>>
>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>> at least some idea where the error occurs. But still: Even these
>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>
>>>>> How can I patch Cython such that some more information on the location
>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>> seem to be the lines that are responsible for printing the warning
>>>>> message
>>>>>    Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>> object has no attribute '_modulus' in  ignored
>>>>>
>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>> responsible for printing the warning?
>>>>>
>>>>> Best regards,
>>>>> Simon
>>>>
>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>> CPython C API function that writes unraisable exceptions. There are
>>>> typically two reasons for unraisable exceptions:
>>>>
>>>>     1) as Robert mentioned, a function that does not allow propagation
>>>> of exceptions, e.g.
>>>>
>>>>         cdef int func():
>>>>             raise Exception
>>>>
>>>>         Here there is no way to propagate the raised exception, so
>>>> instead one should write something like
>>>>
>>>>             cdef int func() except -1: ...
>>>>
>>>>         Alternatively one may use 'except *' in case there is no error
>>>> indicator and Cython should always check, or "except ? -1" which means
>>>> "-1 may or may not indicate an error".
>>>>
>>>>     2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>
>>>> For functions the right thing is to add an except clause, for
>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>
>>>>     try:
>>>>         ...
>>>>     except:
>>>>         traceback.print_exc()
>>>>
>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>
>>> Actually, I don't see why the default is to write unraisable
>>> exceptions. Instead Cython could detect that exceptions may propagate
>>> and have callers do the check (i.e. make it implicitly "except *").
> 
> As for speed, there's optimizations on this, e.g., "except? 32434623" if
> the return type is int, "except? 0xfffff..." if the return type is a pointer.
> 
> And for floating point, we could make our own NaN -- that's obscure enough
> that it could probably be made "except cython.cython_exception_nan" by
> default, not "except? cython.cython_exception_nan".

The problem with that is that we can't be sure that Cython will be the only
caller. So exceptions may still not propagate in cases, and users will have
to know about these "obscure" values and that they must deal with them
manually then.

You could add that we'd just have to disable this when user code takes a
pointer from a function, but then, how many rules are there that users will
have to learn and remember after such a change? And what's that for a
language that changes the calling semantics of a function because way down
in the code someone happens to take a pointer to it?


>>> Was this not implemented because Cython only knows whether functions
>>> may propagate exceptions at code generation time by looking at the
>>> presence of an error label?
>>> Maybe it could keep code insertion points around for every call to
>>> such a potential function and if the function uses the error label
>>> have the caller perform the check? Although I do forsee problems for
>>> external such functions... maybe Cython could have it's own
>>> threadstate regardless of the GIL which would indicate whether an
>>> error has occurred? e.g. CyErr_Occurred()?
>>
>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>> default.
> 
> Still,

I wasn't really advocating this behaviour, just indicating that it's hard
to do "better", because this "better" isn't all that clear. It's also not
"better" for all code, which means that we get from one trade-off to
another, while breaking existing code at the same time. Not exactly
paradise on either side of the tunnel.

One example that keeps popping up in my mind is callback functions that
cannot propagate errors, at least not the CPython way. I have a couple of
those in lxml, even some returning void. So I wrap their code in a bare
try-except and when an exception strikes, I set a C error flag to tell the
C library that something went wrong and return normally. No Python code
outside of the try block. But Cython still generates code for unraisable
errors. Why? Because the internal code that handles the bare except clause
may fail and raise an exception. How about that?


> the need to explicitly declare "except *" keeps coming up again and
> again, and is really a blemish on the usability of Cython. When teaching
> people Cython, then it's really irritating to have to follow "all you need
> to do is add some 'cdef' and some types" with "and then you need to
> remember to say "except *", or you're in deep trouble". Cython sort of
> looks very elegant until that point...

I know what this feels like. The problem is that these things *are* complex.


> Long-term we should change CPython to make sure that PyErr_Occurred doesn't
> need the GIL :-) (there's really no reason it should need to go beyond
> checking a thread-local variable).

I always wondered about that, too. Still, "long-term" here basically means
"when all current CPython versions that work like this are out of use",
because we cannot base language semantics on specific runtime CPython versions.

Stefan

From markflorisson88 at gmail.com  Fri Jan 27 22:23:41 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 27 Jan 2012 21:23:41 +0000
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <4F23109E.3030203@behnel.de>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
Message-ID: <CANg26EWEu=Fz8Bw+ca=tYB2XK0rgKckDQQzbnh5L9LSW7UZt9A@mail.gmail.com>

On 27 January 2012 21:01, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 27.01.2012 21:03:
>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>> mark florisson, 27.01.2012 17:30:
>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com> ?wrote:
>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>
>>>>>> How can I patch Cython such that some more information on the location
>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>> message
>>>>>> ? ?Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>> object has no attribute '_modulus' in ?ignored
>>>>>>
>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>> responsible for printing the warning?
>>>>>>
>>>>>> Best regards,
>>>>>> Simon
>>>>>
>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>> typically two reasons for unraisable exceptions:
>>>>>
>>>>> ? ? 1) as Robert mentioned, a function that does not allow propagation
>>>>> of exceptions, e.g.
>>>>>
>>>>> ? ? ? ? cdef int func():
>>>>> ? ? ? ? ? ? raise Exception
>>>>>
>>>>> ? ? ? ? Here there is no way to propagate the raised exception, so
>>>>> instead one should write something like
>>>>>
>>>>> ? ? ? ? ? ? cdef int func() except -1: ...
>>>>>
>>>>> ? ? ? ? Alternatively one may use 'except *' in case there is no error
>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>> "-1 may or may not indicate an error".
>>>>>
>>>>> ? ? 2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>
>>>>> For functions the right thing is to add an except clause, for
>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>
>>>>> ? ? try:
>>>>> ? ? ? ? ...
>>>>> ? ? except:
>>>>> ? ? ? ? traceback.print_exc()
>>>>>
>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>
>>>> Actually, I don't see why the default is to write unraisable
>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>> and have callers do the check (i.e. make it implicitly "except *").
>>
>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>
>> And for floating point, we could make our own NaN -- that's obscure enough
>> that it could probably be made "except cython.cython_exception_nan" by
>> default, not "except? cython.cython_exception_nan".
>
> The problem with that is that we can't be sure that Cython will be the only
> caller. So exceptions may still not propagate in cases, and users will have
> to know about these "obscure" values and that they must deal with them
> manually then.

I don't think users would need to learn about the return values, the
thing is that if exceptions propagate in the first place, then the
return value is undefined to begin with. Users could use
PyErr_Occurred(), or in case the user is calling from a nogil context,
CyErr_Occurred() (not recommending this, bear with me :).

> You could add that we'd just have to disable this when user code takes a
> pointer from a function, but then, how many rules are there that users will
> have to learn and remember after such a change? And what's that for a
> language that changes the calling semantics of a function because way down
> in the code someone happens to take a pointer to it?
>

Right. I think we could generate two functions, one function that acts
like it does now and writes unraisable exceptions. This would also be
the public or api function, and the function you get the address to
when taking it. The second function can be implemented however we want
as only Cython will have to care. This one would propagate errors.

>>>> Was this not implemented because Cython only knows whether functions
>>>> may propagate exceptions at code generation time by looking at the
>>>> presence of an error label?
>>>> Maybe it could keep code insertion points around for every call to
>>>> such a potential function and if the function uses the error label
>>>> have the caller perform the check? Although I do forsee problems for
>>>> external such functions... maybe Cython could have it's own
>>>> threadstate regardless of the GIL which would indicate whether an
>>>> error has occurred? e.g. CyErr_Occurred()?
>>>
>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>> default.
>>
>> Still,
>
> I wasn't really advocating this behaviour, just indicating that it's hard
> to do "better", because this "better" isn't all that clear. It's also not
> "better" for all code, which means that we get from one trade-off to
> another, while breaking existing code at the same time. Not exactly
> paradise on either side of the tunnel.
>
> One example that keeps popping up in my mind is callback functions that
> cannot propagate errors, at least not the CPython way. I have a couple of
> those in lxml, even some returning void. So I wrap their code in a bare
> try-except and when an exception strikes, I set a C error flag to tell the
> C library that something went wrong and return normally. No Python code
> outside of the try block. But Cython still generates code for unraisable
> errors. Why? Because the internal code that handles the bare except clause
> may fail and raise an exception. How about that?
>

That's a good point (so two functions could work here).

>> the need to explicitly declare "except *" keeps coming up again and
>> again, and is really a blemish on the usability of Cython. When teaching
>> people Cython, then it's really irritating to have to follow "all you need
>> to do is add some 'cdef' and some types" with "and then you need to
>> remember to say "except *", or you're in deep trouble". Cython sort of
>> looks very elegant until that point...
>
> I know what this feels like. The problem is that these things *are* complex.
>
>
>> Long-term we should change CPython to make sure that PyErr_Occurred doesn't
>> need the GIL :-) (there's really no reason it should need to go beyond
>> checking a thread-local variable).
>
> I always wondered about that, too. Still, "long-term" here basically means
> "when all current CPython versions that work like this are out of use",
> because we cannot base language semantics on specific runtime CPython versions.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sat Jan 28 08:09:09 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jan 2012 08:09:09 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPTSpJ728UHVVd-E7o5Vy3azaxnc+LRuJBbyf68G3N--RA@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<CAKGHGPQC1j+kUc89hBWtQmL9Dwf1uZPqnCP=pfdAuu-rBHFL9Q@mail.gmail.com>
	<4F22AD47.5020901@behnel.de>
	<CAKGHGPTSpJ728UHVVd-E7o5Vy3azaxnc+LRuJBbyf68G3N--RA@mail.gmail.com>
Message-ID: <4F239F15.4040402@behnel.de>

Vitja Makarov, 27.01.2012 20:17:
> https://github.com/cython/cython/commit/7ae9d5b9a66bb586cd0d03b3aa137eb762602087

Looks like it worked:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-pybenchmarks-py3k/318/artifact/bench_chart.html#nqueens

https://sage.math.washington.edu:8091/hudson/job/cython-devel-pybenchmarks-py27/300/artifact/bench_chart.html#nqueens

Stefan

From vitja.makarov at gmail.com  Sat Jan 28 17:05:58 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 28 Jan 2012 20:05:58 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <4F2083B0.9020209@creativetrax.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
Message-ID: <CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>

2012/1/26 Jason Grout <jason-sage at creativetrax.com>:
> On 1/25/12 11:39 AM, Robert Bradshaw wrote:
>>
>> install
>>
>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>> by downloading it and running "sage -i cython-devel.spkg"
>
>
>
> In fact, you could just do
>
> sage -i
> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>
> and Sage will (at least, should) download it for you, so that's even one
> less step!
>
> Jason
>

Thanks for detailed instruction! I've successfully built it.

"sage -t -gdb ./...." doesn't work, is that a bug?

vitja at mchome:~/Downloads/sage-4.8$ ./sage  -t -gdb
devel/sage/sage/combinat/sf/macdonald.py
sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
********************************************************************************
Type r at the (gdb) prompt to run the doctests.
Type bt if there is a crash to see a traceback.
********************************************************************************
gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
starting cmd gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
ImportError: No module named site
	 [0.2 s]

----------------------------------------------------------------------
The following tests failed:


	sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
Total time for all tests: 0.2 seconds


I've found another way to run tests (using sage -sh and then direct
python ~/.sage/tmp/...py)

So I found one of the problems. Here is minimal cython example:

def foo(values):
    return (0,)*len(values)
foo([1,2,3])

len(values) somehow is passed as an integer to PyObject_Multiply()


-- 
vitja.

From stefan_ml at behnel.de  Sat Jan 28 19:38:25 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jan 2012 19:38:25 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F225A10.6020709@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
Message-ID: <4F2440A1.7050505@behnel.de>

Stefan Behnel, 27.01.2012 09:02:
> any exception *propagation* is
> still substantially slower than necessary, and that's a general issue.

Here's a general take on a code object cache for exception propagation.

https://github.com/scoder/cython/commit/ad18e0208

When I raise an exception in test code that propagates through a Python
call hierarchy of four functions before being caught, the cache gives me
something like a 2x speedup in total. Not bad. When I do the same for cdef
functions, it's more like 4-5x.

The main idea is to cache the objects in a reallocable C array and bisect
into it based on the C code "__LINE__" of the exception, which should be
unique enough for a given module.

It's a global cache that doesn't limit the lifetime of code objects  (well,
up to the lifetime of the module, obviously). I don't know if that's a
problem because the number of code objects is only bounded by the number of
exception origination points in the C source code, which is usually quite
large. However, only a tiny fraction of those will ever raise or propagate
an exception in practice, so the real number of cached code objects will be
substantially smaller.

Maybe thorough test suites with lots of failure testing would notice a
difference in memory consumption, even though a single code objects isn't
all that large either...

What do you think?

Stefan

From markflorisson88 at gmail.com  Sat Jan 28 20:07:30 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 28 Jan 2012 19:07:30 +0000
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F2440A1.7050505@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
Message-ID: <CANg26EWoGQwGN9XaMyiv-tYXhHx2thQrNxyMG=ZnrcFHyTH+YA@mail.gmail.com>

On 28 January 2012 18:38, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 27.01.2012 09:02:
>> any exception *propagation* is
>> still substantially slower than necessary, and that's a general issue.
>
> Here's a general take on a code object cache for exception propagation.
>
> https://github.com/scoder/cython/commit/ad18e0208
>
> When I raise an exception in test code that propagates through a Python
> call hierarchy of four functions before being caught, the cache gives me
> something like a 2x speedup in total. Not bad. When I do the same for cdef
> functions, it's more like 4-5x.
>
> The main idea is to cache the objects in a reallocable C array and bisect
> into it based on the C code "__LINE__" of the exception, which should be
> unique enough for a given module.
>
> It's a global cache that doesn't limit the lifetime of code objects ?(well,
> up to the lifetime of the module, obviously). I don't know if that's a
> problem because the number of code objects is only bounded by the number of
> exception origination points in the C source code, which is usually quite
> large. However, only a tiny fraction of those will ever raise or propagate
> an exception in practice, so the real number of cached code objects will be
> substantially smaller.
>
> Maybe thorough test suites with lots of failure testing would notice a
> difference in memory consumption, even though a single code objects isn't
> all that large either...
>
> What do you think?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Nice. I have a question, couldn't you save the frame object instead of
the code object?
I do think PyCodeObject is rather large, on my 64 bit machine it is
120 bytes, not accounting for any of the objects it holds (not saying
that's a problem, just pointing it out).

Would it help if we would pass in the position information string
object constant, to avoid the PyString_Format? That optimization would
only save 14% though. But additionally, the function name could be a
string object constant, which could be shared by all exceptions in one
function, avoiding another PyString_FromString.

From vitja.makarov at gmail.com  Sat Jan 28 20:41:46 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 28 Jan 2012 23:41:46 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F2440A1.7050505@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
Message-ID: <CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>

2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 27.01.2012 09:02:
>> any exception *propagation* is
>> still substantially slower than necessary, and that's a general issue.
>
> Here's a general take on a code object cache for exception propagation.
>
> https://github.com/scoder/cython/commit/ad18e0208
>
> When I raise an exception in test code that propagates through a Python
> call hierarchy of four functions before being caught, the cache gives me
> something like a 2x speedup in total. Not bad. When I do the same for cdef
> functions, it's more like 4-5x.
>
> The main idea is to cache the objects in a reallocable C array and bisect
> into it based on the C code "__LINE__" of the exception, which should be
> unique enough for a given module.
>
> It's a global cache that doesn't limit the lifetime of code objects ?(well,
> up to the lifetime of the module, obviously). I don't know if that's a
> problem because the number of code objects is only bounded by the number of
> exception origination points in the C source code, which is usually quite
> large. However, only a tiny fraction of those will ever raise or propagate
> an exception in practice, so the real number of cached code objects will be
> substantially smaller.
>
> Maybe thorough test suites with lots of failure testing would notice a
> difference in memory consumption, even though a single code objects isn't
> all that large either...
>
> What do you think?
>

We already have --no-c-in-traceback flag that disables C line numbers
in traceback.
What's about enabling it by default?

-- 
vitja.

From markflorisson88 at gmail.com  Sat Jan 28 20:48:36 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 28 Jan 2012 19:48:36 +0000
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
Message-ID: <CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>

On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>> Stefan Behnel, 27.01.2012 09:02:
>>> any exception *propagation* is
>>> still substantially slower than necessary, and that's a general issue.
>>
>> Here's a general take on a code object cache for exception propagation.
>>
>> https://github.com/scoder/cython/commit/ad18e0208
>>
>> When I raise an exception in test code that propagates through a Python
>> call hierarchy of four functions before being caught, the cache gives me
>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>> functions, it's more like 4-5x.
>>
>> The main idea is to cache the objects in a reallocable C array and bisect
>> into it based on the C code "__LINE__" of the exception, which should be
>> unique enough for a given module.
>>
>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>> up to the lifetime of the module, obviously). I don't know if that's a
>> problem because the number of code objects is only bounded by the number of
>> exception origination points in the C source code, which is usually quite
>> large. However, only a tiny fraction of those will ever raise or propagate
>> an exception in practice, so the real number of cached code objects will be
>> substantially smaller.
>>
>> Maybe thorough test suites with lots of failure testing would notice a
>> difference in memory consumption, even though a single code objects isn't
>> all that large either...
>>
>> What do you think?
>>
>
> We already have --no-c-in-traceback flag that disables C line numbers
> in traceback.
> What's about enabling it by default?
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I'm quite attached to that feature actually :), it would be pretty
annoying to disable that flag every time. And what would disabling
that option gain, as the current code still formats the filename and
function name.

From markflorisson88 at gmail.com  Sat Jan 28 20:51:45 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 28 Jan 2012 19:51:45 +0000
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
Message-ID: <CANg26EWD31HsY2PY4-3ONCES56kg0NczobVK9k6U5Frbacx97A@mail.gmail.com>

On 28 January 2012 19:48, mark florisson <markflorisson88 at gmail.com> wrote:
> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>> Stefan Behnel, 27.01.2012 09:02:
>>>> any exception *propagation* is
>>>> still substantially slower than necessary, and that's a general issue.
>>>
>>> Here's a general take on a code object cache for exception propagation.
>>>
>>> https://github.com/scoder/cython/commit/ad18e0208
>>>
>>> When I raise an exception in test code that propagates through a Python
>>> call hierarchy of four functions before being caught, the cache gives me
>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>> functions, it's more like 4-5x.
>>>
>>> The main idea is to cache the objects in a reallocable C array and bisect
>>> into it based on the C code "__LINE__" of the exception, which should be
>>> unique enough for a given module.
>>>
>>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>>> up to the lifetime of the module, obviously). I don't know if that's a
>>> problem because the number of code objects is only bounded by the number of
>>> exception origination points in the C source code, which is usually quite
>>> large. However, only a tiny fraction of those will ever raise or propagate
>>> an exception in practice, so the real number of cached code objects will be
>>> substantially smaller.
>>>
>>> Maybe thorough test suites with lots of failure testing would notice a
>>> difference in memory consumption, even though a single code objects isn't
>>> all that large either...
>>>
>>> What do you think?
>>>
>>
>> We already have --no-c-in-traceback flag that disables C line numbers
>> in traceback.
>> What's about enabling it by default?
>>
>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> I'm quite attached to that feature actually :), it would be pretty
> annoying to disable that flag every time. And what would disabling
> that option gain, as the current code still formats the filename and
> function name.

Ah, you mean it would cache less code objects for multiple possible
errors in expressions (or statements) on a single source line?

From vitja.makarov at gmail.com  Sat Jan 28 20:58:13 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 28 Jan 2012 23:58:13 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
Message-ID: <CAKGHGPRkS17nY2X=adxR2o4VBqGek7WYiyuBt+fMYVHbvoR_WA@mail.gmail.com>

2012/1/28 mark florisson <markflorisson88 at gmail.com>:
> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>> Stefan Behnel, 27.01.2012 09:02:
>>>> any exception *propagation* is
>>>> still substantially slower than necessary, and that's a general issue.
>>>
>>> Here's a general take on a code object cache for exception propagation.
>>>
>>> https://github.com/scoder/cython/commit/ad18e0208
>>>
>>> When I raise an exception in test code that propagates through a Python
>>> call hierarchy of four functions before being caught, the cache gives me
>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>> functions, it's more like 4-5x.
>>>
>>> The main idea is to cache the objects in a reallocable C array and bisect
>>> into it based on the C code "__LINE__" of the exception, which should be
>>> unique enough for a given module.
>>>
>>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>>> up to the lifetime of the module, obviously). I don't know if that's a
>>> problem because the number of code objects is only bounded by the number of
>>> exception origination points in the C source code, which is usually quite
>>> large. However, only a tiny fraction of those will ever raise or propagate
>>> an exception in practice, so the real number of cached code objects will be
>>> substantially smaller.
>>>
>>> Maybe thorough test suites with lots of failure testing would notice a
>>> difference in memory consumption, even though a single code objects isn't
>>> all that large either...
>>>
>>> What do you think?
>>>
>>
>> We already have --no-c-in-traceback flag that disables C line numbers
>> in traceback.
>> What's about enabling it by default?
>>
> I'm quite attached to that feature actually :), it would be pretty
> annoying to disable that flag every time. And what would disabling
> that option gain, as the current code still formats the filename and
> function name.

It's rather useful for developers or debugging. Most of the people
don't need it.

Here is simple benchmark:
# upstream/master: 6.38ms
# upstream/master (no-c-in-traceback): 3.07ms
# scoder/master: 1.31ms
def foo():
    raise ValueError

def testit():
    cdef int i
    for i in range(10000):
        try:
            foo()
        except:
            pass

Stefan's branch wins but:
 - there is only one item in the cache and it's always hit
 - we can still avoid calling PyString_FromString() making function
name and source file name a python const (I've tried it and I get
2.28ms)

-- 
vitja.

From vitja.makarov at gmail.com  Sat Jan 28 20:59:55 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 28 Jan 2012 23:59:55 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CANg26EWD31HsY2PY4-3ONCES56kg0NczobVK9k6U5Frbacx97A@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
	<CANg26EWD31HsY2PY4-3ONCES56kg0NczobVK9k6U5Frbacx97A@mail.gmail.com>
Message-ID: <CAKGHGPTro=kMviA44GdTumDNPWAypmxJCSSYz+kUuCbA4yP1hw@mail.gmail.com>

2012/1/28 mark florisson <markflorisson88 at gmail.com>:
> On 28 January 2012 19:48, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Stefan Behnel, 27.01.2012 09:02:
>>>>> any exception *propagation* is
>>>>> still substantially slower than necessary, and that's a general issue.
>>>>
>>>> Here's a general take on a code object cache for exception propagation.
>>>>
>>>> https://github.com/scoder/cython/commit/ad18e0208
>>>>
>>>> When I raise an exception in test code that propagates through a Python
>>>> call hierarchy of four functions before being caught, the cache gives me
>>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>>> functions, it's more like 4-5x.
>>>>
>>>> The main idea is to cache the objects in a reallocable C array and bisect
>>>> into it based on the C code "__LINE__" of the exception, which should be
>>>> unique enough for a given module.
>>>>
>>>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>>>> up to the lifetime of the module, obviously). I don't know if that's a
>>>> problem because the number of code objects is only bounded by the number of
>>>> exception origination points in the C source code, which is usually quite
>>>> large. However, only a tiny fraction of those will ever raise or propagate
>>>> an exception in practice, so the real number of cached code objects will be
>>>> substantially smaller.
>>>>
>>>> Maybe thorough test suites with lots of failure testing would notice a
>>>> difference in memory consumption, even though a single code objects isn't
>>>> all that large either...
>>>>
>>>> What do you think?
>>>>
>>>
>>> We already have --no-c-in-traceback flag that disables C line numbers
>>> in traceback.
>>> What's about enabling it by default?
>>>
>>> --
>>> vitja.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> I'm quite attached to that feature actually :), it would be pretty
>> annoying to disable that flag every time. And what would disabling
>> that option gain, as the current code still formats the filename and
>> function name.
>
> Ah, you mean it would cache less code objects for multiple possible
> errors in expressions (or statements) on a single source line?

Not exactly. I mean PyString_Format() is called to add C filename and
C lineno to python function name.


-- 
vitja.

From stefan_ml at behnel.de  Sat Jan 28 21:14:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jan 2012 21:14:28 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CANg26EWoGQwGN9XaMyiv-tYXhHx2thQrNxyMG=ZnrcFHyTH+YA@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CANg26EWoGQwGN9XaMyiv-tYXhHx2thQrNxyMG=ZnrcFHyTH+YA@mail.gmail.com>
Message-ID: <4F245724.8070208@behnel.de>

mark florisson, 28.01.2012 20:07:
> On 28 January 2012 18:38, Stefan Behnel wrote:
>> Stefan Behnel, 27.01.2012 09:02:
>>> any exception *propagation* is
>>> still substantially slower than necessary, and that's a general issue.
>>
>> Here's a general take on a code object cache for exception propagation.
>>
>> https://github.com/scoder/cython/commit/ad18e0208
>>
>> When I raise an exception in test code that propagates through a Python
>> call hierarchy of four functions before being caught, the cache gives me
>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>> functions, it's more like 4-5x.
>>
>> The main idea is to cache the objects in a reallocable C array and bisect
>> into it based on the C code "__LINE__" of the exception, which should be
>> unique enough for a given module.
>>
>> It's a global cache that doesn't limit the lifetime of code objects  (well,
>> up to the lifetime of the module, obviously). I don't know if that's a
>> problem because the number of code objects is only bounded by the number of
>> exception origination points in the C source code, which is usually quite
>> large. However, only a tiny fraction of those will ever raise or propagate
>> an exception in practice, so the real number of cached code objects will be
>> substantially smaller.
>>
>> Maybe thorough test suites with lots of failure testing would notice a
>> difference in memory consumption, even though a single code objects isn't
>> all that large either...
>>
>> What do you think?
> 
> Nice. I have a question, couldn't you save the frame object instead of
> the code object?

Technically, yes. However, eventually, I'd like to make the CodeObject
constant for the whole function and let CPython calculate the Python code
source line based on the frame's "f_lasti" field when the line number is
actually accessed.

For now, I wouldn't mind cashing the whole frame until the above
optimisation gets implemented.


> I do think PyCodeObject is rather large, on my 64 bit machine it is
> 120 bytes, not accounting for any of the objects it holds (not saying
> that's a problem, just pointing it out).

Hmm, ok. That's not really cheap, especially given the amount of redundancy
in the content. Maybe we could intern the strings after creating them.


> Would it help if we would pass in the position information string
> object constant, to avoid the PyString_Format? That optimization would
> only save 14% though.

PyString_FromFormat() is impressively expensive. So, yes, as Vitja figured
out, that would help. But I actually like the feature.


> But additionally, the function name could be a
> string object constant, which could be shared by all exceptions in one
> function, avoiding another PyString_FromString.

Yes, and in many cases, certainly for all Python functions, it's already
there anyway.

I originally rejected that idea because more string constants add to the
module initialisation time (performance analysis pending here). It may
still be worth doing at least for Python functions.

Stefan

From stefan_ml at behnel.de  Sat Jan 28 21:24:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jan 2012 21:24:35 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPRkS17nY2X=adxR2o4VBqGek7WYiyuBt+fMYVHbvoR_WA@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
	<CAKGHGPRkS17nY2X=adxR2o4VBqGek7WYiyuBt+fMYVHbvoR_WA@mail.gmail.com>
Message-ID: <4F245983.9000709@behnel.de>

Vitja Makarov, 28.01.2012 20:58:
> 2012/1/28 mark florisson <markflorisson88 at gmail.com>:
>> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Stefan Behnel, 27.01.2012 09:02:
>>>>> any exception *propagation* is
>>>>> still substantially slower than necessary, and that's a general issue.
>>>>
>>>> Here's a general take on a code object cache for exception propagation.
>>>>
>>>> https://github.com/scoder/cython/commit/ad18e0208
>>>>
>>>> When I raise an exception in test code that propagates through a Python
>>>> call hierarchy of four functions before being caught, the cache gives me
>>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>>> functions, it's more like 4-5x.
>>>>
>>>> The main idea is to cache the objects in a reallocable C array and bisect
>>>> into it based on the C code "__LINE__" of the exception, which should be
>>>> unique enough for a given module.
>>>>
>>>> It's a global cache that doesn't limit the lifetime of code objects  (well,
>>>> up to the lifetime of the module, obviously). I don't know if that's a
>>>> problem because the number of code objects is only bounded by the number of
>>>> exception origination points in the C source code, which is usually quite
>>>> large. However, only a tiny fraction of those will ever raise or propagate
>>>> an exception in practice, so the real number of cached code objects will be
>>>> substantially smaller.
>>>>
>>>> Maybe thorough test suites with lots of failure testing would notice a
>>>> difference in memory consumption, even though a single code objects isn't
>>>> all that large either...
>>>>
>>>> What do you think?
>>>>
>>>
>>> We already have --no-c-in-traceback flag that disables C line numbers
>>> in traceback.
>>> What's about enabling it by default?
>>>
>> I'm quite attached to that feature actually :), it would be pretty
>> annoying to disable that flag every time. And what would disabling
>> that option gain, as the current code still formats the filename and
>> function name.
> 
> It's rather useful for developers or debugging. Most of the people
> don't need it.

Not untrue. However, at least a majority of developers should be able to
make use of it when it's there, and code is several times more often built
for testing and debugging than for production. So I consider it a virtue
that it's on by default.


> Here is simple benchmark:
> # upstream/master: 6.38ms
> # upstream/master (no-c-in-traceback): 3.07ms
> # scoder/master: 1.31ms
> def foo():
>     raise ValueError
> 
> def testit():
>     cdef int i
>     for i in range(10000):
>         try:
>             foo()
>         except:
>             pass
> 
> Stefan's branch wins but:
>  - there is only one item in the cache and it's always hit

Even if there were substantially more, binary search is so fast you'd
hardly notice the difference.

(BTW, I just noticed that my binary search implementation is buggy - not a
complete surprise. I'll add some tests for it.)


>  - we can still avoid calling PyString_FromString() making function
> name and source file name a python const (I've tried it and I get
> 2.28ms)

I wouldn't mind, but it would be nice to get lazy initialisation for them.

Stefan

From markflorisson88 at gmail.com  Sat Jan 28 21:25:30 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 28 Jan 2012 20:25:30 +0000
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPTro=kMviA44GdTumDNPWAypmxJCSSYz+kUuCbA4yP1hw@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
	<CANg26EWD31HsY2PY4-3ONCES56kg0NczobVK9k6U5Frbacx97A@mail.gmail.com>
	<CAKGHGPTro=kMviA44GdTumDNPWAypmxJCSSYz+kUuCbA4yP1hw@mail.gmail.com>
Message-ID: <CANg26EVgyu8tF4883b+2DgCO3N1ZODXnGmv4+PQCSvMJhHcktg@mail.gmail.com>

On 28 January 2012 19:59, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/28 mark florisson <markflorisson88 at gmail.com>:
>> On 28 January 2012 19:48, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>>>> Stefan Behnel, 27.01.2012 09:02:
>>>>>> any exception *propagation* is
>>>>>> still substantially slower than necessary, and that's a general issue.
>>>>>
>>>>> Here's a general take on a code object cache for exception propagation.
>>>>>
>>>>> https://github.com/scoder/cython/commit/ad18e0208
>>>>>
>>>>> When I raise an exception in test code that propagates through a Python
>>>>> call hierarchy of four functions before being caught, the cache gives me
>>>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>>>> functions, it's more like 4-5x.
>>>>>
>>>>> The main idea is to cache the objects in a reallocable C array and bisect
>>>>> into it based on the C code "__LINE__" of the exception, which should be
>>>>> unique enough for a given module.
>>>>>
>>>>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>>>>> up to the lifetime of the module, obviously). I don't know if that's a
>>>>> problem because the number of code objects is only bounded by the number of
>>>>> exception origination points in the C source code, which is usually quite
>>>>> large. However, only a tiny fraction of those will ever raise or propagate
>>>>> an exception in practice, so the real number of cached code objects will be
>>>>> substantially smaller.
>>>>>
>>>>> Maybe thorough test suites with lots of failure testing would notice a
>>>>> difference in memory consumption, even though a single code objects isn't
>>>>> all that large either...
>>>>>
>>>>> What do you think?
>>>>>
>>>>
>>>> We already have --no-c-in-traceback flag that disables C line numbers
>>>> in traceback.
>>>> What's about enabling it by default?
>>>>
>>>> --
>>>> vitja.
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> I'm quite attached to that feature actually :), it would be pretty
>>> annoying to disable that flag every time. And what would disabling
>>> that option gain, as the current code still formats the filename and
>>> function name.
>>
>> Ah, you mean it would cache less code objects for multiple possible
>> errors in expressions (or statements) on a single source line?
>
> Not exactly. I mean PyString_Format() is called to add C filename and
> C lineno to python function name.
>

Ah indeed, the source lineno is only added to the code object of course.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Sat Jan 28 21:41:15 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 29 Jan 2012 00:41:15 +0400
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F245983.9000709@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
	<CAKGHGPRkS17nY2X=adxR2o4VBqGek7WYiyuBt+fMYVHbvoR_WA@mail.gmail.com>
	<4F245983.9000709@behnel.de>
Message-ID: <CAKGHGPS=ON0u9YJPe=-pLMcJ+-ttR4teP13Ddb7oN65YhOtw2Q@mail.gmail.com>

2012/1/29 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 28.01.2012 20:58:
>> 2012/1/28 mark florisson <markflorisson88 at gmail.com>:
>>> On 28 January 2012 19:41, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/1/28 Stefan Behnel <stefan_ml at behnel.de>:
>>>>> Stefan Behnel, 27.01.2012 09:02:
>>>>>> any exception *propagation* is
>>>>>> still substantially slower than necessary, and that's a general issue.
>>>>>
>>>>> Here's a general take on a code object cache for exception propagation.
>>>>>
>>>>> https://github.com/scoder/cython/commit/ad18e0208
>>>>>
>>>>> When I raise an exception in test code that propagates through a Python
>>>>> call hierarchy of four functions before being caught, the cache gives me
>>>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>>>> functions, it's more like 4-5x.
>>>>>
>>>>> The main idea is to cache the objects in a reallocable C array and bisect
>>>>> into it based on the C code "__LINE__" of the exception, which should be
>>>>> unique enough for a given module.
>>>>>
>>>>> It's a global cache that doesn't limit the lifetime of code objects ?(well,
>>>>> up to the lifetime of the module, obviously). I don't know if that's a
>>>>> problem because the number of code objects is only bounded by the number of
>>>>> exception origination points in the C source code, which is usually quite
>>>>> large. However, only a tiny fraction of those will ever raise or propagate
>>>>> an exception in practice, so the real number of cached code objects will be
>>>>> substantially smaller.
>>>>>
>>>>> Maybe thorough test suites with lots of failure testing would notice a
>>>>> difference in memory consumption, even though a single code objects isn't
>>>>> all that large either...
>>>>>
>>>>> What do you think?
>>>>>
>>>>
>>>> We already have --no-c-in-traceback flag that disables C line numbers
>>>> in traceback.
>>>> What's about enabling it by default?
>>>>
>>> I'm quite attached to that feature actually :), it would be pretty
>>> annoying to disable that flag every time. And what would disabling
>>> that option gain, as the current code still formats the filename and
>>> function name.
>>
>> It's rather useful for developers or debugging. Most of the people
>> don't need it.
>
> Not untrue. However, at least a majority of developers should be able to
> make use of it when it's there, and code is several times more often built
> for testing and debugging than for production. So I consider it a virtue
> that it's on by default.
>
>
>> Here is simple benchmark:
>> # upstream/master: 6.38ms
>> # upstream/master (no-c-in-traceback): 3.07ms
>> # scoder/master: 1.31ms
>> def foo():
>> ? ? raise ValueError
>>
>> def testit():
>> ? ? cdef int i
>> ? ? for i in range(10000):
>> ? ? ? ? try:
>> ? ? ? ? ? ? foo()
>> ? ? ? ? except:
>> ? ? ? ? ? ? pass
>>
>> Stefan's branch wins but:
>> ?- there is only one item in the cache and it's always hit
>
> Even if there were substantially more, binary search is so fast you'd
> hardly notice the difference.
>

Yes, I'm a little bit worried about insertions. But anyway I like it.

With --no-c-in-traceback python lineno should be used as a key.


> (BTW, I just noticed that my binary search implementation is buggy - not a
> complete surprise. I'll add some tests for it.)
>
>
>> ?- we can still avoid calling PyString_FromString() making function
>> name and source file name a python const (I've tried it and I get
>> 2.28ms)
>
> I wouldn't mind, but it would be nice to get lazy initialisation for them.
>



-- 
vitja.

From stefan_ml at behnel.de  Sun Jan 29 09:54:37 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 29 Jan 2012 09:54:37 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <CAKGHGPS=ON0u9YJPe=-pLMcJ+-ttR4teP13Ddb7oN65YhOtw2Q@mail.gmail.com>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CAKGHGPQDkxokp-WNoqcjiSF6F_gC8FG5DQDrW-mdfR6YaDYnKQ@mail.gmail.com>
	<CANg26EVPCbVP-P1svGV8qMsCwbkb=fJ_czn09d8kvO+V_TT27A@mail.gmail.com>
	<CAKGHGPRkS17nY2X=adxR2o4VBqGek7WYiyuBt+fMYVHbvoR_WA@mail.gmail.com>
	<4F245983.9000709@behnel.de>
	<CAKGHGPS=ON0u9YJPe=-pLMcJ+-ttR4teP13Ddb7oN65YhOtw2Q@mail.gmail.com>
Message-ID: <4F25094D.9050207@behnel.de>

Vitja Makarov, 28.01.2012 21:41:
> 2012/1/29 Stefan Behnel:
>> Vitja Makarov, 28.01.2012 20:58:
>>> 2012/1/28 mark florisson:
>>>> On 28 January 2012 19:41, Vitja Makarov wrote:
>>>>> 2012/1/28 Stefan Behnel:
>>>>>> Here's a general take on a code object cache for exception propagation.
>>>>>>
>>>>>> https://github.com/scoder/cython/commit/ad18e0208
>>>>>>
>>>>>> When I raise an exception in test code that propagates through a Python
>>>>>> call hierarchy of four functions before being caught, the cache gives me
>>>>>> something like a 2x speedup in total. Not bad. When I do the same for cdef
>>>>>> functions, it's more like 4-5x.
>>>>>>
>>>>>> The main idea is to cache the objects in a reallocable C array and bisect
>>>>>> into it based on the C code "__LINE__" of the exception, which should be
>>>>>> unique enough for a given module.
>>>>>>
>>>>>> It's a global cache that doesn't limit the lifetime of code objects  (well,
>>>>>> up to the lifetime of the module, obviously). I don't know if that's a
>>>>>> problem because the number of code objects is only bounded by the number of
>>>>>> exception origination points in the C source code, which is usually quite
>>>>>> large. However, only a tiny fraction of those will ever raise or propagate
>>>>>> an exception in practice, so the real number of cached code objects will be
>>>>>> substantially smaller.
>>>>>>
>>>>>> Maybe thorough test suites with lots of failure testing would notice a
>>>>>> difference in memory consumption, even though a single code objects isn't
>>>>>> all that large either...
>>>>>
>>>>> We already have --no-c-in-traceback flag that disables C line numbers
>>>>> in traceback. What's about enabling it by default?
>>>>>
>>>> I'm quite attached to that feature actually :), it would be pretty
>>>> annoying to disable that flag every time. And what would disabling
>>>> that option gain, as the current code still formats the filename and
>>>> function name.
>>>
>>> It's rather useful for developers or debugging. Most of the people
>>> don't need it.
>>
>> Not untrue. However, at least a majority of developers should be able to
>> make use of it when it's there, and code is several times more often built
>> for testing and debugging than for production. So I consider it a virtue
>> that it's on by default.
>>
>>
>>> Here is simple benchmark:
>>> # upstream/master: 6.38ms
>>> # upstream/master (no-c-in-traceback): 3.07ms
>>> # scoder/master: 1.31ms
>>> def foo():
>>>     raise ValueError
>>>
>>> def testit():
>>>     cdef int i
>>>     for i in range(10000):
>>>         try:
>>>             foo()
>>>         except:
>>>             pass
>>>
>>> Stefan's branch wins but:
>>>  - there is only one item in the cache and it's always hit
>>
>> Even if there were substantially more, binary search is so fast you'd
>> hardly notice the difference.
> 
> Yes, I'm a little bit worried about insertions.

I know, that's O(n), but it only strikes when a new exception is raised or
propagated from a code line that has never raised an exception before. That
makes it *very* unlikely that it hits a performance critical spot.


> With --no-c-in-traceback python lineno should be used as a key.

Good call, I added that.

https://github.com/scoder/cython/commit/8b50da874#diff-0

That means that using this option additionally improves the caching
performance now because you get less code objects overall, at most one per
Cython source code line (as opposed to C source code line).

Stefan

From stefan_ml at behnel.de  Sun Jan 29 13:10:10 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 29 Jan 2012 13:10:10 +0100
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F245724.8070208@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CANg26EWoGQwGN9XaMyiv-tYXhHx2thQrNxyMG=ZnrcFHyTH+YA@mail.gmail.com>
	<4F245724.8070208@behnel.de>
Message-ID: <4F253722.4020707@behnel.de>

Stefan Behnel, 28.01.2012 21:14:
> mark florisson, 28.01.2012 20:07:
>> On 28 January 2012 18:38, Stefan Behnel wrote:
>>> Stefan Behnel, 27.01.2012 09:02:
>>>> any exception *propagation* is
>>>> still substantially slower than necessary, and that's a general issue.
>>>
>>> Here's a general take on a code object cache for exception propagation.
>>> [...]
>> Nice. I have a question, couldn't you save the frame object instead of
>> the code object?
> 
> Technically, yes. However, eventually, I'd like to make the CodeObject
> constant for the whole function and let CPython calculate the Python code
> source line based on the frame's "f_lasti" field when the line number is
> actually accessed.
> 
> For now, I wouldn't mind cashing the whole frame until the above
> optimisation gets implemented.

Actually, I was wrong. The frame object cannot be cached because it is
mutable and even gets modified when building the stack trace call
hierarchy. So caching the frame instance may have user visible side effects.

Stefan

From markflorisson88 at gmail.com  Sun Jan 29 13:30:50 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 29 Jan 2012 12:30:50 +0000
Subject: [Cython] AddTraceback() slows down generators
In-Reply-To: <4F253722.4020707@behnel.de>
References: <4F1B0902.1050903@behnel.de>
	<CADiQ+QC3eHm5ogW=coU9JXLHrQ8qRHXnkqVFDtrFoGssGwERoQ@mail.gmail.com>
	<4F21B163.2080006@behnel.de>
	<CAKGHGPQM=M1Jd-46U2VXdDEkpqVVvsV2APbRpGZJk8M8d5uN_g@mail.gmail.com>
	<4F21BE46.5050509@behnel.de> <4F225A10.6020709@behnel.de>
	<4F2440A1.7050505@behnel.de>
	<CANg26EWoGQwGN9XaMyiv-tYXhHx2thQrNxyMG=ZnrcFHyTH+YA@mail.gmail.com>
	<4F245724.8070208@behnel.de> <4F253722.4020707@behnel.de>
Message-ID: <CANg26EXwR7xtGMUrsvjDq08omRV3fkN2C1pM56=ZW=brxGktEg@mail.gmail.com>

On 29 January 2012 12:10, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 28.01.2012 21:14:
>> mark florisson, 28.01.2012 20:07:
>>> On 28 January 2012 18:38, Stefan Behnel wrote:
>>>> Stefan Behnel, 27.01.2012 09:02:
>>>>> any exception *propagation* is
>>>>> still substantially slower than necessary, and that's a general issue.
>>>>
>>>> Here's a general take on a code object cache for exception propagation.
>>>> [...]
>>> Nice. I have a question, couldn't you save the frame object instead of
>>> the code object?
>>
>> Technically, yes. However, eventually, I'd like to make the CodeObject
>> constant for the whole function and let CPython calculate the Python code
>> source line based on the frame's "f_lasti" field when the line number is
>> actually accessed.
>>
>> For now, I wouldn't mind cashing the whole frame until the above
>> optimisation gets implemented.
>
> Actually, I was wrong. The frame object cannot be cached because it is
> mutable and even gets modified when building the stack trace call
> hierarchy. So caching the frame instance may have user visible side effects.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Ah right, the f_back attribute?

From vitja.makarov at gmail.com  Sun Jan 29 15:10:03 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 29 Jan 2012 18:10:03 +0400
Subject: [Cython] Bug in multiplied tuple optimization
Message-ID: <CAKGHGPQkuSWhoDUuV++TcSsGQw4op+hcf4bxRd2GEeevfvm+dg@mail.gmail.com>

Investigating sage-tests segfaults I found that this code causes sigsegv:

def foo():
    return (0,) * len('abc')
foo()

-- 
vitja.

From wesmckinn at gmail.com  Sun Jan 29 18:38:21 2012
From: wesmckinn at gmail.com (Wes McKinney)
Date: Sun, 29 Jan 2012 12:38:21 -0500
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <4F225509.3070605@behnel.de>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<4F225509.3070605@behnel.de>
Message-ID: <CAJPUwMDpK6-ek-T-0ycUykLasJxwQWwGC7qf=6zH+=BrjrUTjw@mail.gmail.com>

On Fri, Jan 27, 2012 at 2:40 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Wes McKinney, 26.01.2012 18:56:
>> Just wanted to bring this issue to your guys' attention in case you
>> knew what was responsible for this:
>>
>> https://github.com/ipython/ipython/issues/1317#issuecomment-3652550
>>
>> I traced down the problem (with git bisect) to a seemingly innocuous
>> commit referenced in that GitHub thread. The issue seemed to only
>> present itself in IPython, so likely there was some problem with
>> inspecting the Cython frames for giving context around the full
>> traceback.
>
> That's not impossible. Traceback frames behave differently in Cython for
> two reasons: a) because they are only being constructed after the fact in
> an error handling case, not for normal functions, and b) because Cython
> doesn't have real code objects for inspection and fakes them in a rather
> ad-hoc way. For example, there is currently no function signature
> information in them, and line number computation doesn't use the normal
> CPython way that matches the byte code position with a compressed line
> table. Instead, Cython creates a new code object for a given line on the
> fly and just pretends that the function starts there. This usually works
> well enough for a traceback, but this kind of differences makes it quite
> possible that IPython makes assumptions about inspection here that Cython
> doesn't meet.
>
> In another thread ("AddTraceback() slows down generators"), I was proposing
> to cache the code object for a given function. That would then imply
> providing a (fake) byte code position map as well and would make it easier
> to provide static signature information, thus improving the overall
> compatibility with the code objects that CPython creates.
>
> Note that it's unclear without further investigation if the problem you ran
> into really has to do with these internal details. I'm just raising a
> possible explanation here.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

My curiosity about this issue was overwhelming so I profiled and
tracked down what's going on. Basically looks like a bug in IPython
and the fact that it appeared then disappeared depending on the Cython
version was completely idiosyncratic. Should have known since the
Cython commit that git bisect yielded was so innocuous. More on this
here:

https://github.com/ipython/ipython/issues/1317

- Wes

From fperez.net at gmail.com  Sun Jan 29 19:35:58 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 29 Jan 2012 10:35:58 -0800
Subject: [Cython] Slow traceback reconstruction in IPython between
 0.15.1 and master
In-Reply-To: <CAJPUwMDpK6-ek-T-0ycUykLasJxwQWwGC7qf=6zH+=BrjrUTjw@mail.gmail.com>
References: <CAJPUwMAtgRGzZ-0TK-02G6XLuGjkutu7F_8e88S_owqO+2mwaQ@mail.gmail.com>
	<4F225509.3070605@behnel.de>
	<CAJPUwMDpK6-ek-T-0ycUykLasJxwQWwGC7qf=6zH+=BrjrUTjw@mail.gmail.com>
Message-ID: <CAHAreOq_fL5+jFLAaHge9SDnr3g3jtOkAV9v-zk_=XbB3XE++A@mail.gmail.com>

On Sun, Jan 29, 2012 at 9:38 AM, Wes McKinney <wesmckinn at gmail.com> wrote:
> More on this
> here:
>
> https://github.com/ipython/ipython/issues/1317

Thanks for your persistence!  For those on the cython list, it should
be fixed soon, Thomas already has a PR up.

Cheers,

f

From dalcinl at gmail.com  Mon Jan 30 22:03:03 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 30 Jan 2012 18:03:03 -0300
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and Cython's
	sizeof checks
Message-ID: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>

I'm testing my code with numpy-dev. They are trying to discourage use
of deprecated APIs, this includes direct access to the ndarray struct.
In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
to the C compiler (or #define it before including NumPy headers).

However, they have implemented this feature by exposing the ndarray
type with just the Python object header:
https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695

Obviously, this interact bad with Cython's sizeof check, I'm getting
this runtime warning:

build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
RuntimeWarning: numpy.ndarray size changed, may indicate binary
incompatibility

I think there is nothing Cython can do about this (other than
special-casing NumPy to disable this VERY useful warning).

 I've tried the patch below with success, but I'm not convinced...
Does any of you have a suggestion for NumPy folks about how to improve
this?


diff --git a/numpy/core/include/numpy/ndarraytypes.h
b/numpy/core/include/numpy/ndarraytypes.h
index 0288272..1fcbf52 100644
--- a/numpy/core/include/numpy/ndarraytypes.h
+++ b/numpy/core/include/numpy/ndarraytypes.h
@@ -695,6 +695,7 @@ typedef struct tagPyArrayObject_fields {
 #ifdef NPY_NO_DEPRECATED_API
 typedef struct tagPyArrayObject {
         PyObject_HEAD
+        char _npy_array_fields[sizeof(PyArrayObject_fields)-sizeof(PyObject)];
 } PyArrayObject;
 #else
 /*

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Tue Jan 31 03:12:43 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 30 Jan 2012 18:12:43 -0800
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <4F23109E.3030203@behnel.de>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
Message-ID: <CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>

On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 27.01.2012 21:03:
>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>> mark florisson, 27.01.2012 17:30:
>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com> ?wrote:
>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>
>>>>>> How can I patch Cython such that some more information on the location
>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>> message
>>>>>> ? ?Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>> object has no attribute '_modulus' in ?ignored
>>>>>>
>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>> responsible for printing the warning?
>>>>>>
>>>>>> Best regards,
>>>>>> Simon
>>>>>
>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>> typically two reasons for unraisable exceptions:
>>>>>
>>>>> ? ? 1) as Robert mentioned, a function that does not allow propagation
>>>>> of exceptions, e.g.
>>>>>
>>>>> ? ? ? ? cdef int func():
>>>>> ? ? ? ? ? ? raise Exception
>>>>>
>>>>> ? ? ? ? Here there is no way to propagate the raised exception, so
>>>>> instead one should write something like
>>>>>
>>>>> ? ? ? ? ? ? cdef int func() except -1: ...
>>>>>
>>>>> ? ? ? ? Alternatively one may use 'except *' in case there is no error
>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>> "-1 may or may not indicate an error".
>>>>>
>>>>> ? ? 2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>
>>>>> For functions the right thing is to add an except clause, for
>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>
>>>>> ? ? try:
>>>>> ? ? ? ? ...
>>>>> ? ? except:
>>>>> ? ? ? ? traceback.print_exc()
>>>>>
>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>
>>>> Actually, I don't see why the default is to write unraisable
>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>> and have callers do the check (i.e. make it implicitly "except *").
>>
>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>
>> And for floating point, we could make our own NaN -- that's obscure enough
>> that it could probably be made "except cython.cython_exception_nan" by
>> default, not "except? cython.cython_exception_nan".
>
> The problem with that is that we can't be sure that Cython will be the only
> caller. So exceptions may still not propagate in cases, and users will have
> to know about these "obscure" values and that they must deal with them
> manually then.
>
> You could add that we'd just have to disable this when user code takes a
> pointer from a function, but then, how many rules are there that users will
> have to learn and remember after such a change? And what's that for a
> language that changes the calling semantics of a function because way down
> in the code someone happens to take a pointer to it?
>
>
>>>> Was this not implemented because Cython only knows whether functions
>>>> may propagate exceptions at code generation time by looking at the
>>>> presence of an error label?
>>>> Maybe it could keep code insertion points around for every call to
>>>> such a potential function and if the function uses the error label
>>>> have the caller perform the check? Although I do forsee problems for
>>>> external such functions... maybe Cython could have it's own
>>>> threadstate regardless of the GIL which would indicate whether an
>>>> error has occurred? e.g. CyErr_Occurred()?
>>>
>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>> default.
>>
>> Still,
>
> I wasn't really advocating this behaviour, just indicating that it's hard
> to do "better", because this "better" isn't all that clear. It's also not
> "better" for all code, which means that we get from one trade-off to
> another, while breaking existing code at the same time. Not exactly
> paradise on either side of the tunnel.

I still feel like we're stuck in the wrong default. I'd rather require
more work to interact with C libraries than require more work to
convert innocent-looking Python to Cython.

> One example that keeps popping up in my mind is callback functions that
> cannot propagate errors, at least not the CPython way. I have a couple of
> those in lxml, even some returning void. So I wrap their code in a bare
> try-except and when an exception strikes, I set a C error flag to tell the
> C library that something went wrong and return normally. No Python code
> outside of the try block. But Cython still generates code for unraisable
> errors. Why? Because the internal code that handles the bare except clause
> may fail and raise an exception. How about that?
>
>
>> the need to explicitly declare "except *" keeps coming up again and
>> again, and is really a blemish on the usability of Cython. When teaching
>> people Cython, then it's really irritating to have to follow "all you need
>> to do is add some 'cdef' and some types" with "and then you need to
>> remember to say "except *", or you're in deep trouble". Cython sort of
>> looks very elegant until that point...
>
> I know what this feels like. The problem is that these things *are* complex.

Yes. We've been wrestling with this issue almost since Cython's inception...

I like Mark's two-function idea, with the caveat that f(bad_argument)
now behaves quite differently than (&f)[0](bad_argument) for even more
obscure reasons. But it may be the way to go.

The other option is to embed the error behavior into the signature and
require casts to explicitly go from one to the other. This would
probably require a notation for never raising an exception (e.g.
"except -"). Cdef public or api functions could require an except
declaration (positive or negative), ordinary cdef functions would be
"except *" by default, and cdef extern functions would be "except -"
by default.

Ideally, the default would not just be "except *" but "except
cython.error_value?" and a case could be made for acquiring the GIL to
check in the exceptional case that error_value is returned, or the
information could be passed by checking a bit on some thread-local
Cython global (not sure what the performance impact would be here).

A warning whenever WriteUnraisable is used could be handy too, but how
to handle Stefan's example where the bare except clause could raise an
exception?

>> Long-term we should change CPython to make sure that PyErr_Occurred doesn't
>> need the GIL :-) (there's really no reason it should need to go beyond
>> checking a thread-local variable).
>
> I always wondered about that, too. Still, "long-term" here basically means
> "when all current CPython versions that work like this are out of use",
> because we cannot base language semantics on specific runtime CPython versions.

+1

- Robert

From robertwb at math.washington.edu  Tue Jan 31 03:19:08 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 30 Jan 2012 18:19:08 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
Message-ID: <CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>

On Sat, Jan 28, 2012 at 8:05 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/26 Jason Grout <jason-sage at creativetrax.com>:
>> On 1/25/12 11:39 AM, Robert Bradshaw wrote:
>>>
>>> install
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>> by downloading it and running "sage -i cython-devel.spkg"
>>
>>
>>
>> In fact, you could just do
>>
>> sage -i
>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>
>> and Sage will (at least, should) download it for you, so that's even one
>> less step!
>>
>> Jason
>>
>
> Thanks for detailed instruction! I've successfully built it.
>
> "sage -t -gdb ./...." doesn't work, is that a bug?
>
> vitja at mchome:~/Downloads/sage-4.8$ ./sage ?-t -gdb
> devel/sage/sage/combinat/sf/macdonald.py
> sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
> ********************************************************************************
> Type r at the (gdb) prompt to run the doctests.
> Type bt if there is a crash to see a traceback.
> ********************************************************************************
> gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
> starting cmd gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
> ImportError: No module named site
> ? ? ? ? [0.2 s]
>
> ----------------------------------------------------------------------
> The following tests failed:
>
>
> ? ? ? ?sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
> Total time for all tests: 0.2 seconds

Yes, that's a bug.

> I've found another way to run tests (using sage -sh and then direct
> python ~/.sage/tmp/...py)
>
> So I found one of the problems. Here is minimal cython example:
>
> def foo(values):
> ? ?return (0,)*len(values)
> foo([1,2,3])
>
> len(values) somehow is passed as an integer to PyObject_Multiply()

Yeah, that's a bug too :).

From markflorisson88 at gmail.com  Tue Jan 31 15:29:27 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 31 Jan 2012 14:29:27 +0000
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
Message-ID: <CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>

On 30 January 2012 21:03, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> I'm testing my code with numpy-dev. They are trying to discourage use
> of deprecated APIs, this includes direct access to the ndarray struct.
> In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
> to the C compiler (or #define it before including NumPy headers).
>
> However, they have implemented this feature by exposing the ndarray
> type with just the Python object header:
> https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695
>
> Obviously, this interact bad with Cython's sizeof check, I'm getting
> this runtime warning:
>
> build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
> RuntimeWarning: numpy.ndarray size changed, may indicate binary
> incompatibility
>
> I think there is nothing Cython can do about this (other than
> special-casing NumPy to disable this VERY useful warning).

Weird, shouldn't you be getting an error? Because the size of the
PyArrayObject should be less than what Cython expects.

> ?I've tried the patch below with success, but I'm not convinced...
> Does any of you have a suggestion for NumPy folks about how to improve
> this?
>

I'm not sure this should be fixed in NumPy. Their entire point is that
people shouldn't use those attributes directly. I think numpy.pxd
should be fixed, but the problem is that some attributes might be used
in user code (especially shape), and we still want that to work in
nogil mode. As such, I'm not sure what the best way of fixing it is,
without special casing these attributes in the compiler directly.
Maybe Dag will have some thoughts about this.

> diff --git a/numpy/core/include/numpy/ndarraytypes.h
> b/numpy/core/include/numpy/ndarraytypes.h
> index 0288272..1fcbf52 100644
> --- a/numpy/core/include/numpy/ndarraytypes.h
> +++ b/numpy/core/include/numpy/ndarraytypes.h
> @@ -695,6 +695,7 @@ typedef struct tagPyArrayObject_fields {
> ?#ifdef NPY_NO_DEPRECATED_API
> ?typedef struct tagPyArrayObject {
> ? ? ? ? PyObject_HEAD
> + ? ? ? ?char _npy_array_fields[sizeof(PyArrayObject_fields)-sizeof(PyObject)];
> ?} PyArrayObject;
> ?#else
> ?/*
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Tue Jan 31 16:40:19 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 31 Jan 2012 16:40:19 +0100
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
	<CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
Message-ID: <4F280B63.1070903@astro.uio.no>

On 01/31/2012 03:29 PM, mark florisson wrote:
> On 30 January 2012 21:03, Lisandro Dalcin<dalcinl at gmail.com>  wrote:
>> I'm testing my code with numpy-dev. They are trying to discourage use
>> of deprecated APIs, this includes direct access to the ndarray struct.
>> In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
>> to the C compiler (or #define it before including NumPy headers).
>>
>> However, they have implemented this feature by exposing the ndarray
>> type with just the Python object header:
>> https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695
>>
>> Obviously, this interact bad with Cython's sizeof check, I'm getting
>> this runtime warning:
>>
>> build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
>> RuntimeWarning: numpy.ndarray size changed, may indicate binary
>> incompatibility
>>
>> I think there is nothing Cython can do about this (other than
>> special-casing NumPy to disable this VERY useful warning).

Hmm...but you can still recompile the Cython module and then don't get 
the warning, right?

We've already been through at least one such round. People tend to 
ignore it, or install warning filters...

If one does want a workaround, we don't have to special case NumPy as 
such -- I think it is marginally cleaner to add new obscure syntax which 
we only use in numpy.pxd:

     ctypedef class numpy.ndarray [object PyArrayObject nosizecheck]:

Or, if anybody bothers, a way to register and automatically run the 
functions NumPy provides for checking ABI compatability.

I don't think any changes should be done on the NumPy end.

>
> Weird, shouldn't you be getting an error? Because the size of the
> PyArrayObject should be less than what Cython expects.
>
>>   I've tried the patch below with success, but I'm not convinced...
>> Does any of you have a suggestion for NumPy folks about how to improve
>> this?
>>
>
> I'm not sure this should be fixed in NumPy. Their entire point is that
> people shouldn't use those attributes directly. I think numpy.pxd
> should be fixed, but the problem is that some attributes might be used
> in user code (especially shape), and we still want that to work in
> nogil mode. As such, I'm not sure what the best way of fixing it is,
> without special casing these attributes in the compiler directly.
> Maybe Dag will have some thoughts about this.

Well, we should definitely deprecate direct access to the PyArrayObject 
fields -- you can either use "cdef int[:]", or, if you use "cdef 
np.ndarray[int]", you should use "PyArray_SHAPE".

Problem is that a lot of tutorial material etc. encourages accessing the 
fields directly (my fault). But I think it just needs to happen in the 
user code.

  - Do we just remove the fields from numpy.pxd; or do we put in a 
very-special-case in other to give deprecation warnings for a release? 
(It'd be a very special transform stage, but only for one release and 
then we simply remove both the transform stage and the fields from 
numpy.pxd)

  - Do we deprecate the whole "cdef np.ndarray[int]" syntax in favour of 
"cdef int[:]"? My hunch is against it, as that would render a lot of 
code using deprecated features, but it would "solve" the size warning issue.

Dag Sverre

From dalcinl at gmail.com  Tue Jan 31 16:36:00 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 31 Jan 2012 12:36:00 -0300
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
	<CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
Message-ID: <CAEcYPwAkSgZDZhONmXjm6PZnnB-FN6wEyZy8eNPMs7KwOQCQSw@mail.gmail.com>

On 31 January 2012 11:29, mark florisson <markflorisson88 at gmail.com> wrote:
> On 30 January 2012 21:03, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> I think there is nothing Cython can do about this (other than
>> special-casing NumPy to disable this VERY useful warning).
>
> Weird, shouldn't you be getting an error? Because the size of the
> PyArrayObject should be less than what Cython expects.
>

Well, as long as your code does not access the structure fiels,
everything is OK.

>
> I'm not sure this should be fixed in NumPy. Their entire point is that
> people shouldn't use those attributes directly.
>

This makes me think about the all those arguments about the beauties
of "private:" in C++ (and others OOP langs). IMHO, if they way to
discourage access to these slots, they should declare them with some
weird names.

> I think numpy.pxd
> should be fixed, but the problem is that some attributes might be used
> in user code (especially shape), and we still want that to work in
> nogil mode.

You can still use PyArray_DATA(), PyArray_DIMS(), etc...

> As such, I'm not sure what the best way of fixing it is,
> without special casing these attributes in the compiler directly.
> Maybe Dag will have some thoughts about this.
>

I'v just noticed now NumPy #defines NPY_SIZEOF_PYARRAYOBJECT, that
could serve as a workaround for the sizeof check:
https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L707


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Tue Jan 31 17:30:11 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 31 Jan 2012 16:30:11 +0000
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
	<CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
Message-ID: <CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>

On 31 January 2012 02:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>>> mark florisson, 27.01.2012 17:30:
>>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com> ?wrote:
>>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>>
>>>>>>> How can I patch Cython such that some more information on the location
>>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>>> message
>>>>>>> ? ?Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>>> object has no attribute '_modulus' in ?ignored
>>>>>>>
>>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>>> responsible for printing the warning?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Simon
>>>>>>
>>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>>> typically two reasons for unraisable exceptions:
>>>>>>
>>>>>> ? ? 1) as Robert mentioned, a function that does not allow propagation
>>>>>> of exceptions, e.g.
>>>>>>
>>>>>> ? ? ? ? cdef int func():
>>>>>> ? ? ? ? ? ? raise Exception
>>>>>>
>>>>>> ? ? ? ? Here there is no way to propagate the raised exception, so
>>>>>> instead one should write something like
>>>>>>
>>>>>> ? ? ? ? ? ? cdef int func() except -1: ...
>>>>>>
>>>>>> ? ? ? ? Alternatively one may use 'except *' in case there is no error
>>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>>> "-1 may or may not indicate an error".
>>>>>>
>>>>>> ? ? 2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>>
>>>>>> For functions the right thing is to add an except clause, for
>>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>>
>>>>>> ? ? try:
>>>>>> ? ? ? ? ...
>>>>>> ? ? except:
>>>>>> ? ? ? ? traceback.print_exc()
>>>>>>
>>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>>
>>>>> Actually, I don't see why the default is to write unraisable
>>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>>> and have callers do the check (i.e. make it implicitly "except *").
>>>
>>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>>
>>> And for floating point, we could make our own NaN -- that's obscure enough
>>> that it could probably be made "except cython.cython_exception_nan" by
>>> default, not "except? cython.cython_exception_nan".
>>
>> The problem with that is that we can't be sure that Cython will be the only
>> caller. So exceptions may still not propagate in cases, and users will have
>> to know about these "obscure" values and that they must deal with them
>> manually then.
>>
>> You could add that we'd just have to disable this when user code takes a
>> pointer from a function, but then, how many rules are there that users will
>> have to learn and remember after such a change? And what's that for a
>> language that changes the calling semantics of a function because way down
>> in the code someone happens to take a pointer to it?
>>
>>
>>>>> Was this not implemented because Cython only knows whether functions
>>>>> may propagate exceptions at code generation time by looking at the
>>>>> presence of an error label?
>>>>> Maybe it could keep code insertion points around for every call to
>>>>> such a potential function and if the function uses the error label
>>>>> have the caller perform the check? Although I do forsee problems for
>>>>> external such functions... maybe Cython could have it's own
>>>>> threadstate regardless of the GIL which would indicate whether an
>>>>> error has occurred? e.g. CyErr_Occurred()?
>>>>
>>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>>> default.
>>>
>>> Still,
>>
>> I wasn't really advocating this behaviour, just indicating that it's hard
>> to do "better", because this "better" isn't all that clear. It's also not
>> "better" for all code, which means that we get from one trade-off to
>> another, while breaking existing code at the same time. Not exactly
>> paradise on either side of the tunnel.
>
> I still feel like we're stuck in the wrong default. I'd rather require
> more work to interact with C libraries than require more work to
> convert innocent-looking Python to Cython.
>
>> One example that keeps popping up in my mind is callback functions that
>> cannot propagate errors, at least not the CPython way. I have a couple of
>> those in lxml, even some returning void. So I wrap their code in a bare
>> try-except and when an exception strikes, I set a C error flag to tell the
>> C library that something went wrong and return normally. No Python code
>> outside of the try block. But Cython still generates code for unraisable
>> errors. Why? Because the internal code that handles the bare except clause
>> may fail and raise an exception. How about that?
>>
>>
>>> the need to explicitly declare "except *" keeps coming up again and
>>> again, and is really a blemish on the usability of Cython. When teaching
>>> people Cython, then it's really irritating to have to follow "all you need
>>> to do is add some 'cdef' and some types" with "and then you need to
>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>> looks very elegant until that point...
>>
>> I know what this feels like. The problem is that these things *are* complex.
>
> Yes. We've been wrestling with this issue almost since Cython's inception...
>
> I like Mark's two-function idea, with the caveat that f(bad_argument)
> now behaves quite differently than (&f)[0](bad_argument) for even more
> obscure reasons. But it may be the way to go.
>
> The other option is to embed the error behavior into the signature and
> require casts to explicitly go from one to the other. This would
> probably require a notation for never raising an exception (e.g.
> "except -"). Cdef public or api functions could require an except
> declaration (positive or negative), ordinary cdef functions would be
> "except *" by default, and cdef extern functions would be "except -"
> by default.

Only except * and except ? have ever made some sense to me. Except +
is the most mysterious syntax ever, imho it should have been 'except
cpperror' or something. And when you try to search for "except +" or
"except *" etc on docs.cython.org it doesn't find anything, which
makes it hard for people reading the code and unfamiliar with the
syntax to figure out what it means. In general I also think decorators
would have been clearer when defining such functions. Let's please not
introduce more weird syntax.

In any event I don't see why we'd want 'except -', as we're trying to
get rid of the except clause. So you can still get your old behaviour
for function pointers by not using the except clause and having it
write unraisable exceptions in the function, but in Cython space you'd
simply get better semantics (that is, propagating exceptions).

> Ideally, the default would not just be "except *" but "except
> cython.error_value?" and a case could be made for acquiring the GIL to
> check in the exceptional case that error_value is returned, or the
> information could be passed by checking a bit on some thread-local
> Cython global (not sure what the performance impact would be here).
>
> A warning whenever WriteUnraisable is used could be handy too, but how
> to handle Stefan's example where the bare except clause could raise an
> exception?
>
>>> Long-term we should change CPython to make sure that PyErr_Occurred doesn't
>>> need the GIL :-) (there's really no reason it should need to go beyond
>>> checking a thread-local variable).
>>
>> I always wondered about that, too. Still, "long-term" here basically means
>> "when all current CPython versions that work like this are out of use",
>> because we cannot base language semantics on specific runtime CPython versions.
>
> +1
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Tue Jan 31 18:05:52 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 31 Jan 2012 18:05:52 +0100
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>	<4F22D7A2.1050806@behnel.de>
	<4F230312.9050506@astro.uio.no>	<4F23109E.3030203@behnel.de>	<CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
	<CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
Message-ID: <4F281F70.8030604@astro.uio.no>

On 01/31/2012 05:30 PM, mark florisson wrote:
> On 31 January 2012 02:12, Robert Bradshaw<robertwb at math.washington.edu>  wrote:
>> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>>> the need to explicitly declare "except *" keeps coming up again and
>>>> again, and is really a blemish on the usability of Cython. When teaching
>>>> people Cython, then it's really irritating to have to follow "all you need
>>>> to do is add some 'cdef' and some types" with "and then you need to
>>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>>> looks very elegant until that point...
>>>
>>> I know what this feels like. The problem is that these things *are* complex.
>>
>> Yes. We've been wrestling with this issue almost since Cython's inception...
>>
>> I like Mark's two-function idea, with the caveat that f(bad_argument)
>> now behaves quite differently than (&f)[0](bad_argument) for even more
>> obscure reasons. But it may be the way to go.

Keep in mind that we talked about doing more or less the same to get rid 
of "nogil"/"with gil". I believe that should be treated in the same 
discussion. E.g., when doing

&f

you could get a version which acquires the GIL and which has a C ABI, 
while calling f from Cython would step over the GIL-checking and has a 
Cython-defined ABI.

Dag

From stefan_ml at behnel.de  Tue Jan 31 18:17:01 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 31 Jan 2012 18:17:01 +0100
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
	<CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
	<CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
Message-ID: <4F28220D.5050608@behnel.de>

mark florisson, 31.01.2012 17:30:
> Only except * and except ? have ever made some sense to me. Except +
> is the most mysterious syntax ever, imho it should have been 'except
> cpperror' or something. And when you try to search for "except +" or
> "except *" etc on docs.cython.org it doesn't find anything, which
> makes it hard for people reading the code and unfamiliar with the
> syntax to figure out what it means. In general I also think decorators
> would have been clearer when defining such functions. Let's please not
> introduce more weird syntax.

+1

Stefan

From markflorisson88 at gmail.com  Tue Jan 31 21:53:10 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 31 Jan 2012 20:53:10 +0000
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <4F280B63.1070903@astro.uio.no>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
	<CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
	<4F280B63.1070903@astro.uio.no>
Message-ID: <CANg26EU69KozScA8qwg-dzbN+0m6Uf5_-QACog=CebP0LCyv-A@mail.gmail.com>

On 31 January 2012 15:40, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 01/31/2012 03:29 PM, mark florisson wrote:
>>
>> On 30 January 2012 21:03, Lisandro Dalcin<dalcinl at gmail.com> ?wrote:
>>>
>>> I'm testing my code with numpy-dev. They are trying to discourage use
>>> of deprecated APIs, this includes direct access to the ndarray struct.
>>> In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
>>> to the C compiler (or #define it before including NumPy headers).
>>>
>>> However, they have implemented this feature by exposing the ndarray
>>> type with just the Python object header:
>>>
>>> https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695
>>>
>>> Obviously, this interact bad with Cython's sizeof check, I'm getting
>>> this runtime warning:
>>>
>>> build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
>>> RuntimeWarning: numpy.ndarray size changed, may indicate binary
>>> incompatibility
>>>
>>> I think there is nothing Cython can do about this (other than
>>> special-casing NumPy to disable this VERY useful warning).
>
>
> Hmm...but you can still recompile the Cython module and then don't get the
> warning, right?
>
> We've already been through at least one such round. People tend to ignore
> it, or install warning filters...
>
> If one does want a workaround, we don't have to special case NumPy as such
> -- I think it is marginally cleaner to add new obscure syntax which we only
> use in numpy.pxd:
>
> ? ?ctypedef class numpy.ndarray [object PyArrayObject nosizecheck]:
>
> Or, if anybody bothers, a way to register and automatically run the
> functions NumPy provides for checking ABI compatability.
>
> I don't think any changes should be done on the NumPy end.
>

I really don't care about the warning, more about the possibility of
numpy rearranging/removing or adding to its private fields (I guess
they won't be doing that anytime soon, but prudence is a virtue). I
suppose we would notice any changes soon enough though, as it would
break all the code.

>>
>> Weird, shouldn't you be getting an error? Because the size of the
>> PyArrayObject should be less than what Cython expects.
>>
>>> ?I've tried the patch below with success, but I'm not convinced...
>>> Does any of you have a suggestion for NumPy folks about how to improve
>>> this?
>>>
>>
>> I'm not sure this should be fixed in NumPy. Their entire point is that
>> people shouldn't use those attributes directly. I think numpy.pxd
>> should be fixed, but the problem is that some attributes might be used
>> in user code (especially shape), and we still want that to work in
>> nogil mode. As such, I'm not sure what the best way of fixing it is,
>> without special casing these attributes in the compiler directly.
>> Maybe Dag will have some thoughts about this.
>
>
> Well, we should definitely deprecate direct access to the PyArrayObject
> fields -- you can either use "cdef int[:]", or, if you use "cdef
> np.ndarray[int]", you should use "PyArray_SHAPE".

Yeah. However, PyArray_SHAPE seems to be new in numpy 1.7. I also see
PyArray_BASE and PyArray_DESCR are commented out (because apparently
they may be NULL. Should we, for the sake of consistency, rename
'get_array_base' to PyArray_BASE in numpy.pxd?

And maybe we could provide our own implementation of PyArray_SHAPE,
which would be portable across numpy versions.

> Problem is that a lot of tutorial material etc. encourages accessing the
> fields directly (my fault). But I think it just needs to happen in the user
> code.
>
> ?- Do we just remove the fields from numpy.pxd; or do we put in a
> very-special-case in other to give deprecation warnings for a release? (It'd
> be a very special transform stage, but only for one release and then we
> simply remove both the transform stage and the fields from numpy.pxd)

That would be a good idea.

> ?- Do we deprecate the whole "cdef np.ndarray[int]" syntax in favour of
> "cdef int[:]"? My hunch is against it, as that would render a lot of code
> using deprecated features, but it would "solve" the size warning issue.

I think that's more for the long run. Memoryviews still behave
differently, as they coerce to memoryview objects instead of to the
original numpy array. So users can't simply adjust their declarations
and expect things to work.

Maybe we could allow users to install a hook for object coercion (e.g.
cython.view.set_object_coercion_hook(numpy.asarray))? The only problem
with that is a potentially large additional overhead, as re-aquiring a
memoryview would have to go through the buffer interface and would
have to re-parse the format string. Although that is currently the
same situation for memoryviews, it would be an easy hack to optimize
that and just compare the type pointers. I'm not yet sure how to make
that work across modules, however. Maybe if pointers don't match it
could compare all the fields of the struct, which would probably still
be cheaper than parsing buffer format strings and obtaining a buffer
view.

> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Tue Jan 31 22:11:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 31 Jan 2012 22:11:47 +0100
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <CANg26EU69KozScA8qwg-dzbN+0m6Uf5_-QACog=CebP0LCyv-A@mail.gmail.com>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>	<CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>	<4F280B63.1070903@astro.uio.no>
	<CANg26EU69KozScA8qwg-dzbN+0m6Uf5_-QACog=CebP0LCyv-A@mail.gmail.com>
Message-ID: <4F285913.1060106@astro.uio.no>

On 01/31/2012 09:53 PM, mark florisson wrote:
> On 31 January 2012 15:40, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 01/31/2012 03:29 PM, mark florisson wrote:
>>>
>>> On 30 January 2012 21:03, Lisandro Dalcin<dalcinl at gmail.com>    wrote:
>>>>
>>>> I'm testing my code with numpy-dev. They are trying to discourage use
>>>> of deprecated APIs, this includes direct access to the ndarray struct.
>>>> In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
>>>> to the C compiler (or #define it before including NumPy headers).
>>>>
>>>> However, they have implemented this feature by exposing the ndarray
>>>> type with just the Python object header:
>>>>
>>>> https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695
>>>>
>>>> Obviously, this interact bad with Cython's sizeof check, I'm getting
>>>> this runtime warning:
>>>>
>>>> build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
>>>> RuntimeWarning: numpy.ndarray size changed, may indicate binary
>>>> incompatibility
>>>>
>>>> I think there is nothing Cython can do about this (other than
>>>> special-casing NumPy to disable this VERY useful warning).
>>
>>
>> Hmm...but you can still recompile the Cython module and then don't get the
>> warning, right?
>>
>> We've already been through at least one such round. People tend to ignore
>> it, or install warning filters...
>>
>> If one does want a workaround, we don't have to special case NumPy as such
>> -- I think it is marginally cleaner to add new obscure syntax which we only
>> use in numpy.pxd:
>>
>>     ctypedef class numpy.ndarray [object PyArrayObject nosizecheck]:
>>
>> Or, if anybody bothers, a way to register and automatically run the
>> functions NumPy provides for checking ABI compatability.
>>
>> I don't think any changes should be done on the NumPy end.
>>
>
> I really don't care about the warning, more about the possibility of
> numpy rearranging/removing or adding to its private fields (I guess
> they won't be doing that anytime soon, but prudence is a virtue). I
> suppose we would notice any changes soon enough though, as it would
> break all the code.
>
>>>
>>> Weird, shouldn't you be getting an error? Because the size of the
>>> PyArrayObject should be less than what Cython expects.
>>>
>>>>   I've tried the patch below with success, but I'm not convinced...
>>>> Does any of you have a suggestion for NumPy folks about how to improve
>>>> this?
>>>>
>>>
>>> I'm not sure this should be fixed in NumPy. Their entire point is that
>>> people shouldn't use those attributes directly. I think numpy.pxd
>>> should be fixed, but the problem is that some attributes might be used
>>> in user code (especially shape), and we still want that to work in
>>> nogil mode. As such, I'm not sure what the best way of fixing it is,
>>> without special casing these attributes in the compiler directly.
>>> Maybe Dag will have some thoughts about this.
>>
>>
>> Well, we should definitely deprecate direct access to the PyArrayObject
>> fields -- you can either use "cdef int[:]", or, if you use "cdef
>> np.ndarray[int]", you should use "PyArray_SHAPE".
>
> Yeah. However, PyArray_SHAPE seems to be new in numpy 1.7. I also see
> PyArray_BASE and PyArray_DESCR are commented out (because apparently
> they may be NULL. Should we, for the sake of consistency, rename
> 'get_array_base' to PyArray_BASE in numpy.pxd?
>
> And maybe we could provide our own implementation of PyArray_SHAPE,
> which would be portable across numpy versions.

PyArray_DIMS

Dag

From markflorisson88 at gmail.com  Tue Jan 31 22:58:01 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 31 Jan 2012 21:58:01 +0000
Subject: [Cython] Upcoming issues with NumPy deprecated APIs and
 Cython's sizeof checks
In-Reply-To: <4F285913.1060106@astro.uio.no>
References: <CAEcYPwD+UFxjBUeQdymA=6xB1KwR+pkdAa=3R6d99fD5Y3pbFw@mail.gmail.com>
	<CANg26EUqQwLKtfXmYDG_ZNAJChOSpB9YrDa2j4iyFK6A8w5y+Q@mail.gmail.com>
	<4F280B63.1070903@astro.uio.no>
	<CANg26EU69KozScA8qwg-dzbN+0m6Uf5_-QACog=CebP0LCyv-A@mail.gmail.com>
	<4F285913.1060106@astro.uio.no>
Message-ID: <CANg26EVg1EjnQOodZvuRWEnGyHXEFk2nGeUdeq8FvYuc42SouQ@mail.gmail.com>

On 31 January 2012 21:11, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 01/31/2012 09:53 PM, mark florisson wrote:
>>
>> On 31 January 2012 15:40, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 01/31/2012 03:29 PM, mark florisson wrote:
>>>>
>>>>
>>>> On 30 January 2012 21:03, Lisandro Dalcin<dalcinl at gmail.com> ? ?wrote:
>>>>>
>>>>>
>>>>> I'm testing my code with numpy-dev. They are trying to discourage use
>>>>> of deprecated APIs, this includes direct access to the ndarray struct.
>>>>> In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API
>>>>> to the C compiler (or #define it before including NumPy headers).
>>>>>
>>>>> However, they have implemented this feature by exposing the ndarray
>>>>> type with just the Python object header:
>>>>>
>>>>>
>>>>> https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L695
>>>>>
>>>>> Obviously, this interact bad with Cython's sizeof check, I'm getting
>>>>> this runtime warning:
>>>>>
>>>>> build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64:
>>>>> RuntimeWarning: numpy.ndarray size changed, may indicate binary
>>>>> incompatibility
>>>>>
>>>>> I think there is nothing Cython can do about this (other than
>>>>> special-casing NumPy to disable this VERY useful warning).
>>>
>>>
>>>
>>> Hmm...but you can still recompile the Cython module and then don't get
>>> the
>>> warning, right?
>>>
>>> We've already been through at least one such round. People tend to ignore
>>> it, or install warning filters...
>>>
>>> If one does want a workaround, we don't have to special case NumPy as
>>> such
>>> -- I think it is marginally cleaner to add new obscure syntax which we
>>> only
>>> use in numpy.pxd:
>>>
>>> ? ?ctypedef class numpy.ndarray [object PyArrayObject nosizecheck]:
>>>
>>> Or, if anybody bothers, a way to register and automatically run the
>>> functions NumPy provides for checking ABI compatability.
>>>
>>> I don't think any changes should be done on the NumPy end.
>>>
>>
>> I really don't care about the warning, more about the possibility of
>> numpy rearranging/removing or adding to its private fields (I guess
>> they won't be doing that anytime soon, but prudence is a virtue). I
>> suppose we would notice any changes soon enough though, as it would
>> break all the code.
>>
>>>>
>>>> Weird, shouldn't you be getting an error? Because the size of the
>>>> PyArrayObject should be less than what Cython expects.
>>>>
>>>>> ?I've tried the patch below with success, but I'm not convinced...
>>>>> Does any of you have a suggestion for NumPy folks about how to improve
>>>>> this?
>>>>>
>>>>
>>>> I'm not sure this should be fixed in NumPy. Their entire point is that
>>>> people shouldn't use those attributes directly. I think numpy.pxd
>>>> should be fixed, but the problem is that some attributes might be used
>>>> in user code (especially shape), and we still want that to work in
>>>> nogil mode. As such, I'm not sure what the best way of fixing it is,
>>>> without special casing these attributes in the compiler directly.
>>>> Maybe Dag will have some thoughts about this.
>>>
>>>
>>>
>>> Well, we should definitely deprecate direct access to the PyArrayObject
>>> fields -- you can either use "cdef int[:]", or, if you use "cdef
>>> np.ndarray[int]", you should use "PyArray_SHAPE".
>>
>>
>> Yeah. However, PyArray_SHAPE seems to be new in numpy 1.7. I also see
>> PyArray_BASE and PyArray_DESCR are commented out (because apparently
>> they may be NULL. Should we, for the sake of consistency, rename
>> 'get_array_base' to PyArray_BASE in numpy.pxd?
>>
>> And maybe we could provide our own implementation of PyArray_SHAPE,
>> which would be portable across numpy versions.
>
>
> PyArray_DIMS

Ah, neat :)

> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel