From dalcinl at gmail.com  Fri Nov  1 13:55:35 2013
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Nov 2013 15:55:35 +0300
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
	in C code
In-Reply-To: <5272929A.7000302@behnel.de>
References: <5272929A.7000302@behnel.de>
Message-ID: <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>

On 31 October 2013 20:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I just came across this paper:
>
> http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf
>
> They describe an analysis tool that checks C code for bugs that exploit
> undefined behaviour and that are thus up to the mercy of compiler
> assumptions and "optimisations" to do the right thing or not. They made it
> available on github:
>
> https://github.com/xiw/stack/
>
> If anyone wants to take the time to set it up for checking some Cython
> generated code, I'd be interested to see if it finds something.
>

I tested with mpi4py and got 0 warnings. I'll run it on Cython's testsuite.


-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Fri Nov  1 14:16:04 2013
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Nov 2013 16:16:04 +0300
Subject: [Cython] Files with executable bit on
Message-ID: <CAEcYPwAYPtKx_XnpU3UB9rgZC8w2gU9hpgahT0_bs0WsWAgZJw@mail.gmail.com>

This is in branch 0.19.x. These files should not have the executable
bit on. Can any of you please fix them?

[dalcinl at macarena cython-dev]$ ls -al $(find Cython -name '*.py') | grep rwx
-rwxr-xr-x  1 dalcinl  staff    4286 Feb 14  2013
Cython/Build/BuildExecutable.py
-rwxr-xr-x  1 dalcinl  staff  420289 Oct 15 23:39 Cython/Compiler/ExprNodes.py
-rwxr-xr-x  1 dalcinl  staff  137439 Oct 15 23:39 Cython/Compiler/PyrexTypes.py

-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Fri Nov  1 14:52:48 2013
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Nov 2013 16:52:48 +0300
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
	in C code
In-Reply-To: <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
References: <5272929A.7000302@behnel.de>
 <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
Message-ID: <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>

On 1 November 2013 15:55, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 31 October 2013 20:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> I just came across this paper:
>>
>> http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf
>>
>> They describe an analysis tool that checks C code for bugs that exploit
>> undefined behaviour and that are thus up to the mercy of compiler
>> assumptions and "optimisations" to do the right thing or not. They made it
>> available on github:
>>
>> https://github.com/xiw/stack/
>>
>> If anyone wants to take the time to set it up for checking some Cython
>> generated code, I'd be interested to see if it finds something.
>>
>
> I tested with mpi4py and got 0 warnings. I'll run it on Cython's testsuite.
>
>

I got two warnings out of 'python setup.py build'


Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.17387.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Plex/Actions.17412.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Runtime/refnanny.17706.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Code.17659.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.17614.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Lexicon.17437.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Visitor.17577.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Scanning.17462.ll
Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Parsing.17489.ll
Generated 2 warnings, see pstack.txt for details.
[dalcinl at kw2060 build.bak]$ cat pstack.txt
---
bug: anti-dce
model: |
  %523 = icmp ne i64 %522, 0, !dbg !1139
  -->  false
  ************************************************************
  :
  %518 = load %struct._object** @PyExc_UnboundLocalError, align 8, !dbg !1140
  %519 = call %struct._object* (%struct._object*, i8*, ...)*
@PyErr_Format(%struct._object* %518, i8* getelementptr inbounds ([49 x
i8]* @.str100, i32 0, i32 0), i8* getelementptr inbounds ([6 x i8]*
@.str52, i32 0, i32 0)), !dbg !1140
  br label %605, !dbg !1143
stack:
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6651:0
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:1752:0
ncore: 1
core:
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6511:0
    - null pointer dereference
---
bug: anti-simplify
model: |
  %7699 = icmp ne %struct._object* %78, null, !dbg !4455
  -->  true
stack:
  - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:16304:0
ncore: 1
core:
  - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:30198:0
    - null pointer dereference



-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From robertwb at gmail.com  Fri Nov  1 17:05:43 2013
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 1 Nov 2013 09:05:43 -0700
Subject: [Cython] Files with executable bit on
In-Reply-To: <CAEcYPwAYPtKx_XnpU3UB9rgZC8w2gU9hpgahT0_bs0WsWAgZJw@mail.gmail.com>
References: <CAEcYPwAYPtKx_XnpU3UB9rgZC8w2gU9hpgahT0_bs0WsWAgZJw@mail.gmail.com>
Message-ID: <CADiQ+QAQ=GnFV64LfJJf6b=MRJ_8B6BrEJgf-QXaoqOV2eic8g@mail.gmail.com>

Done.

On Fri, Nov 1, 2013 at 6:16 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> This is in branch 0.19.x. These files should not have the executable
> bit on. Can any of you please fix them?
>
> [dalcinl at macarena cython-dev]$ ls -al $(find Cython -name '*.py') | grep rwx
> -rwxr-xr-x  1 dalcinl  staff    4286 Feb 14  2013
> Cython/Build/BuildExecutable.py
> -rwxr-xr-x  1 dalcinl  staff  420289 Oct 15 23:39 Cython/Compiler/ExprNodes.py
> -rwxr-xr-x  1 dalcinl  staff  137439 Oct 15 23:39 Cython/Compiler/PyrexTypes.py
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (UNL/CONICET)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1016)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Nov  1 19:53:37 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 01 Nov 2013 19:53:37 +0100
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
 in C code
In-Reply-To: <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>
References: <5272929A.7000302@behnel.de>
 <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
 <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>
Message-ID: <5273F8B1.9020706@behnel.de>

Lisandro Dalcin, 01.11.2013 14:52:
> On 1 November 2013 15:55, Lisandro Dalcin wrote:
>> On 31 October 2013 20:25, Stefan Behnel wrote:
>>> I just came across this paper:
>>>
>>> http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf
>>>
>>> They describe an analysis tool that checks C code for bugs that exploit
>>> undefined behaviour and that are thus up to the mercy of compiler
>>> assumptions and "optimisations" to do the right thing or not. They made it
>>> available on github:
>>>
>>> https://github.com/xiw/stack/
>>>
>>> If anyone wants to take the time to set it up for checking some Cython
>>> generated code, I'd be interested to see if it finds something.
>>
>> I tested with mpi4py and got 0 warnings. I'll run it on Cython's testsuite.
> 
> I got two warnings out of 'python setup.py build'

Thanks for testing!


> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.17387.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Plex/Actions.17412.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Runtime/refnanny.17706.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Code.17659.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.17614.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Lexicon.17437.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Visitor.17577.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Scanning.17462.ll
> Analyzing ./build/temp.linux-x86_64-2.7/home/dalcinl/Devel/cython-dev/Cython/Compiler/Parsing.17489.ll
> Generated 2 warnings, see pstack.txt for details.
> [dalcinl at kw2060 build.bak]$ cat pstack.txt
> ---
> bug: anti-dce
> model: |
>   %523 = icmp ne i64 %522, 0, !dbg !1139
>   -->  false
>   ************************************************************
>   :
>   %518 = load %struct._object** @PyExc_UnboundLocalError, align 8, !dbg !1140
>   %519 = call %struct._object* (%struct._object*, i8*, ...)*
> @PyErr_Format(%struct._object* %518, i8* getelementptr inbounds ([49 x
> i8]* @.str100, i32 0, i32 0), i8* getelementptr inbounds ([6 x i8]*
> @.str52, i32 0, i32 0)), !dbg !1140
>   br label %605, !dbg !1143
> stack:
>   - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6651:0
>   - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:1752:0
> ncore: 1
> core:
>   - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6511:0
>     - null pointer dereference
> ---
> bug: anti-simplify
> model: |
>   %7699 = icmp ne %struct._object* %78, null, !dbg !4455
>   -->  true
> stack:
>   - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:16304:0
> ncore: 1
> core:
>   - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:30198:0
>     - null pointer dereference

Erm - interesting. I looked through the code lines above and couldn't find
anything that looked suspicious. I hope I used the same source version as
you did (latest master?). Maybe I was just blinded by macros, but what I
saw looked rather reasonable...

Stefan


From dalcinl at gmail.com  Fri Nov  1 21:47:47 2013
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Nov 2013 23:47:47 +0300
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
 in C code
In-Reply-To: <5273F8B1.9020706@behnel.de>
References: <5272929A.7000302@behnel.de>
 <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
 <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>
 <5273F8B1.9020706@behnel.de>
Message-ID: <CAEcYPwAYOLUpThxKTHRY0K-ZrEaOfiHkaB7sn6pf7PM57=fRpA@mail.gmail.com>

On 1 November 2013 21:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Erm - interesting. I looked through the code lines above and couldn't find
> anything that looked suspicious. I hope I used the same source version as
> you did (latest master?). Maybe I was just blinded by macros, but what I
> saw looked rather reasonable...
>

Oh! Sorry, I've used branch 0.19.x, I'll check master and report back
my findings.


-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Sat Nov  2 12:09:46 2013
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 2 Nov 2013 14:09:46 +0300
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
 in C code
In-Reply-To: <CAEcYPwAYOLUpThxKTHRY0K-ZrEaOfiHkaB7sn6pf7PM57=fRpA@mail.gmail.com>
References: <5272929A.7000302@behnel.de>
 <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
 <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>
 <5273F8B1.9020706@behnel.de>
 <CAEcYPwAYOLUpThxKTHRY0K-ZrEaOfiHkaB7sn6pf7PM57=fRpA@mail.gmail.com>
Message-ID: <CAEcYPwCm9GXWro48AAckoySkYV2Di6gX9k9zi=8RGPcx-nh=7w@mail.gmail.com>

On 1 November 2013 23:47, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 1 November 2013 21:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>
>> Erm - interesting. I looked through the code lines above and couldn't find
>> anything that looked suspicious. I hope I used the same source version as
>> you did (latest master?). Maybe I was just blinded by macros, but what I
>> saw looked rather reasonable...
>>
>
> Oh! Sorry, I've used branch 0.19.x, I'll check master and report back
> my findings.
>

OK, this is the commit in master I've used for test:

$ git show
commit 2cf64fd4e3e1f5d570d856c2e98fdc51930f1a50
Author: Robert Bradshaw <robertwb at gmail.com>
Date:   Fri Nov 1 09:05:24 2013 -0700

    Fix executable bits.

I got three warnings, see below. I understand the ones in _tempita.c
and FlowControl.c, but not sure about the others.



---
bug: anti-dce
model: |
  %523 = icmp ne i64 %522, 0, !dbg !1146
  -->  false
  ************************************************************
  :
  %518 = load %struct._object** @PyExc_UnboundLocalError, align 8, !dbg !1147
  %519 = call %struct._object* (%struct._object*, i8*, ...)*
@PyErr_Format(%struct._object* %518, i8* getelementptr inbounds ([49 x
i8]* @.str101, i32 0, i32 0), i8* getelementptr inbounds ([6 x i8]*
@.str52, i32 0, i32 0)), !dbg !1147
  br label %605, !dbg !1150
stack:
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6632:0
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:1734:0
ncore: 1
core:
  - /home/dalcinl/Devel/cython-dev/Cython/Plex/Scanners.c:6492:0
    - null pointer dereference
---
bug: anti-simplify
model: |
  %2387 = icmp ne %struct._object* %__pyx_t_4.1, null, !dbg !3029
  -->  true
stack:
  - /home/dalcinl/Devel/cython-dev/Cython/Tempita/_tempita.c:24184:0
ncore: 1
core:
  - /home/dalcinl/Devel/cython-dev/Cython/Tempita/_tempita.c:591:0
    - null pointer dereference
---
bug: anti-simplify
model: |
  %7850 = icmp ne %struct._object* %78, null, !dbg !4512
  -->  true
stack:
  - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:16478:0
ncore: 1
core:
  - /home/dalcinl/Devel/cython-dev/Cython/Compiler/FlowControl.c:30460:0
    - null pointer dereference



-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Sat Nov  2 15:09:14 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 02 Nov 2013 15:09:14 +0100
Subject: [Cython] [cython-users] "Stack" checker for undefined behaviour
 in C code
In-Reply-To: <CAEcYPwCm9GXWro48AAckoySkYV2Di6gX9k9zi=8RGPcx-nh=7w@mail.gmail.com>
References: <5272929A.7000302@behnel.de>
 <CAEcYPwBLT8ZNCF7HmiJ5d0aDdF3vfLkH3DkEqdOjidNOzOhT-Q@mail.gmail.com>
 <CAEcYPwDGk+Cw95y_X3_ggetNgmuTzR=dXOLdGtGb7CPOQ=XPYg@mail.gmail.com>
 <5273F8B1.9020706@behnel.de>
 <CAEcYPwAYOLUpThxKTHRY0K-ZrEaOfiHkaB7sn6pf7PM57=fRpA@mail.gmail.com>
 <CAEcYPwCm9GXWro48AAckoySkYV2Di6gX9k9zi=8RGPcx-nh=7w@mail.gmail.com>
Message-ID: <5275078A.6090205@behnel.de>

Lisandro Dalcin, 02.11.2013 12:09:
> On 1 November 2013 23:47, Lisandro Dalcin wrote:
>> On 1 November 2013 21:53, Stefan Behnel wrote:
>>> Erm - interesting. I looked through the code lines above and couldn't find
>>> anything that looked suspicious. I hope I used the same source version as
>>> you did (latest master?). Maybe I was just blinded by macros, but what I
>>> saw looked rather reasonable...
>>
>> Oh! Sorry, I've used branch 0.19.x, I'll check master and report back
>> my findings.
> 
> OK, this is the commit in master I've used for test:
> 
> $ git show
> commit 2cf64fd4e3e1f5d570d856c2e98fdc51930f1a50

Thanks, that makes more sense.


> I got three warnings, see below. I understand the ones in _tempita.c
> and FlowControl.c, but not sure about the others.

They all look like false positives to me, e.g. dereferencing global
pointers to Python's builtin types.

Stefan


From nouiz at nouiz.org  Tue Nov  5 18:08:25 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 5 Nov 2013 12:08:25 -0500
Subject: [Cython] Supporting new and old NumPy C API
Message-ID: <CADKKbtj8hSEUiGuUs5cgQ6JQkCOyofj7GyT8kSSmP=b=kkUu0Q@mail.gmail.com>

Hi,

As you probably know, cython generated c code use the old NumPy C API,
so when we compile them with newer version of NumPy this generate
warnings.

In Theano we compile many different module during the user script. So
this cause many warning being printed to the users, so we updated
Theano to work with newer and older NumPy without warning.

As we disable the old NumPy C API, we are not able to compile the
cython generated code. So I manually updated the cython code to work
with the new and old api.

I did a post on the numpy mailing list that explain how I did this, in
case someone want to tackle this problem in Cython:
http://mail.scipy.org/pipermail/numpy-discussion/2013-November/068209.html

Fr?d?ric Bastien

From bistromathic1 at gmail.com  Tue Nov  5 15:03:06 2013
From: bistromathic1 at gmail.com (James Sanders)
Date: Tue, 5 Nov 2013 14:03:06 +0000
Subject: [Cython] Bug report: some incorrect typed memoryview indexing
	operations cause crashes
Message-ID: <CADnyRoo0AnB=_2nsh_ZTcwxpTjuhaO-Lro+JcGSEwTeaApoWTw@mail.gmail.com>

If I try to compile the following with the current master version:

def buggy(int N, double[:] x):
    x[0, N-1]

I get a crash.  This only seems to happen when too many indices are used,
and at least one of them is an expression involving a variable (for
example, things like x[0, N] and x[0, 2-1] don't cause crashes).  Here is
the full output (this probably isn't related, but I'm also a bit confused
about all these "non-trivial type declarators" warnings, which keep
appearing in code I thought was working correctly):

warning: View.MemoryView:410:43: Non-trivial type declarators in shared
declaration.
warning: View.MemoryView:583:32: Non-trivial type declarators in shared
declaration.
warning: View.MemoryView:588:32: Non-trivial type declarators in shared
declaration.
warning: View.MemoryView:1020:20: Non-trivial type declarators in shared
declaration.
warning: View.MemoryView:1020:28: Non-trivial type declarators in shared
declaration.
warning: View.MemoryView:1020:38: Non-trivial type declarators in shared
declaration.

Error compiling Cython file:
------------------------------
------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Too many indices specified for type double[:]

Error compiling Cython file:
------------------------------------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls

ModuleNode.body = StatListNode(bug.pyx:1:0)
StatListNode.stats[0] = DefNode(bug.pyx:1:0,
    modifiers = [...]/0,
    name = 'buggy',
    num_required_args = 2,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
DefNode.body = StatListNode(bug.pyx:2:5)
StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5)
ExprStatNode.expr = IndexNode(bug.pyx:2:5,
    use_managed_ref = True)
IndexNode.index = TupleNode(bug.pyx:2:5,
    is_sequence_constructor = 1,
    result_is_used = True,
    use_managed_ref = True)
TupleNode.args[1] = SubNode(bug.pyx:2:10,
    infix = True,
    operator = '-',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "Visitor.py", line 170, in
Cython.Compiler.Visitor.TreeVisitor._visit
(/home/.../cython/Cython/Compiler/Visitor.c:4122)
  File "Visitor.py", line 505, in
Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode
(/home/.../cython/Cython/Compiler/Visitor.c:9459)
  File "Visitor.py", line 516, in
Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node
(/home/.../cython/Cython/Compiler/Visitor.c:9613)
AttributeError: 'NoneType' object has no attribute 'is_builtin_type'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20131105/08af07a8/attachment.html>

From josh.ayers at gmail.com  Wed Nov  6 19:35:39 2013
From: josh.ayers at gmail.com (Josh Ayers)
Date: Wed, 6 Nov 2013 10:35:39 -0800
Subject: [Cython] Bug Report: can't assign zero length array to contiguous
	2D memoryview
Message-ID: <CACOB4aPicUUA0TPkCFcEfBPwxj+thGHJbmGFXSM+HyWM-JWZXw@mail.gmail.com>

Assigning a NumPy array with a zero length dimension to a memoryview fails
if the memoryview is declared contiguous in the zero-length dimension.  I'm
getting a run-time error that the buffer is not C-contiguous or Fortran
contiguous.

See examples below.

Thanks,
Josh Ayers


cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works
cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works
cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not
C-contiguous)
cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works

cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works
cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails
(buffer not Fortran contiguous)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20131106/52460a43/attachment.html>

From stefan_ml at behnel.de  Fri Nov  8 15:16:38 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Nov 2013 15:16:38 +0100
Subject: [Cython] question on Get/SetItemInt*() utility functions
Message-ID: <527CF246.2030509@behnel.de>

Hi,

the Get/SetItemInt helper macros usually look something like this:

"""
#define __Pyx_GetItemInt(o, i, size, to_py_func, is_list, wraparound,
boundscheck) \
    (((size) <= sizeof(Py_ssize_t)) ? \
    __Pyx_GetItemInt_Fast(o, i, is_list, wraparound, boundscheck) : \
    __Pyx_GetItemInt_Generic(o, to_py_func(i)))
"""

e.g. here:

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/ObjectHandling.c#L248

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/StringTools.c#L318

The "generic" part will almost never be called, except when the index
*type* is really huge. However, if that happens, there's a rather large
impact due to object creation, even if the value in question is tiny.

Is there a way to safely extend this compile time size check with checks
that compare the actual runtime value with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX ?

Within those bounds, a cast would do the trick just fine, and we could
still use the fast path. If it fails that bounds check, however, and the
type we are processing is a builtin container, we'd already know that an
exception is going to be raised and could simply drop the generic
implementation, replacing it with an appropriate IndexError directly.
Obviously, non-builtins would still need to fall back to the generic
implementation, but there is a lot of special casing code for builtins that
(IMHO) could be removed and replaced by a simple exception.

I'm aware that the C comparison rules for unsigned vs. signed might make
this tricky, but I also know that some on this list know their C better
than I do. :)

BTW, I also wonder if the above sizeof() comparison is safe enough - what
if "i" has an unsigned type of the same size as Py_ssize_t and its value is
larger than PY_SSIZE_T_MAX ? Wouldn't it wrap around in that case?

Stefan

From stefan_ml at behnel.de  Sat Nov  9 07:17:38 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Nov 2013 07:17:38 +0100
Subject: [Cython] question on Get/SetItemInt*() utility functions
In-Reply-To: <527CF246.2030509@behnel.de>
References: <527CF246.2030509@behnel.de>
Message-ID: <527DD382.2070702@behnel.de>

Having thought about this come more:

Stefan Behnel, 08.11.2013 15:16:
> the Get/SetItemInt helper macros usually look something like this:
> 
> """
> #define __Pyx_GetItemInt(o, i, size, to_py_func, is_list, wraparound,
> boundscheck) \
>     (((size) <= sizeof(Py_ssize_t)) ? \

I noticed that there is code in ExprNodes.py that increases the "size"
value by one if the type is not signed, so this is currently a safe check.
However, we can do better than that.


>     __Pyx_GetItemInt_Fast(o, i, is_list, wraparound, boundscheck) : \
>     __Pyx_GetItemInt_Generic(o, to_py_func(i)))
> """
> 
> e.g. here:
> 
> https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/ObjectHandling.c#L248
> 
> https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/StringTools.c#L318
> 
> The "generic" part will almost never be called, except when the index
> *type* is really huge. However, if that happens, there's a rather large
> impact due to object creation, even if the value in question is tiny.
> 
> Is there a way to safely extend this compile time size check with checks
> that compare the actual runtime value with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX ?

Ok, so what are the cases we have here?

1) sizeof(type_of_i) < sizeof(Py_ssize_t)
- no problem (note that this includes char, the only integer type of
undefined signedness)

2) sizeof(type_of_i) > sizeof(Py_ssize_t)
- A comparison "i <= PY_SSIZE_T_MAX" will promote the rhs to the type of i,
which makes it safe.
- A comparison "i >= PY_SSIZE_T_MIN" is only safe if i is signed,
otherwise, it is not needed.
-> Cython knows at compile time if the index type is unsigned and thus can
pass a flag into the C macro, i.e. this should be safe:

 (i <= PY_SSIZE_T_MAX)  &&  (!signed || i >= PY_SSIZE_T_MIN)

3) sizeof(type_of_i) == sizeof(Py_ssize_t)
- If i is signed, both types must be identical (both are integers), so this
is safe.
- If i is unsigned, comparing i <= PY_SSIZE_T_MAX will promote the rhs to
the type of i and is thus safe.
-> this should be safe:

 signed || (i <= PY_SSIZE_T_MAX)


Putting it all together, we can use the fast path in these cases:

    (sizeof(type_of_i) < sizeof(Py_ssize_t))  ||

    (sizeof(type_of_i) > sizeof(Py_ssize_t) &&
          (i <= (type_of_i)PY_SSIZE_T_MAX)  &&
          (!signed || i >= (type_of_i)PY_SSIZE_T_MIN))  ||

    (sizeof(type_of_i) == sizeof(Py_ssize_t) &&
          (signed || (i <= (type_of_i)PY_SSIZE_T_MAX)))


The casts I added should keep the C compiler from complaining about unsafe
comparisons.

Then, for builtin container types, the fallback path already knows that the
index is out of bounds and can always raise an IndexError. For unknown
types, the existing "Generic" fallback function can be used.

Does the above appear correct?

Stefan


From stefan_ml at behnel.de  Sat Nov  9 08:20:00 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Nov 2013 08:20:00 +0100
Subject: [Cython] question on Get/SetItemInt*() utility functions
In-Reply-To: <527DD382.2070702@behnel.de>
References: <527CF246.2030509@behnel.de> <527DD382.2070702@behnel.de>
Message-ID: <527DE220.6050303@behnel.de>

Stefan Behnel, 09.11.2013 07:17:
> Putting it all together, we can use the fast path in these cases:
> 
>     (sizeof(type_of_i) < sizeof(Py_ssize_t))  ||
> 
>     (sizeof(type_of_i) > sizeof(Py_ssize_t) &&
>           (i <= (type_of_i)PY_SSIZE_T_MAX)  &&
>           (!signed || i >= (type_of_i)PY_SSIZE_T_MIN))  ||
> 
>     (sizeof(type_of_i) == sizeof(Py_ssize_t) &&
>           (signed || (i <= (type_of_i)PY_SSIZE_T_MAX)))
> 
> 
> The casts I added should keep the C compiler from complaining about unsafe
> comparisons.
> 
> Then, for builtin container types, the fallback path already knows that the
> index is out of bounds and can always raise an IndexError. For unknown
> types, the existing "Generic" fallback function can be used.

https://github.com/cython/cython/commit/b66981604baf59ea465027cc10baa42af7c76571

https://github.com/cython/cython/commit/0cae66191bfe42657ff0010e4ca13d032cb523b8

Stefan


From stefan_ml at behnel.de  Sat Nov  9 10:16:21 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Nov 2013 10:16:21 +0100
Subject: [Cython] question on Get/SetItemInt*() utility functions
In-Reply-To: <527DE220.6050303@behnel.de>
References: <527CF246.2030509@behnel.de> <527DD382.2070702@behnel.de>
 <527DE220.6050303@behnel.de>
Message-ID: <527DFD65.8020402@behnel.de>

Stefan Behnel, 09.11.2013 08:20:
> Stefan Behnel, 09.11.2013 07:17:
>> Putting it all together, we can use the fast path in these cases:
>>
>>     (sizeof(type_of_i) < sizeof(Py_ssize_t))  ||
>>
>>     (sizeof(type_of_i) > sizeof(Py_ssize_t) &&
>>           (i <= (type_of_i)PY_SSIZE_T_MAX)  &&
>>           (!signed || i >= (type_of_i)PY_SSIZE_T_MIN))  ||
>>
>>     (sizeof(type_of_i) == sizeof(Py_ssize_t) &&
>>           (signed || (i <= (type_of_i)PY_SSIZE_T_MAX)))
>>
>>
>> The casts I added should keep the C compiler from complaining about unsafe
>> comparisons.
>>
>> Then, for builtin container types, the fallback path already knows that the
>> index is out of bounds and can always raise an IndexError. For unknown
>> types, the existing "Generic" fallback function can be used.
> 
> https://github.com/cython/cython/commit/b66981604baf59ea465027cc10baa42af7c76571
> 
> https://github.com/cython/cython/commit/0cae66191bfe42657ff0010e4ca13d032cb523b8

And if anyone has a way to make this less ugly, I'd be happy to hear about it:

https://github.com/cython/cython/commit/de12778fe0995660e8f1c726194cef40885d7a15

Stefan


From stefan_ml at behnel.de  Sun Nov 10 12:29:10 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 10 Nov 2013 12:29:10 +0100
Subject: [Cython] Fwd: [cython-users] How cythonize support c++?
In-Reply-To: <CABkOF6SX_TnaTyv+5CYkxJ50QL4rZkshQ63Tzjpby6h5JzX3sg@mail.gmail.com>
References: <CABkOF6SX_TnaTyv+5CYkxJ50QL4rZkshQ63Tzjpby6h5JzX3sg@mail.gmail.com>
Message-ID: <527F6E06.6010209@behnel.de>

Given how often this has come up already, we should consider it a major
usability issue.

IMHO, stricter argument validation in cythonize() would help, as would
better documentation about which option to use where.

Stefan


-------- Original-Message --------
Subject: [cython-users] How cythonize support c++?
Date: Sun, 10 Nov 2013 19:19:48 +0800
To: cython-users at googlegroups.com

Hi All

   I encountered a problem on writing a setup.py for cython extension.
Following a tutorial from cython.org I wrote some codes like:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize

setup(cmdclass={'build_ext':build_ext}, ext_modules = cythonize("my.pyx",
language="c++"))

when I typed python setup.py build_ext --inplace on windows 7 with cython
0.19.2
It complained a lot of error.

I founded out if I write setup file like this:

extModules = [Extension("my", sources=["my.pyx"],language="c++")]
setup(cmdclass={'build_ext':build_ext}, ext_modules=extModules)

It ran fine without any problem.

Do I miss sth so that cythonize can't support c++ ?

Regards

gelin yan


From vinay_sajip at yahoo.co.uk  Mon Nov 25 11:35:04 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 25 Nov 2013 10:35:04 +0000 (UTC)
Subject: [Cython] Cython crash when using forward declarations
Message-ID: <loom.20131125T111644-55@post.gmane.org>

The following Cython source:

cdef extern from "header":
    cdef cppclass String
    cdef cppclass List[T]
    cdef cppclass StringList(List[String])

    cdef cppclass String:
        String()
        int length()

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        StringList()
        void sort()

causes a crash:

Error compiling Cython file:
------------------------------------------------------------
...

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        ^
------------------------------------------------------------

s.pyx:14:9: Cannot inherit from incomplete type

Error compiling Cython file:
------------------------------------------------------------
...
    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        StringList()
       ^
------------------------------------------------------------

s.pyx:15:8: Compiler crash in AnalyseDeclarationsTransform

File 'ModuleNode.py', line 101, in analyse_declarations: ModuleNode(s.pyx:1:0,
    full_module_name = 's')
File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:1:5)
File 'Nodes.py', line 438, in analyse_declarations: CDefExternNode(s.pyx:1:5,
    include_file = u'header')
File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:2:4)
File 'Nodes.py', line 1313, in analyse_declarations: CppClassNode(s.pyx:14:9,
    base_classes = [...]/1,
    name = u'StringList',
    visibility = u'extern')
File 'Nodes.py', line 1203, in analyse_declarations: CVarDefNode(s.pyx:15:8,
    modifiers = [...]/0,
    visibility = u'extern')

Compiler crash traceback from this point on:
  File ".../Cython/Compiler/Nodes.py", line 1203, in analyse_declarations
    api = self.api, modifiers = self.modifiers)
  File ".../Cython/Compiler/Symtab.py", line 2131, in declare_cfunction
    self.check_base_default_constructor(pos)
  File ".../Cython/Compiler/Symtab.py", line 2108, in \
                                            check_base_default_constructor
    temp_entry = base_class.scope.lookup_here("<init>")
AttributeError: 'NoneType' object has no attribute 'lookup_here'

There seem to be two problems here: the AttributeError crash, but also the
earlier "Cannot inherit from incomplete type" message. None of the types look
incomplete; if you comment out the forward declarations, no error is reported:
so it looks as if the forward declarations aren't being tied up with the actual
declarations. In the actual use case, of course, I need the forward decls.

Regards,

Vinay Sajip


From daniele at grinta.net  Tue Nov 26 16:18:51 2013
From: daniele at grinta.net (Daniele Nicolodi)
Date: Tue, 26 Nov 2013 16:18:51 +0100
Subject: [Cython] BUG: assignment to typed memoryview
Message-ID: <5294BBDB.1090609@grinta.net>

Hello,

I believe there is a bug in how assignment to typed memoryviews is
handled in the compiler.  I think the following code should be valid code:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b = np.empty(a.size // 2)
  b[:] = a[::2]

However, the Cython compiler complains with:

  Memoryview 'double[:]' not conformable to memoryview 'double[::1]'.

A similar thing happens with memoryview copies:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b
  b = a[::2].copy()

In both cases I would expect Cython to copy the non contiguous elements
of the source array to the destination array (or to a newly created
array in the second case).  The copy() method is defined here
https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#L592
and (while I haven't spent much time analyzing it in detail) it seems to
do the right thing.

Indeed, if I short-circuit the src_conforms_to_dst() function
https://github.com/cython/cython/blob/master/Cython/Compiler/MemoryView.py#L141
the code compiles and runs correctly in both cases.

I don't know much about how the Cython compiler works, therefore I don't
know where to dig for the source of this problem.  Can someone point me
in the right direction?

Thanks. Cheers,
Daniele

From yury at shurup.com  Fri Nov 29 14:26:36 2013
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 29 Nov 2013 14:26:36 +0100
Subject: [Cython] 'Referenced before assignment' warning triggered due to
 OpenMP directives?
Message-ID: <1385731596.2924.183.camel@newpride>

Hi,

I'm trying to find out whether the number of threads has been actually
set to what I wanted it to be, but the compiler tells me that I'm trying
to reference a variable before assignment. This doesn't happen when I
simply release the GIL, but only within a parallel section.

Is it a problem with Cython or I'm missing something obvious?

Thanks!

$ cat ompt.pyx 
cimport openmp
from cython.parallel import parallel, prange

cdef Py_ssize_t x = 1

with nogil, parallel():
    x = openmp.omp_get_num_threads()

print("Number of OpenMP threads is '{}'!".format(x))

$ cython ompt.pyx 

Error compiling Cython file:
------------------------------------------------------------
...
cdef Py_ssize_t x = 1

with nogil, parallel():
    x = openmp.omp_get_num_threads()

print("Number of OpenMP threads is '{}'!".format(x))
                                                 ^
------------------------------------------------------------

ompt.pyx:9:50: local variable 'x' referenced before assignment


-- 
Sincerely yours,
Yury V. Zaytsev



From stefan_ml at behnel.de  Fri Nov 29 17:03:04 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 29 Nov 2013 17:03:04 +0100
Subject: [Cython] "embedsignature" and "autotestdict" features in Py3.4
Message-ID: <5298BAB8.3080909@behnel.de>

Hi,

recent changes in Py3.4 affect the functionality of the two directives
"embedsignature" and "autotestdict".

The so-called "argument clinic" extracts any embedded signatures from
docstrings and moves them into a new property "__text_signature__". This is
done at runtime, i.e. when either "__doc__" or "__text_signature__" are
requested.

http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182

I personally consider this a feature (and it works nicely with the
signatures that Cython embeds), but you may or may not agree. It broke some
of our own doctests, at least, because the "__doc__" value that we tested
for was no longer the same.

Regarding the "autotestdict", CPython also got smarter and now finds
doctests embedded in C implemented functions all by itself. This is clearly
an improvement compared to the previous behaviour. However, this also means
that it now executes both the doctest of the function itself and its copy
in the generated "__test__" dict (stored in the module under that name),
meaning that it executes all function doctests twice. This also broke some
of Cython's own doctests for us, which modified global state and thus
failed the second time.

I'm yet unsure if we should try to do something about this. Currently, the
explicit doctest dict is generated by default, so all code that uses
doctests in function docstrings suffers from this. We might be able to make
Cython a bit smarter so that these docstrings are left out of the test dict
when (C-)compiling in CPython 3.4. Deciding which ones are still needed
might be a bit tricky, though...

Stefan