From stefan_ml at behnel.de  Fri Jan  3 19:22:24 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Jan 2014 19:22:24 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
Message-ID: <52C6FFE0.9060600@behnel.de>

Hi,

I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
got it to use the latest NumPy release 1.8. This made one of the tests fail:

"""
    Traceback (most recent call last):
      File ".../doctest.py", line 1313, in __run
        compileflags, 1), test.globs)
      File "<doctest relaxed_strides.__test__.test_one_sized (line
29)[3]>", line 1, in <module>
        test_one_sized(a)[0]
      File "relaxed_strides.pyx", line 38, in
relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
      File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
(relaxed_strides.cpp:7568)
      File "stringsource", line 327, in
View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)

    ValueError: ndarray is not C-contiguous
"""

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console

According to the comments in the test file and the corresponding NumPy pull
request, this seems to be somewhat expected.

https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx

https://github.com/numpy/numpy/pull/3162

Does someone know enough about this to figure out what to do?

Stefan

From markflorisson88 at gmail.com  Fri Jan  3 23:28:37 2014
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Jan 2014 22:28:37 +0000
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C6FFE0.9060600@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
Message-ID: <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>

On 3 January 2014 18:22, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
> got it to use the latest NumPy release 1.8. This made one of the tests fail:
>
> """
>     Traceback (most recent call last):
>       File ".../doctest.py", line 1313, in __run
>         compileflags, 1), test.globs)
>       File "<doctest relaxed_strides.__test__.test_one_sized (line
> 29)[3]>", line 1, in <module>
>         test_one_sized(a)[0]
>       File "relaxed_strides.pyx", line 38, in
> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>       File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
> (relaxed_strides.cpp:7568)
>       File "stringsource", line 327, in
> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>
>     ValueError: ndarray is not C-contiguous
> """
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>
> According to the comments in the test file and the corresponding NumPy pull
> request, this seems to be somewhat expected.
>
> https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>
> https://github.com/numpy/numpy/pull/3162
>
> Does someone know enough about this to figure out what to do?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

Hey Stefan,

It seems to come from the call to __Pyx_GetBuffer, which is
PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
for an extent of 1, but instead only checking the stride (which is a
multiple of the itemsize)?

Maybe Nathaniel or Sebastien can shed more light on this matter.

Mark

From robertwb at gmail.com  Sat Jan  4 06:01:01 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 3 Jan 2014 21:01:01 -0800
Subject: [Cython] Cython 0.20 beta 1
Message-ID: <CADiQ+QC_O9GJLG0dn5g7fj_O18fBWwuY=f-A_bXS09O3CCX+oQ@mail.gmail.com>

I just uploaded the first beta release for Cython 0.20, it's tagged as
0.20b1 in the git repository or you can find it at
http://cython.org/release/Cython-0.20b1.tar.gz . A summary of the
changes can be found at
https://github.com/cython/cython/blob/0.20b1/CHANGES.rst , please try
it out and let us know if you have any issues.

- Robert

From stefan_ml at behnel.de  Sat Jan  4 12:53:32 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 12:53:32 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
Message-ID: <52C7F63C.9090707@behnel.de>

mark florisson, 03.01.2014 23:28:
> On 3 January 2014 18:22, Stefan Behnel wrote:
>> I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
>> got it to use the latest NumPy release 1.8. This made one of the tests fail:
>>
>> """
>>     Traceback (most recent call last):
>>       File ".../doctest.py", line 1313, in __run
>>         compileflags, 1), test.globs)
>>       File "<doctest relaxed_strides.__test__.test_one_sized (line
>> 29)[3]>", line 1, in <module>
>>         test_one_sized(a)[0]
>>       File "relaxed_strides.pyx", line 38, in
>> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>>       File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
>> (relaxed_strides.cpp:7568)
>>       File "stringsource", line 327, in
>> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>>
>>     ValueError: ndarray is not C-contiguous
>> """
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>>
>> According to the comments in the test file and the corresponding NumPy pull
>> request, this seems to be somewhat expected.
>>
>> https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>>
>> https://github.com/numpy/numpy/pull/3162
>>
>> Does someone know enough about this to figure out what to do?
> 
> It seems to come from the call to __Pyx_GetBuffer, which is
> PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
> for an extent of 1, but instead only checking the stride (which is a
> multiple of the itemsize)?

No, Py3 doesn't do any validation here, it does a straight call into the
object's slot function, i.e. into the NumPy array itself.

So, the question is: who's wrong here? The test or NumPy?

Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
environment variable for the NumPy build. Let's try that first.

http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking

Stefan


From stefan_ml at behnel.de  Sat Jan  4 15:46:14 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 15:46:14 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C7F63C.9090707@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
Message-ID: <52C81EB6.3060401@behnel.de>

Stefan Behnel, 04.01.2014 12:53:
> mark florisson, 03.01.2014 23:28:
>> On 3 January 2014 18:22, Stefan Behnel wrote:
>>> I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
>>> got it to use the latest NumPy release 1.8. This made one of the tests fail:
>>>
>>> """
>>>     Traceback (most recent call last):
>>>       File ".../doctest.py", line 1313, in __run
>>>         compileflags, 1), test.globs)
>>>       File "<doctest relaxed_strides.__test__.test_one_sized (line
>>> 29)[3]>", line 1, in <module>
>>>         test_one_sized(a)[0]
>>>       File "relaxed_strides.pyx", line 38, in
>>> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>>>       File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
>>> (relaxed_strides.cpp:7568)
>>>       File "stringsource", line 327, in
>>> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>>>
>>>     ValueError: ndarray is not C-contiguous
>>> """
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>>>
>>> According to the comments in the test file and the corresponding NumPy pull
>>> request, this seems to be somewhat expected.
>>>
>>> https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>>>
>>> https://github.com/numpy/numpy/pull/3162
>>>
>>> Does someone know enough about this to figure out what to do?
>>
>> It seems to come from the call to __Pyx_GetBuffer, which is
>> PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
>> for an extent of 1, but instead only checking the stride (which is a
>> multiple of the itemsize)?
> 
> No, Py3 doesn't do any validation here, it does a straight call into the
> object's slot function, i.e. into the NumPy array itself.
> 
> So, the question is: who's wrong here? The test or NumPy?
> 
> Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
> environment variable for the NumPy build. Let's try that first.
> 
> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking

Yep, that was it. Seems to work nicely now.

Stefan


From robertwb at gmail.com  Sat Jan  4 17:40:16 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 4 Jan 2014 08:40:16 -0800
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C81EB6.3060401@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de> <52C81EB6.3060401@behnel.de>
Message-ID: <CADiQ+QCQVs1=icHg7uXQDBJ0yh5wAY=+xmbvChsTf4-u1+V7mw@mail.gmail.com>

Excellent, thanks. I'll make sure this gets into the release.

On Sat, Jan 4, 2014 at 6:46 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 04.01.2014 12:53:
>> mark florisson, 03.01.2014 23:28:
>>> On 3 January 2014 18:22, Stefan Behnel wrote:
>>>> I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
>>>> got it to use the latest NumPy release 1.8. This made one of the tests fail:
>>>>
>>>> """
>>>>     Traceback (most recent call last):
>>>>       File ".../doctest.py", line 1313, in __run
>>>>         compileflags, 1), test.globs)
>>>>       File "<doctest relaxed_strides.__test__.test_one_sized (line
>>>> 29)[3]>", line 1, in <module>
>>>>         test_one_sized(a)[0]
>>>>       File "relaxed_strides.pyx", line 38, in
>>>> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>>>>       File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
>>>> (relaxed_strides.cpp:7568)
>>>>       File "stringsource", line 327, in
>>>> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>>>>
>>>>     ValueError: ndarray is not C-contiguous
>>>> """
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>>>>
>>>> According to the comments in the test file and the corresponding NumPy pull
>>>> request, this seems to be somewhat expected.
>>>>
>>>> https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>>>>
>>>> https://github.com/numpy/numpy/pull/3162
>>>>
>>>> Does someone know enough about this to figure out what to do?
>>>
>>> It seems to come from the call to __Pyx_GetBuffer, which is
>>> PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
>>> for an extent of 1, but instead only checking the stride (which is a
>>> multiple of the itemsize)?
>>
>> No, Py3 doesn't do any validation here, it does a straight call into the
>> object's slot function, i.e. into the NumPy array itself.
>>
>> So, the question is: who's wrong here? The test or NumPy?
>>
>> Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
>> environment variable for the NumPy build. Let's try that first.
>>
>> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking
>
> Yep, that was it. Seems to work nicely now.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sat Jan  4 17:45:15 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 17:45:15 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CADiQ+QCQVs1=icHg7uXQDBJ0yh5wAY=+xmbvChsTf4-u1+V7mw@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de> <52C81EB6.3060401@behnel.de>
 <CADiQ+QCQVs1=icHg7uXQDBJ0yh5wAY=+xmbvChsTf4-u1+V7mw@mail.gmail.com>
Message-ID: <52C83A9B.2010706@behnel.de>

Robert Bradshaw, 04.01.2014 17:40:
> On Sat, Jan 4, 2014 at 6:46 AM, Stefan Behnel wrote:
>> Stefan Behnel, 04.01.2014 12:53:
>>> I didn't define the NPY_RELAXED_STRIDES_CHECKING
>>> environment variable for the NumPy build. Let's try that first.
>>>
>>> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking
>>
>> Yep, that was it. Seems to work nicely now.
>
> Excellent, thanks. I'll make sure this gets into the release.

It's not a change in Cython, just a config option for the NumPy build (i.e.
I changed it in the Jenkins jobs that install the external packages into
CPython).

Stefan


From njs at pobox.com  Sat Jan  4 17:54:29 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 4 Jan 2014 10:54:29 -0600
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C7F63C.9090707@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
Message-ID: <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>

On 4 Jan 2014 11:53, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>
> mark florisson, 03.01.2014 23:28:
> > On 3 January 2014 18:22, Stefan Behnel wrote:
> >> I enabled the NumPy build for our Py3.3 test runs and while I was at
it, I
> >> got it to use the latest NumPy release 1.8. This made one of the tests
fail:
> >>
> >> """
> >>     Traceback (most recent call last):
> >>       File ".../doctest.py", line 1313, in __run
> >>         compileflags, 1), test.globs)
> >>       File "<doctest relaxed_strides.__test__.test_one_sized (line
> >> 29)[3]>", line 1, in <module>
> >>         test_one_sized(a)[0]
> >>       File "relaxed_strides.pyx", line 38, in
> >> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
> >>       File "stringsource", line 622, in
View.MemoryView.memoryview_cwrapper
> >> (relaxed_strides.cpp:7568)
> >>       File "stringsource", line 327, in
> >> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
> >>
> >>     ValueError: ndarray is not C-contiguous
> >> """
> >>
> >>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
> >>
> >> According to the comments in the test file and the corresponding NumPy
pull
> >> request, this seems to be somewhat expected.
> >>
> >>
https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
> >>
> >> https://github.com/numpy/numpy/pull/3162
> >>
> >> Does someone know enough about this to figure out what to do?
> >
> > It seems to come from the call to __Pyx_GetBuffer, which is
> > PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
> > for an extent of 1, but instead only checking the stride (which is a
> > multiple of the itemsize)?
>
> No, Py3 doesn't do any validation here, it does a straight call into the
> object's slot function, i.e. into the NumPy array itself.
>
> So, the question is: who's wrong here? The test or NumPy?
>
> Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
> environment variable for the NumPy build. Let's try that first.
>
> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking

It's probably nicer though to write the Cython tests in such a way that
they can pass against a default numpy installation? To avoid user confusion
and all that.

If those tests depend on numpy having relaxed strides enabled, then I'd
suggest checking for this a test time and skipping the tests if not found.
The relaxed strides docs give a recipe for determining how numpy was built.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140104/185f261c/attachment-0001.html>

From robertwb at gmail.com  Sat Jan  4 18:06:55 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 4 Jan 2014 09:06:55 -0800
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
Message-ID: <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>

Yes, that'd be good. Do you know how? I won't have time to look at this
'till next week.
On Jan 4, 2014 9:54 AM, "Nathaniel Smith" <njs at pobox.com> wrote:

> On 4 Jan 2014 11:53, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
> >
> > mark florisson, 03.01.2014 23:28:
> > > On 3 January 2014 18:22, Stefan Behnel wrote:
> > >> I enabled the NumPy build for our Py3.3 test runs and while I was at
> it, I
> > >> got it to use the latest NumPy release 1.8. This made one of the
> tests fail:
> > >>
> > >> """
> > >>     Traceback (most recent call last):
> > >>       File ".../doctest.py", line 1313, in __run
> > >>         compileflags, 1), test.globs)
> > >>       File "<doctest relaxed_strides.__test__.test_one_sized (line
> > >> 29)[3]>", line 1, in <module>
> > >>         test_one_sized(a)[0]
> > >>       File "relaxed_strides.pyx", line 38, in
> > >> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
> > >>       File "stringsource", line 622, in
> View.MemoryView.memoryview_cwrapper
> > >> (relaxed_strides.cpp:7568)
> > >>       File "stringsource", line 327, in
> > >> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
> > >>
> > >>     ValueError: ndarray is not C-contiguous
> > >> """
> > >>
> > >>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
> > >>
> > >> According to the comments in the test file and the corresponding
> NumPy pull
> > >> request, this seems to be somewhat expected.
> > >>
> > >>
> https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
> > >>
> > >> https://github.com/numpy/numpy/pull/3162
> > >>
> > >> Does someone know enough about this to figure out what to do?
> > >
> > > It seems to come from the call to __Pyx_GetBuffer, which is
> > > PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
> > > for an extent of 1, but instead only checking the stride (which is a
> > > multiple of the itemsize)?
> >
> > No, Py3 doesn't do any validation here, it does a straight call into the
> > object's slot function, i.e. into the NumPy array itself.
> >
> > So, the question is: who's wrong here? The test or NumPy?
> >
> > Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
> > environment variable for the NumPy build. Let's try that first.
> >
> >
> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking
>
> It's probably nicer though to write the Cython tests in such a way that
> they can pass against a default numpy installation? To avoid user confusion
> and all that.
>
> If those tests depend on numpy having relaxed strides enabled, then I'd
> suggest checking for this a test time and skipping the tests if not found.
> The relaxed strides docs give a recipe for determining how numpy was built.
>
> -n
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140104/63e6f4ef/attachment.html>

From njs at pobox.com  Sat Jan  4 18:36:14 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 4 Jan 2014 11:36:14 -0600
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
Message-ID: <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>

On 4 Jan 2014 17:07, "Robert Bradshaw" <robertwb at gmail.com> wrote:
>
> Yes, that'd be good. Do you know how? I won't have time to look at this
'till next week.

Something like

if not np.ones((10, 1), order="C").flags.f_contiguous:
  # numpy without relaxed stride support
  raise SkipTest

> On Jan 4, 2014 9:54 AM, "Nathaniel Smith" <njs at pobox.com> wrote:
>>
>> On 4 Jan 2014 11:53, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>> >
>> > mark florisson, 03.01.2014 23:28:
>> > > On 3 January 2014 18:22, Stefan Behnel wrote:
>> > >> I enabled the NumPy build for our Py3.3 test runs and while I was
at it, I
>> > >> got it to use the latest NumPy release 1.8. This made one of the
tests fail:
>> > >>
>> > >> """
>> > >>     Traceback (most recent call last):
>> > >>       File ".../doctest.py", line 1313, in __run
>> > >>         compileflags, 1), test.globs)
>> > >>       File "<doctest relaxed_strides.__test__.test_one_sized (line
>> > >> 29)[3]>", line 1, in <module>
>> > >>         test_one_sized(a)[0]
>> > >>       File "relaxed_strides.pyx", line 38, in
>> > >> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>> > >>       File "stringsource", line 622, in
View.MemoryView.memoryview_cwrapper
>> > >> (relaxed_strides.cpp:7568)
>> > >>       File "stringsource", line 327, in
>> > >> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>> > >>
>> > >>     ValueError: ndarray is not C-contiguous
>> > >> """
>> > >>
>> > >>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>> > >>
>> > >> According to the comments in the test file and the corresponding
NumPy pull
>> > >> request, this seems to be somewhat expected.
>> > >>
>> > >>
https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>> > >>
>> > >> https://github.com/numpy/numpy/pull/3162
>> > >>
>> > >> Does someone know enough about this to figure out what to do?
>> > >
>> > > It seems to come from the call to __Pyx_GetBuffer, which is
>> > > PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
>> > > for an extent of 1, but instead only checking the stride (which is a
>> > > multiple of the itemsize)?
>> >
>> > No, Py3 doesn't do any validation here, it does a straight call into
the
>> > object's slot function, i.e. into the NumPy array itself.
>> >
>> > So, the question is: who's wrong here? The test or NumPy?
>> >
>> > Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
>> > environment variable for the NumPy build. Let's try that first.
>> >
>> >
http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking
>>
>> It's probably nicer though to write the Cython tests in such a way that
they can pass against a default numpy installation? To avoid user confusion
and all that.
>>
>> If those tests depend on numpy having relaxed strides enabled, then I'd
suggest checking for this a test time and skipping the tests if not found.
The relaxed strides docs give a recipe for determining how numpy was built.
>>
>> -n
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140104/0a3709bc/attachment.html>

From stefan_ml at behnel.de  Sat Jan  4 22:47:50 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 22:47:50 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
Message-ID: <52C88186.30603@behnel.de>

Nathaniel Smith, 04.01.2014 18:36:
> On 4 Jan 2014 17:07, "Robert Bradshaw" wrote:
>> Yes, that'd be good. Do you know how? I won't have time to look at this
> 'till next week.
> 
> Something like
> 
> if not np.ones((10, 1), order="C").flags.f_contiguous:
>   # numpy without relaxed stride support
>   raise SkipTest

https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa

Stefan


From stefan_ml at behnel.de  Sat Jan  4 22:51:30 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 22:51:30 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C88186.30603@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de>
Message-ID: <52C88262.5030900@behnel.de>

Stefan Behnel, 04.01.2014 22:47:
> Nathaniel Smith, 04.01.2014 18:36:
>> On 4 Jan 2014 17:07, "Robert Bradshaw" wrote:
>>> Yes, that'd be good. Do you know how? I won't have time to look at this
>> 'till next week.
>>
>> Something like
>>
>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>   # numpy without relaxed stride support
>>   raise SkipTest
> 
> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa

Hmm, but this doesn't seem to work for me in older NumPy versions, although
the original test used to work there. Should we explicitly test for NumPy
0.18+ as well?

Stefan


From stefan_ml at behnel.de  Sat Jan  4 23:01:30 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jan 2014 23:01:30 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C88262.5030900@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
Message-ID: <52C884BA.4040500@behnel.de>

Stefan Behnel, 04.01.2014 22:51:
> Stefan Behnel, 04.01.2014 22:47:
>> Nathaniel Smith, 04.01.2014 18:36:
>>> On 4 Jan 2014 17:07, "Robert Bradshaw" wrote:
>>>> Yes, that'd be good. Do you know how? I won't have time to look at this
>>> 'till next week.
>>>
>>> Something like
>>>
>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>>   # numpy without relaxed stride support
>>>   raise SkipTest
>>
>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
> 
> Hmm, but this doesn't seem to work for me in older NumPy versions, although
> the original test used to work there. Should we explicitly test for NumPy
> 0.18+ as well?

(Looks like the Cython versioning scheme starts messing up my sense for
good versioning...)

I meant NumPy 1.8, 'obviously', i.e. this:

https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a

Stefan


From njs at pobox.com  Sun Jan  5 02:07:04 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 4 Jan 2014 19:07:04 -0600
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C884BA.4040500@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
 <52C884BA.4040500@behnel.de>
Message-ID: <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>

No, I'm missing something now; AFAIK there are only two numpy behaviors:
with relaxed strides and without relaxed strides, and version number should
be irrelevant beyond that. What's different between
1.8-without-relaxed-strides and 1.7 that makes the test break?
On 4 Jan 2014 22:01, "Stefan Behnel" <stefan_ml at behnel.de> wrote:

> Stefan Behnel, 04.01.2014 22:51:
> > Stefan Behnel, 04.01.2014 22:47:
> >> Nathaniel Smith, 04.01.2014 18:36:
> >>> On 4 Jan 2014 17:07, "Robert Bradshaw" wrote:
> >>>> Yes, that'd be good. Do you know how? I won't have time to look at
> this
> >>> 'till next week.
> >>>
> >>> Something like
> >>>
> >>> if not np.ones((10, 1), order="C").flags.f_contiguous:
> >>>   # numpy without relaxed stride support
> >>>   raise SkipTest
> >>
> >>
> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
> >
> > Hmm, but this doesn't seem to work for me in older NumPy versions,
> although
> > the original test used to work there. Should we explicitly test for NumPy
> > 0.18+ as well?
>
> (Looks like the Cython versioning scheme starts messing up my sense for
> good versioning...)
>
> I meant NumPy 1.8, 'obviously', i.e. this:
>
>
> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140104/8879f9ff/attachment.html>

From stefan_ml at behnel.de  Sun Jan  5 09:03:15 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jan 2014 09:03:15 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
 <52C884BA.4040500@behnel.de>
 <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>
Message-ID: <52C911C3.5070609@behnel.de>

Nathaniel Smith, 05.01.2014 02:07:
> On 4 Jan 2014 22:01, "Stefan Behnel" wrote:
>> Stefan Behnel, 04.01.2014 22:51:
>>> Stefan Behnel, 04.01.2014 22:47:
>>>> Nathaniel Smith, 04.01.2014 18:36:
>>>>>
>>>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>>>>   # numpy without relaxed stride support
>>>>>   raise SkipTest
>>>>
>>>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
>>>
>>> Hmm, but this doesn't seem to work for me in older NumPy versions, although
>>> the original test used to work there. Should we explicitly test for NumPy
>>> 1.8+ as well?
>>
>> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>
> No, I'm missing something now; AFAIK there are only two numpy behaviors:
> with relaxed strides and without relaxed strides, and version number should
> be irrelevant beyond that. What's different between
> 1.8-without-relaxed-strides and 1.7 that makes the test break?

Mark would certainly know better than me.

In any case, the test works with both NumPy 1.7 (tested with Py 2.x and
3.[12] on Jenkins) and NumPy 1.8 with relaxed strides support, but not with
NumPy 1.8 without relaxed strides. The last two were tested in Py3.3 only,
in case that matters. I also tested it locally now (in 3.3) and your
snippet successfully distinguishes the two builds for me, but the test
starts to fail when I disable relaxed strides in NumPy and works when it's
enabled.

Stefan


From stefan_ml at behnel.de  Sun Jan  5 09:26:51 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jan 2014 09:26:51 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C911C3.5070609@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
 <52C884BA.4040500@behnel.de>
 <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>
 <52C911C3.5070609@behnel.de>
Message-ID: <52C9174B.3070000@behnel.de>

Stefan Behnel, 05.01.2014 09:03:
> Nathaniel Smith, 05.01.2014 02:07:
>> On 4 Jan 2014 22:01, "Stefan Behnel" wrote:
>>> Stefan Behnel, 04.01.2014 22:51:
>>>> Stefan Behnel, 04.01.2014 22:47:
>>>>> Nathaniel Smith, 04.01.2014 18:36:
>>>>>>
>>>>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>>>>>   # numpy without relaxed stride support
>>>>>>   raise SkipTest
>>>>>
>>>>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
>>>>
>>>> Hmm, but this doesn't seem to work for me in older NumPy versions, although
>>>> the original test used to work there. Should we explicitly test for NumPy
>>>> 1.8+ as well?
>>>
>>> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>>
>> No, I'm missing something now; AFAIK there are only two numpy behaviors:
>> with relaxed strides and without relaxed strides, and version number should
>> be irrelevant beyond that. What's different between
>> 1.8-without-relaxed-strides and 1.7 that makes the test break?
> 
> Mark would certainly know better than me.
> 
> In any case, the test works with both NumPy 1.7 (tested with Py 2.x and
> 3.[12] on Jenkins) and NumPy 1.8 with relaxed strides support, but not with
> NumPy 1.8 without relaxed strides. The last two were tested in Py3.3 only,
> in case that matters. I also tested it locally now (in 3.3) and your
> snippet successfully distinguishes the two builds for me, but the test
> starts to fail when I disable relaxed strides in NumPy and works when it's
> enabled.

I should add that this

	np.ones((10, 1), order="C").flags.f_contiguous

returns False on NumPy 1.7 and only True on 1.8 with relaxed strides, thus
the additional version test for <1.8.

Stefan


From Gerard.Brunick at constellation.com  Tue Jan  7 00:18:01 2014
From: Gerard.Brunick at constellation.com (Brunick, Gerard:(Constellation))
Date: Mon, 6 Jan 2014 23:18:01 +0000
Subject: [Cython] Possible bug in Python string to C++ string conversion
Message-ID: <71E0ECF7BE49E84CBD47914C9E19FFED548E9D@exchm-omf-22.exelonds.com>

The following extension module crashes when passed a non-string argument rather than throwing a TypeError:

************* string_bug.pyx *****************
from libcpp.string cimport string

cdef extern from "stdio.h":
   int printf(char *format, ...) nogil

def blow_up(string text):
    printf(text.c_str())

*****************************************

To see this run

************* show_bug.py *****************
import string_bug

# This is O.K.
string_bug.blow_up("Testing...")

# This seg faults for me.
string_bug.blow_up(1)
*****************************************

Here is my setup.py for your convenience:

************** setup.py *******************
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(cmdclass={'build_ext': build_ext},
      ext_modules=[Extension("string_bug",
                             sources=["string_bug.pyx"],
                             language="c++")])
******************************************

Also, I have:

> python
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import Cython
>>> Cython.__version__
'0.19.1'

Cheers,
Gerard
This e-mail and any attachments are confidential, may contain legal,
professional or other privileged information, and are intended solely for the
addressee. If you are not the intended recipient, do not use the information
in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140106/30b441a5/attachment.html>

From stefan_ml at behnel.de  Tue Jan  7 12:25:20 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jan 2014 12:25:20 +0100
Subject: [Cython] [cython-users] Re: Cython 0.20 beta 1
In-Reply-To: <ccf58b04-a48b-4cd6-95a1-3d7d10a3efef@googlegroups.com>
References: <CADiQ+QC_O9GJLG0dn5g7fj_O18fBWwuY=f-A_bXS09O3CCX+oQ@mail.gmail.com>
 <ccf58b04-a48b-4cd6-95a1-3d7d10a3efef@googlegroups.com>
Message-ID: <52CBE420.1090503@behnel.de>

Hi,

thanks for reporting.

Andreas van Cranenburgh, 07.01.2014 12:02:
> There is a new warning:
> 
>     Non-trivial type declarators in shared declaration.
> 
> It wasn't immediately obvious to me what this means (trivial & shared could 
> be lots of things), but looking at the code it seems to suggest that 
> declarations should be on their own line except for primitive types.

Right. It's meant to prepare a potential switch of the way pointers are
declared, as well as making it less likely to get the declarations wrong. C
is suprisingly relaxed about these things, but Cython shouldn't be.

I guess the message could be improved, though. Maybe add something like
"mix of pointers and values" as a hinting example.


> I also get an error in code with string formatting. Turns out "%s" % foo 
> used to be translated with PyNumber_Remainder, but now it 
> uses __Pyx_PyString_Format with type checking which fails. This is probably 
> an issue in my code (trying to make things work in 2 and 3) but string 
> formatting is not mentioned in the changelog.

I added an entry. Could you provide a code snippet that shows what you were
doing? Just to be sure it's really a problem in your code and not a wrong
assumption in Cython. Optimisations shouldn't break code.

Stefan


From stefan_ml at behnel.de  Tue Jan  7 13:29:15 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jan 2014 13:29:15 +0100
Subject: [Cython] [cython-users] Re: Cython 0.20 beta 1
In-Reply-To: <4d6d8fa4-3881-47c0-b77f-8b652c669264@googlegroups.com>
References: <CADiQ+QC_O9GJLG0dn5g7fj_O18fBWwuY=f-A_bXS09O3CCX+oQ@mail.gmail.com>
 <ccf58b04-a48b-4cd6-95a1-3d7d10a3efef@googlegroups.com>
 <52CBE420.1090503@behnel.de>
 <4d6d8fa4-3881-47c0-b77f-8b652c669264@googlegroups.com>
Message-ID: <52CBF31B.2060302@behnel.de>

Hi,

let's keep this on the cython-devel list.


Andreas van Cranenburgh, 07.01.2014 12:54:
> On Tuesday, January 7, 2014 12:25:20 PM UTC+1, Stefan Behnel wrote:
>> I added an entry. Could you provide a code snippet that shows what you 
>> were 
>> doing? Just to be sure it's really a problem in your code and not a wrong 
>> assumption in Cython. Optimisations shouldn't break code. 
> 
> Here goes:
> 
> $ cat t.pyx
> cdef str foo
> bar = u'bar'
> foo = 'foo%s' % (bar, )
> print(foo)
> $ python -c 'import pyximport; pyximport.install(); import t'  # Cython 
> 0.19.2
> foobar

Interesting. I assume "python" is Py2.x.


> $ python -c 'import pyximport; pyximport.install(); import t'  # Cython 
> 0.20b1
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File 
> "/home/acranenb/.local/lib/python2.7/site-packages/Cython-0.20b1-py2.7-linux-x86_64.egg/pyximport/pyximport.py", 
> line 431, in load_module
>     language_level=self.language_level)
>   File 
> "/home/acranenb/.local/lib/python2.7/site-packages/Cython-0.20b1-py2.7-linux-x86_64.egg/pyximport/pyximport.py", 
> line 210, in load_module
>     mod = imp.load_dynamic(name, so_path)
>   File "t.pyx", line 3, in init t 
> (/home/acranenb/.pyxbld/temp.linux-x86_64-2.7/pyrex/t.c:824)
>     foo = 'foo%s' % (bar, )
> ImportError: Building module t failed: ['TypeError: Expected str, got 
> unicode\n']

Hmm, the problem here is not the string formatting, it's the explicitly
typed "foo". The string formatting expands the plain str formatting string
into a Unicode string in Py2, but that's not a "str" any more, so the
subsequent assignment fails due to Cython's type check. You could simply
your code to this:

    bar = u'bar'
    cdef str foo = bar

(if you do this inside of a function, Cython's type inference should be
able to deduct a compile time error, but it doesn't do that at the global
module level)

I wonder why it doesn't produce an error in Cython 0.19 for you...

Stefan


From stefan_ml at behnel.de  Tue Jan  7 15:09:37 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jan 2014 15:09:37 +0100
Subject: [Cython] [cython-users] Re: Cython 0.20 beta 1
In-Reply-To: <52CBF31B.2060302@behnel.de>
References: <CADiQ+QC_O9GJLG0dn5g7fj_O18fBWwuY=f-A_bXS09O3CCX+oQ@mail.gmail.com>
 <ccf58b04-a48b-4cd6-95a1-3d7d10a3efef@googlegroups.com>
 <52CBE420.1090503@behnel.de>
 <4d6d8fa4-3881-47c0-b77f-8b652c669264@googlegroups.com>
 <52CBF31B.2060302@behnel.de>
Message-ID: <52CC0AA1.1090304@behnel.de>

CC-ing cython-users again, since others might run into this, too.

Stefan Behnel, 07.01.2014 13:29:
> Andreas van Cranenburgh, 07.01.2014 12:54:
>> On Tuesday, January 7, 2014 12:25:20 PM UTC+1, Stefan Behnel wrote:
>>> I added an entry. Could you provide a code snippet that shows what you 
>>> were doing? Just to be sure it's really a problem in your code and not a
>>> wrong assumption in Cython. Optimisations shouldn't break code.
>>
>> Here goes:
>>
>> $ cat t.pyx
>> cdef str foo
>> bar = u'bar'
>> foo = 'foo%s' % (bar, )
>> print(foo)
>> $ python -c 'import pyximport; pyximport.install(); import t'  # Cython 
>> 0.19.2
>> foobar
> 
> Interesting. I assume "python" is Py2.x.
> 
> 
>> $ python -c 'import pyximport; pyximport.install(); import t'  # Cython 
>> 0.20b1
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File 
>> "/home/acranenb/.local/lib/python2.7/site-packages/Cython-0.20b1-py2.7-linux-x86_64.egg/pyximport/pyximport.py", 
>> line 431, in load_module
>>     language_level=self.language_level)
>>   File 
>> "/home/acranenb/.local/lib/python2.7/site-packages/Cython-0.20b1-py2.7-linux-x86_64.egg/pyximport/pyximport.py", 
>> line 210, in load_module
>>     mod = imp.load_dynamic(name, so_path)
>>   File "t.pyx", line 3, in init t 
>> (/home/acranenb/.pyxbld/temp.linux-x86_64-2.7/pyrex/t.c:824)
>>     foo = 'foo%s' % (bar, )
>> ImportError: Building module t failed: ['TypeError: Expected str, got 
>> unicode\n']
> 
> Hmm, the problem here is not the string formatting, it's the explicitly
> typed "foo". The string formatting expands the plain str formatting string
> into a Unicode string in Py2, but that's not a "str" any more, so the
> subsequent assignment fails due to Cython's type check. You could simply
> your code to this:
> 
>     bar = u'bar'
>     cdef str foo = bar
> 
> (if you do this inside of a function, Cython's type inference should be
> able to deduct a compile time error, but it doesn't do that at the global
> module level)
> 
> I wonder why it doesn't produce an error in Cython 0.19 for you...

Tried it, and it seems like it simply doesn't generate a type check at all.
That's clearly wrong and apparently fixed in 0.20, although I don't
remember a change specifically in that corner.

What you can do in Cython 0.20 to make it work with a statically typed
variable, is to use "cdef basestring foo" as a type instead of "str", but
that doesn't work in older Cython versions.

Or, in fact, you could not type it at all. Cython 0.20 should be able to
figure it out all by itself now (as it understands what "some_string %
something_else" actually does) and automatically type the result as basestring.

Stefan


From simon at simonjagoe.com  Wed Jan  8 21:17:08 2014
From: simon at simonjagoe.com (Simon Jagoe)
Date: Wed, 8 Jan 2014 20:17:08 +0000
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
Message-ID: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>

Hi all,

I recently posted to the cython-users list about this problem. This
email is to submita potential patch to fix the issue. The issue is as
follows:


At Enthought we are busy porting Traits
(http://github.com/enthought/traits) from C to Cython.

An issue has come up that that prevents classes inheriting from a
cdef-class from playing nicely with ABCs.  It is type's tp_new that
checks for abstractmethods that have not been implemented, so if type's
tp_new is bypassed, the check is never run and you can instantiate an
abstract base class.

It is equivalent to this simple example below (taken from
http://stackoverflow.com/questions/20432335). If you replace the simple
class A with a Cython cdef class, the effect is the same.


import abc

class A(object):
    def __new__(cls):
        # self = object.__new__(cls)
        return 42

class B(A):
    __metaclass__ = abc.ABCMeta

    @abc.abstractmethod
    def foo(self):
        pass

b = B()  # No exception.


Thanks,
Simon
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-BUG-Call-PyBaseObject_Type.tp_new-instead-of-NewType.patch
Type: text/x-patch
Size: 989 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140108/bc7c5b12/attachment.bin>

From stefan_ml at behnel.de  Thu Jan  9 08:39:41 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 09 Jan 2014 08:39:41 +0100
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
In-Reply-To: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
References: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
Message-ID: <52CE523C.1070706@behnel.de>

Simon Jagoe, 08.01.2014 21:17:
> I recently posted to the cython-users list about this problem. This
> email is to submita potential patch to fix the issue.

Looking at the code in Py3, ISTM that it would be sufficient to do this iff

	(t->tp_flags & Py_TPFLAGS_IS_ABSTRACT) != 0

which should be rare enough.

Obviously requires some user side testing to see if it breaks any other code...

Stefan


From simon at simonjagoe.com  Thu Jan  9 14:04:20 2014
From: simon at simonjagoe.com (Simon Jagoe)
Date: Thu, 9 Jan 2014 13:04:20 +0000
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
In-Reply-To: <52CE523C.1070706@behnel.de>
References: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
 <52CE523C.1070706@behnel.de>
Message-ID: <CAD+CC_C+3FANZzdnWZEVunDsPJOXK0piKFqmaPBPsuMf9+0adw@mail.gmail.com>

Hi Stefan,

> Looking at the code in Py3, ISTM that it would be sufficient to do this iff
>
>         (t->tp_flags & Py_TPFLAGS_IS_ABSTRACT) != 0
>
> which should be rare enough.
>
> Obviously requires some user side testing to see if it breaks any other code...

Thanks for looking at this. I am wondering if there is a reason why the
tp_alloc can't be replaced with the PyBaseObject_Type->tp_new directly?

My understanding is that initially, object's tp_new essentially just
called tp_alloc as Cython does now.  Later the abc feature was added
with the abstract check happening in object's tp_new, while not changing
previous (non-abstract class) behavior.  If Cython were to use this
instead of tp_alloc, it would be forward-compatible with future changes
to object.

Aside from your comment that it would need extensive testing, am I
missing some other problems?

Thanks,
Simon

From stefan_ml at behnel.de  Fri Jan 10 10:11:44 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jan 2014 10:11:44 +0100
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
In-Reply-To: <CAD+CC_C+3FANZzdnWZEVunDsPJOXK0piKFqmaPBPsuMf9+0adw@mail.gmail.com>
References: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
 <52CE523C.1070706@behnel.de>
 <CAD+CC_C+3FANZzdnWZEVunDsPJOXK0piKFqmaPBPsuMf9+0adw@mail.gmail.com>
Message-ID: <52CFB950.20207@behnel.de>

Simon Jagoe, 09.01.2014 14:04:
>> Looking at the code in Py3, ISTM that it would be sufficient to do this iff
>>
>>         (t->tp_flags & Py_TPFLAGS_IS_ABSTRACT) != 0
>>
>> which should be rare enough.
>>
>> Obviously requires some user side testing to see if it breaks any other code...
> 
> Thanks for looking at this. I am wondering if there is a reason why the
> tp_alloc can't be replaced with the PyBaseObject_Type->tp_new directly?

I don't know. At least, it's some unnecessary overhead in almost all cases.
Not sure if that matters, there's enough costly stuff happening at this
point anyway.

ISTM that we'd also have to fix the issue that Nils Bruin found, otherwise
we'd risk even more trouble on user side.

http://thread.gmane.org/gmane.comp.python.cython.user/10267


> My understanding is that initially, object's tp_new essentially just
> called tp_alloc as Cython does now.  Later the abc feature was added
> with the abstract check happening in object's tp_new, while not changing
> previous (non-abstract class) behavior.  If Cython were to use this
> instead of tp_alloc, it would be forward-compatible with future changes
> to object.

Agreed.

Stefan


From stefan_ml at behnel.de  Fri Jan 10 10:52:44 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jan 2014 10:52:44 +0100
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
In-Reply-To: <52CFB950.20207@behnel.de>
References: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
 <52CE523C.1070706@behnel.de>
 <CAD+CC_C+3FANZzdnWZEVunDsPJOXK0piKFqmaPBPsuMf9+0adw@mail.gmail.com>
 <52CFB950.20207@behnel.de>
Message-ID: <52CFC2EC.4050007@behnel.de>

Stefan Behnel, 10.01.2014 10:11:
> Simon Jagoe, 09.01.2014 14:04:
>>> Looking at the code in Py3, ISTM that it would be sufficient to do this iff
>>>
>>>         (t->tp_flags & Py_TPFLAGS_IS_ABSTRACT) != 0
>>>
>>> which should be rare enough.
>>>
>>> Obviously requires some user side testing to see if it breaks any other code...
>>
>> Thanks for looking at this. I am wondering if there is a reason why the
>> tp_alloc can't be replaced with the PyBaseObject_Type->tp_new directly?
> 
> I don't know. At least, it's some unnecessary overhead in almost all cases.
> Not sure if that matters, there's enough costly stuff happening at this
> point anyway.
> 
> ISTM that we'd also have to fix the issue that Nils Bruin found, otherwise
> we'd risk even more trouble on user side.
> 
> http://thread.gmane.org/gmane.comp.python.cython.user/10267
> 
> 
>> My understanding is that initially, object's tp_new essentially just
>> called tp_alloc as Cython does now.  Later the abc feature was added
>> with the abstract check happening in object's tp_new, while not changing
>> previous (non-abstract class) behavior.  If Cython were to use this
>> instead of tp_alloc, it would be forward-compatible with future changes
>> to object.
> 
> Agreed.

Let's see what Jenkins has to say about it.

https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b

Stefan


From simon at simonjagoe.com  Fri Jan 10 11:51:59 2014
From: simon at simonjagoe.com (Simon Jagoe)
Date: Fri, 10 Jan 2014 10:51:59 +0000
Subject: [Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?
In-Reply-To: <52CFC2EC.4050007@behnel.de>
References: <CAD+CC_CHf8u-ArV3nTWf6Y5COBg9OoV_hcC3vN-zr0q6hu+S-A@mail.gmail.com>
 <52CE523C.1070706@behnel.de>
 <CAD+CC_C+3FANZzdnWZEVunDsPJOXK0piKFqmaPBPsuMf9+0adw@mail.gmail.com>
 <52CFB950.20207@behnel.de> <52CFC2EC.4050007@behnel.de>
Message-ID: <CAD+CC_Bf2c+vrFZkR6bq5YkYeh9fR5--7qVQ2jvy8AMJowEFnQ@mail.gmail.com>

Thanks, Stefan. Let me know if I can help in any way. I will track Cython master
while I continue my port from C to Cython.

On 10 January 2014 09:52, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 10.01.2014 10:11:
>> Simon Jagoe, 09.01.2014 14:04:
>>>> Looking at the code in Py3, ISTM that it would be sufficient to do this iff
>>>>
>>>>         (t->tp_flags & Py_TPFLAGS_IS_ABSTRACT) != 0
>>>>
>>>> which should be rare enough.
>>>>
>>>> Obviously requires some user side testing to see if it breaks any other code...
>>>
>>> Thanks for looking at this. I am wondering if there is a reason why the
>>> tp_alloc can't be replaced with the PyBaseObject_Type->tp_new directly?
>>
>> I don't know. At least, it's some unnecessary overhead in almost all cases.
>> Not sure if that matters, there's enough costly stuff happening at this
>> point anyway.
>>
>> ISTM that we'd also have to fix the issue that Nils Bruin found, otherwise
>> we'd risk even more trouble on user side.
>>
>> http://thread.gmane.org/gmane.comp.python.cython.user/10267
>>
>>
>>> My understanding is that initially, object's tp_new essentially just
>>> called tp_alloc as Cython does now.  Later the abc feature was added
>>> with the abstract check happening in object's tp_new, while not changing
>>> previous (non-abstract class) behavior.  If Cython were to use this
>>> instead of tp_alloc, it would be forward-compatible with future changes
>>> to object.
>>
>> Agreed.
>
> Let's see what Jenkins has to say about it.
>
> https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Jan 10 17:30:31 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jan 2014 17:30:31 +0100
Subject: [Cython] [cython-users] which __new__ to use?
In-Reply-To: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
References: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
Message-ID: <52D02027.5000404@behnel.de>

Nils Bruin, 22.11.2013 00:10:
> With the following classes:
> 
> cdef class A(object):
>     cdef int a
>     pass
> cdef class B(object):
>     pass
> 
> and the following multiply inherited classes:
> 
> The problem arises 
> 
> class C(object):
>     pass
> class D(C,B):
>     pass
> class F(B,C):
>     pass
> class E(C,A):
>     pass
> class G(A,C):
>     pass
> 
> I observe:
> 
> >>> D.__new__(D)
> 
> B.__new__(D) is not safe, use object.__new__()
> 
> >>> E.__new__(E)
> 
>  <__main__.E object at 0x6c56488>
> 
> >>> F.__new__(F)
> 
> <__main__.F object at 0x6c4fdd0>
> >>> G.__new__(G)
> <__main__.G object at 0x6c4fdd0>
> 
> 
> As you can see, a problem only arises for D, in which case there is 
> actually NOT a layout problem: when the class A is involved, where it does 
> matter which __new__ is used, that right one (I assume) is chosen in all 
> cases. A little googling suggests that this error might be coming from 
> tp_new_wrapper in Objects/typeobject.c, which tries to do some sanity 
> checks before calling type->tp_new. It is not clear to me, however, why 
> those sanity checks fail for D.__new__(D) and succeed for E.__new__(E)

I came up with this fix:

https://github.com/cython/cython/commit/ec37469dc61e0e24db448e9455b2483797afe936

It passes your test case and doesn't seem to break others so far.

However, this is a bit of a deep change, so I'd like to see some serious
user side testing before I'll happily push for its inclusion in Cython 0.20.

Stefan


From stefan_ml at behnel.de  Fri Jan 10 17:42:15 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jan 2014 17:42:15 +0100
Subject: [Cython] [cython-users] which __new__ to use?
In-Reply-To: <52D02027.5000404@behnel.de>
References: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
 <52D02027.5000404@behnel.de>
Message-ID: <52D022E7.4000500@behnel.de>

Stefan Behnel, 10.01.2014 17:30:
> Nils Bruin, 22.11.2013 00:10:
>> With the following classes:
>>
>> cdef class A(object):
>>     cdef int a
>>     pass
>> cdef class B(object):
>>     pass
>>
>> and the following multiply inherited classes:
>>
>> The problem arises 
>>
>> class C(object):
>>     pass
>> class D(C,B):
>>     pass
>> class F(B,C):
>>     pass
>> class E(C,A):
>>     pass
>> class G(A,C):
>>     pass
>>
>> I observe:
>>
>>>>> D.__new__(D)
>>
>> B.__new__(D) is not safe, use object.__new__()
>>
>>>>> E.__new__(E)
>>
>>  <__main__.E object at 0x6c56488>
>>
>>>>> F.__new__(F)
>>
>> <__main__.F object at 0x6c4fdd0>
>>>>> G.__new__(G)
>> <__main__.G object at 0x6c4fdd0>
>>
>>
>> As you can see, a problem only arises for D, in which case there is 
>> actually NOT a layout problem: when the class A is involved, where it does 
>> matter which __new__ is used, that right one (I assume) is chosen in all 
>> cases. A little googling suggests that this error might be coming from 
>> tp_new_wrapper in Objects/typeobject.c, which tries to do some sanity 
>> checks before calling type->tp_new. It is not clear to me, however, why 
>> those sanity checks fail for D.__new__(D) and succeed for E.__new__(E)
> 
> I came up with this fix:
> 
> https://github.com/cython/cython/commit/ec37469dc61e0e24db448e9455b2483797afe936
> 
> It passes your test case and doesn't seem to break others so far.
> 
> However, this is a bit of a deep change, so I'd like to see some serious
> user side testing before I'll happily push for its inclusion in Cython 0.20.

And, not entirely unexpected, it breaks Sage pretty much completely:

https://sage.math.washington.edu:8091/hudson/job/sage-tests/1645/consoleFull

Stefan


From stefan_ml at behnel.de  Fri Jan 10 20:50:09 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jan 2014 20:50:09 +0100
Subject: [Cython] [cython-users] which __new__ to use?
In-Reply-To: <00f5ba4f-cbf8-416c-b25f-35d25d383753@googlegroups.com>
References: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
 <52D02027.5000404@behnel.de> <52D022E7.4000500@behnel.de>
 <00f5ba4f-cbf8-416c-b25f-35d25d383753@googlegroups.com>
Message-ID: <52D04EF1.3060400@behnel.de>

[this should have gone to cython-devel - I *hate* those auto-reply-to
headers...]

Nils Bruin, 10.01.2014 18:55:
> On Friday, January 10, 2014 8:42:15 AM UTC-8, Stefan Behnel wrote:
>> And, not entirely unexpected, it breaks Sage pretty much completely: 
> Hm, the 
> DeprecationWarning: object() takes no parameters

Note how it says "object() takes no parameters" and not "object.__new__()"
or "object.__init__()", as in the error messages in typeobject.c. I
couldn't find such a message in CPython's sources or in Cython's sources,
so I wonder where it comes from.

Stefan


From stefan_ml at behnel.de  Sat Jan 11 06:46:17 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 11 Jan 2014 06:46:17 +0100
Subject: [Cython] [cython-users] which __new__ to use?
In-Reply-To: <92579f97-1486-4c5c-a2b1-6b9baad97e59@googlegroups.com>
References: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
 <52D02027.5000404@behnel.de> <52D022E7.4000500@behnel.de>
 <00f5ba4f-cbf8-416c-b25f-35d25d383753@googlegroups.com>
 <52D04E7D.4080309@behnel.de>
 <92579f97-1486-4c5c-a2b1-6b9baad97e59@googlegroups.com>
Message-ID: <52D0DAA9.2020402@behnel.de>

Nils Bruin, 11.01.2014 00:38:
> In Objects/typeobject.c:2895 I find:
> 
> static PyObject *
> object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
> {
>     int err = 0;
>     if (excess_args(args, kwds)) {
>         if (type->tp_new != object_new &&
>             type->tp_init != object_init)
>         {
>             err = PyErr_WarnEx(PyExc_DeprecationWarning,
>                        "object() takes no parameters",
>                        1);
>         }
>         else if (type->tp_new != object_new ||
>                  type->tp_init == object_init)
>         {
>             PyErr_SetString(PyExc_TypeError,
>                 "object() takes no parameters");
>             err = -1;
>         }
>     }
>     if (err < 0)
>         return NULL;

Ah, ok, that's newer than my Py2.7 copy then. Should have updated that ages
ago...


> so it would seem that the objects in sage have both tp_new and tp_init 
> overridden (and warnings are enabled). I'm puzzled how `object_new` can 
> even end up being called then.

Because of this change, I guess:

http://thread.gmane.org/gmane.comp.python.cython.devel/15140/focus=15145

Stefan


From robertwb at gmail.com  Sun Jan 12 08:53:08 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 11 Jan 2014 23:53:08 -0800
Subject: [Cython] Cython 0.20 beta 2
Message-ID: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>

There was a lot of bugfixes and other good stuff on master, so I
pulled it in and am releasing another beta. You can find it at
http://cython.org/release/Cython-0.20b2.tar.gz . I don't anticipate
merging master again before the release, so this is nearly a release
candidate, but please let us know if you run into any regressions.

- Robert

From stefan_ml at behnel.de  Sun Jan 12 09:20:11 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 12 Jan 2014 09:20:11 +0100
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
Message-ID: <52D2503B.7030107@behnel.de>

Robert Bradshaw, 12.01.2014 08:53:
> There was a lot of bugfixes and other good stuff on master, so I
> pulled it in and am releasing another beta. You can find it at
> http://cython.org/release/Cython-0.20b2.tar.gz . I don't anticipate
> merging master again before the release, so this is nearly a release
> candidate, but please let us know if you run into any regressions.

The only thing that is still a bit unclear to me is whether calling
object.__new__() from extension types was a good idea:

https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b

It means that we are now running a couple of safety checks inside of
CPython on object instantiation that validate the arguments being passed
into __init__() and __new__(), because object() doesn't expect any.

http://thread.gmane.org/gmane.comp.python.cython.user/10267/focus=10519

If we want to play safe for the release, we could restrict this change only
to abstract classes for now (for which we know that CPython will generate
an error). That was the original intention of the change anyway.

Stefan


From stefan_ml at behnel.de  Sun Jan 12 10:21:16 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 12 Jan 2014 10:21:16 +0100
Subject: [Cython] tp_alloc() vs. object.__new__() (was: Cython 0.20 beta 2)
In-Reply-To: <52D2503B.7030107@behnel.de>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D2503B.7030107@behnel.de>
Message-ID: <52D25E8C.2040104@behnel.de>

Stefan Behnel, 12.01.2014 09:20:
> The only thing that is still a bit unclear to me is whether calling
> object.__new__() from extension types was a good idea:
> 
> https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b
> 
> It means that we are now running a couple of safety checks inside of
> CPython on object instantiation that validate the arguments being passed
> into __init__() and __new__(), because object() doesn't expect any.
> 
> http://thread.gmane.org/gmane.comp.python.cython.user/10267/focus=10519
> 
> If we want to play safe for the release, we could restrict this change only
> to abstract classes for now (for which we know that CPython will generate
> an error). That was the original intention of the change anyway.

Proposed patch push to master for testing.

https://github.com/cython/cython/commit/1a56a04bfe1b8460ecf3886c3de9268cb5e0c93a

Stefan


From robertwb at gmail.com  Sun Jan 12 10:32:47 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sun, 12 Jan 2014 01:32:47 -0800
Subject: [Cython] tp_alloc() vs. object.__new__() (was: Cython 0.20 beta
	2)
In-Reply-To: <52D25E8C.2040104@behnel.de>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D2503B.7030107@behnel.de> <52D25E8C.2040104@behnel.de>
Message-ID: <CADiQ+QDahN4pcvvgmtMwbXm-0sA2feKQS15CHXbmT6e0fh_Vpw@mail.gmail.com>

On Sun, Jan 12, 2014 at 1:21 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 12.01.2014 09:20:
>> The only thing that is still a bit unclear to me is whether calling
>> object.__new__() from extension types was a good idea:
>>
>> https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b
>>
>> It means that we are now running a couple of safety checks inside of
>> CPython on object instantiation that validate the arguments being passed
>> into __init__() and __new__(), because object() doesn't expect any.
>>
>> http://thread.gmane.org/gmane.comp.python.cython.user/10267/focus=10519
>>
>> If we want to play safe for the release, we could restrict this change only
>> to abstract classes for now (for which we know that CPython will generate
>> an error). That was the original intention of the change anyway.
>
> Proposed patch push to master for testing.
>
> https://github.com/cython/cython/commit/1a56a04bfe1b8460ecf3886c3de9268cb5e0c93a

Yeah, that might be a bit safer.

FWIW, I did see that you rolled back the earlier code in this area to
get Sage passing again (and also hooked up Jenkins to the latest
(git-based) release of Sage), and all looked good there.

- Robert

From stefan_ml at behnel.de  Sun Jan 12 13:51:37 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 12 Jan 2014 13:51:37 +0100
Subject: [Cython] [cython-users] which __new__ to use?
In-Reply-To: <52D0DAA9.2020402@behnel.de>
References: <0f53904b-9af9-436d-8c51-b6963cca154f@googlegroups.com>
 <52D02027.5000404@behnel.de> <52D022E7.4000500@behnel.de>
 <00f5ba4f-cbf8-416c-b25f-35d25d383753@googlegroups.com>
 <52D04E7D.4080309@behnel.de>
 <92579f97-1486-4c5c-a2b1-6b9baad97e59@googlegroups.com>
 <52D0DAA9.2020402@behnel.de>
Message-ID: <52D28FD9.5010103@behnel.de>

Stefan Behnel, 11.01.2014 06:46:
> Nils Bruin, 11.01.2014 00:38:
>> In Objects/typeobject.c:2895 I find:
>>
>> static PyObject *
>> object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
>> {
>>     int err = 0;
>>     if (excess_args(args, kwds)) {
>>         if (type->tp_new != object_new &&
>>             type->tp_init != object_init)
>>         {
>>             err = PyErr_WarnEx(PyExc_DeprecationWarning,
>>                        "object() takes no parameters",
>>                        1);
>>         }
>>         else if (type->tp_new != object_new ||
>>                  type->tp_init == object_init)
>>         {
>>             PyErr_SetString(PyExc_TypeError,
>>                 "object() takes no parameters");
>>             err = -1;
>>         }
>>     }
>>     if (err < 0)
>>         return NULL;
>> 
>> so it would seem that the objects in sage have both tp_new and tp_init 
>> overridden (and warnings are enabled). I'm puzzled how `object_new` can 
>> even end up being called then.
> 
> Because of this change, I guess:
> 
> http://thread.gmane.org/gmane.comp.python.cython.devel/15140/focus=15145

The corresponding Py3 code looks like this:

    if (excess_args(args, kwds) &&
        (type->tp_init == object_init || type->tp_new != object_new)) {
        PyErr_SetString(PyExc_TypeError, "object() takes no parameters");
        return NULL;
    }

My change would set "type->tp_new = object_new" if possible, which means
that these checks no longer hit in Py2, unless the __init__ code calls up
into object.__init__() with non-empty arguments, which has a similar check.
That kind of code would be broken, though.

But they would hit in Py3 if __init__() is not implemented. This might be
the case for code that uses only the fast "X.__new__(X, some, args)" way of
instantiation for some internal type that does not implement an __init__(),
because it knows that it's not being used anyway.

One could argue that such code is incorrect, even though it currently
works. One could also argue that such code that would be broken by my
change is most likely not common at all, because the change applies only to
types that have no attributes or C methods themselves. Instantiating those
by passing in arguments can't be a very common case.

However, considering that subtypes "innocently" call up their supertype
tp_new() chain, it's not obvious to me what other harmful combinations
there may be.

All in all, I think this has a potential to break stuff, while, at the same
time, it doesn't fix a regression, nor does it fix a serious problem that
users can't work around. So my change will not go into 0.20 (and I've
reverted it in master already), and it will then need some convincing
argumentation to get reapplied in the future, including a good series of
test cases that cover the safety checks in CPython.

Stefan


From jtaylor.debian at googlemail.com  Sun Jan 12 20:35:57 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 12 Jan 2014 20:35:57 +0100
Subject: [Cython] libimobiledevice build broken by 0.20b2
Message-ID: <52D2EE9D.4050702@googlemail.com>

Hi,
libimobiledevice cython file fails to build with cython git head and 0.20b2.
It works with older versions, the bad commit in cython is:
commit db5d5a41b852fa32876688198e3d2d46f12de858
Author: Robert Bradshaw <robertwb at gmail.com>
Date:   Mon Dec 30 22:24:04 2013 -0800

    Fix bug when base classes were declared out-of-order.


its parent commit succeeds the build.


the file is:
https://github.com/libimobiledevice/libimobiledevice/blob/master/cython/imobiledevice.pyx

build can be run with
./autogen.sh
./configure --disable-openssl
make CYTHON=/path/to/cython

imobiledevice.c:13521:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13644:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13783:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13906:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14029:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14345:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14468:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14591:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14791:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:15046:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:26762:89: error: 'struct
__pyx_obj_13imobiledevice_ScreenshotrClient' has no member named
'__pyx_vtab'
imobiledevice.c:53310:4: error: 'struct
__pyx_obj_13imobiledevice_ScreenshotrClient' has no member named
'__pyx_vtab'
imobiledevice.c:53733:4: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'


Note, I know neither cython nor imobiledevice, Yaroslav Halchenko just
encountered this issue while testing packages with 0.20b2 in Debian.
Let me know if you need more information.

From jtaylor.debian at googlemail.com  Sun Jan 12 21:12:18 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 12 Jan 2014 21:12:18 +0100
Subject: [Cython] h5py build broken by 0.20b2
Message-ID: <52D2F722.2000703@googlemail.com>

Hi,
a h5py cython file fails to build with cython git head and 0.20b2.
It works with older versions.
This file fails

The offending file seems to be:
https://github.com/h5py/h5py/blob/master/h5py/h5p.pyd

cython h5py/h5p.pyx
Error compiling Cython file:
------------------------------------------------------------
...
from _objects cimport ObjectID

# --- Base classes ---

cdef class PropID(ObjectID):
    """ Base class for all property lists """
   ^
------------------------------------------------------------
h5py/h5p.pxd:17:4: Executable statement not allowed here
...

it was introduced in cython around this commit:
b6b5152f386ddae503674cc26200a547f3b4c8b0
properly handle expressions at the beginning of func/class/etc. blocks


Is this an intentional change? I did not see anything related in the
CHANGES.rst of the 0.20.x branch.
Please advise if this is a bug/regression in cython or
bug/required-change in h5py.


Note, I know neither cython nor h5py, Yaroslav Halchenko just
encountered this issue while testing packages with 0.20b2 in Debian.
Let me know if you need more information.

From stefan_ml at behnel.de  Mon Jan 13 09:26:04 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Jan 2014 09:26:04 +0100
Subject: [Cython] h5py build broken by 0.20b2
In-Reply-To: <52D2F722.2000703@googlemail.com>
References: <52D2F722.2000703@googlemail.com>
Message-ID: <52D3A31C.9060708@behnel.de>

Hi,

thanks for the report!

Julian Taylor, 12.01.2014 21:12:
> a h5py cython file fails to build with cython git head and 0.20b2.
> It works with older versions.
> This file fails
> 
> The offending file seems to be:
> https://github.com/h5py/h5py/blob/master/h5py/h5p.pyd
> 
> cython h5py/h5p.pyx
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> from _objects cimport ObjectID
> 
> # --- Base classes ---
> 
> cdef class PropID(ObjectID):
>     """ Base class for all property lists """
>    ^
> ------------------------------------------------------------
> h5py/h5p.pxd:17:4: Executable statement not allowed here
> ...
> 
> it was introduced in cython around this commit:
> b6b5152f386ddae503674cc26200a547f3b4c8b0
> properly handle expressions at the beginning of func/class/etc. blocks
> 
> Is this an intentional change?

Well, yes, but apparently one with unintended side-effects. :)

I pushed a proposed quick-fix here, test case pending, so it would be great
if you or Yaroslav could give it a try.

https://github.com/cython/cython/commit/279efc20fb8fb0bbcba5922afd010bdd3424e3b5

Stefan


From cgohlke at uci.edu  Mon Jan 13 09:46:30 2014
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Mon, 13 Jan 2014 00:46:30 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
Message-ID: <52D3A7E6.3050109@uci.edu>

On 1/11/2014 11:53 PM, Robert Bradshaw wrote:
> There was a lot of bugfixes and other good stuff on master, so I
> pulled it in and am releasing another beta. You can find it at
> http://cython.org/release/Cython-0.20b2.tar.gz . I don't anticipate
> merging master again before the release, so this is nearly a release
> candidate, but please let us know if you run into any regressions.
>
> - Robert

Hello,

I have trouble running the tests (`runtests.py`) on Windows:

The script does not get beyond `runTest (__main__.EndToEndTest) 
End-to-end common_include_dir ...`.
Setting `nthreads = 0` in `tests/build/common_include_dir.srctree` gets 
around this but the test fails. The file also uses `ls` and `grep` which 
are not commonly found on Windows.

The file `tests/build/build_dir.srctree` uses `os.symlink()`, which is 
not available on Windows 
<http://docs.python.org/2/library/os.html#os.symlink>.

There are hundreds of test errors, most of kind `IOError: [Errno 24] Too 
many open files`.
The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at 
<http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>. 


Christoph

From stefan_ml at behnel.de  Mon Jan 13 11:24:07 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Jan 2014 11:24:07 +0100
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D3A7E6.3050109@uci.edu>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu>
Message-ID: <52D3BEC7.1050700@behnel.de>

Hi Christoph,

thanks for testing!

Christoph Gohlke, 13.01.2014 09:46:
> I have trouble running the tests (`runtests.py`) on Windows:
> [...]
> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
> many open files`.

The tests start consistently failing at some point, which hints at a file
handle leak, either in the test runner or in the compiler.


> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.

Yes, lots of problems and warnings. Might take a bit to look through all of
them.

Regarding this one:

"""
compiling (cpp) and running final_method_T586 ... final_method_T586.c
final_method_T586.c(8) : fatal error C1083: Cannot open include file:
'pyconfig.h': No such file or directory
ERROR
"""

Is there no "pyconfig.h" amongst the CPython header files on Windows?

Stefan


From jtaylor.debian at googlemail.com  Mon Jan 13 19:39:13 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 13 Jan 2014 19:39:13 +0100
Subject: [Cython] remove timestamps from generated source files
Message-ID: <52D432D1.5040405@googlemail.com>

Hi,
Cython currently places timestamps into all its generated C source files.
This may be ok for developers who normally only rebuild files they
changed anyway but it is a major annoyance for distribution packagers
who often have to rebuild software from scratch including re-cythonizing.
The reason this is annoying is that the adding of timestamps breaks
caching of compiler results e.g. with ccache. Due to the size of the
generated files recompiling the full sources normally takes a large
fraction of the build time of packages.

I'm not sure what purpose the timestamps really serve, the version
number also included should be sufficient information in order to deduce
its origin.

Would it be possible to change Cython to stop putting timestamps into
the source?
I would also be happy if the granularity is reduced from seconds to
hours/days or an non-default option to disable it (e.g. an environment
variable).


Additionally I think it might be useful to have a vendor id added to the
version number added to the source files. This would allow easier origin
tracking of files created with distribution patched versions of Cython.
E.g. if Debian patches 0.20 of Cython it puts in

 Generated by Cython 0.20 (Debian revision 3)

Where the vendor id is added via a Distribution patch of the package and
none is emitted for upstream builds.
(CC. Debian maintainer of Cython for comments)

Cheers,
Julian Taylor

From jtaylor.debian at googlemail.com  Mon Jan 13 19:49:02 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 13 Jan 2014 19:49:02 +0100
Subject: [Cython] h5py build broken by 0.20b2
In-Reply-To: <52D3A31C.9060708@behnel.de>
References: <52D2F722.2000703@googlemail.com> <52D3A31C.9060708@behnel.de>
Message-ID: <52D4351E.3080602@googlemail.com>

On 13.01.2014 09:26, Stefan Behnel wrote:
> Hi,
> ...
>>
>> it was introduced in cython around this commit:
>> b6b5152f386ddae503674cc26200a547f3b4c8b0
>> properly handle expressions at the beginning of func/class/etc. blocks
>>
>> Is this an intentional change?
> 
> Well, yes, but apparently one with unintended side-effects. :)
> 
> I pushed a proposed quick-fix here, test case pending, so it would be great
> if you or Yaroslav could give it a try.
> 
> https://github.com/cython/cython/commit/279efc20fb8fb0bbcba5922afd010bdd3424e3b5
> 
> Stefan


No, I still get the same error message with cython master
dab03a3cb796646c80d2f which includes the fix attempt.

From robertwb at gmail.com  Mon Jan 13 22:24:15 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 13 Jan 2014 13:24:15 -0800
Subject: [Cython] remove timestamps from generated source files
In-Reply-To: <52D432D1.5040405@googlemail.com>
References: <52D432D1.5040405@googlemail.com>
Message-ID: <CADiQ+QA3DmpbGg45gq0OHDLdba_6B1qKVaF3VQwaudAQeu65rA@mail.gmail.com>

The timestamp is in a comment which ccache can strip before looking up
items in the cache; are you sure this is an issue?

On Mon, Jan 13, 2014 at 10:39 AM, Julian Taylor
<jtaylor.debian at googlemail.com> wrote:
> Hi,
> Cython currently places timestamps into all its generated C source files.
> This may be ok for developers who normally only rebuild files they
> changed anyway but it is a major annoyance for distribution packagers
> who often have to rebuild software from scratch including re-cythonizing.
> The reason this is annoying is that the adding of timestamps breaks
> caching of compiler results e.g. with ccache. Due to the size of the
> generated files recompiling the full sources normally takes a large
> fraction of the build time of packages.
>
> I'm not sure what purpose the timestamps really serve, the version
> number also included should be sufficient information in order to deduce
> its origin.
>
> Would it be possible to change Cython to stop putting timestamps into
> the source?
> I would also be happy if the granularity is reduced from seconds to
> hours/days or an non-default option to disable it (e.g. an environment
> variable).
>
>
> Additionally I think it might be useful to have a vendor id added to the
> version number added to the source files. This would allow easier origin
> tracking of files created with distribution patched versions of Cython.
> E.g. if Debian patches 0.20 of Cython it puts in
>
>  Generated by Cython 0.20 (Debian revision 3)
>
> Where the vendor id is added via a Distribution patch of the package and
> none is emitted for upstream builds.
> (CC. Debian maintainer of Cython for comments)
>
> Cheers,
> Julian Taylor
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From jtaylor.debian at googlemail.com  Mon Jan 13 22:48:15 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 13 Jan 2014 22:48:15 +0100
Subject: [Cython] remove timestamps from generated source files
In-Reply-To: <CADiQ+QA3DmpbGg45gq0OHDLdba_6B1qKVaF3VQwaudAQeu65rA@mail.gmail.com>
References: <52D432D1.5040405@googlemail.com>
 <CADiQ+QA3DmpbGg45gq0OHDLdba_6B1qKVaF3VQwaudAQeu65rA@mail.gmail.com>
Message-ID: <52D45F1F.1000501@googlemail.com>

On 13.01.2014 22:24, Robert Bradshaw wrote:
> The timestamp is in a comment which ccache can strip before looking up
> items in the cache; are you sure this is an issue?

well this is embarrassing, it indeed strips the comments...
The issue was setup.py used a compiler symlink not present in my ccache
folder. I draw wrong conclusions to quickly ...
Sorry for the noise.

From robertwb at gmail.com  Tue Jan 14 06:32:45 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 13 Jan 2014 21:32:45 -0800
Subject: [Cython] h5py build broken by 0.20b2
In-Reply-To: <52D4351E.3080602@googlemail.com>
References: <52D2F722.2000703@googlemail.com> <52D3A31C.9060708@behnel.de>
 <52D4351E.3080602@googlemail.com>
Message-ID: <CADiQ+QB9dcJoGa_MjFo2ercOjOv7q4WBawh1Gd4vmiLy3AHrOQ@mail.gmail.com>

I've verified that Stefan's latest fix lets h5py compile.

On Mon, Jan 13, 2014 at 10:49 AM, Julian Taylor
<jtaylor.debian at googlemail.com> wrote:
> On 13.01.2014 09:26, Stefan Behnel wrote:
>> Hi,
>> ...
>>>
>>> it was introduced in cython around this commit:
>>> b6b5152f386ddae503674cc26200a547f3b4c8b0
>>> properly handle expressions at the beginning of func/class/etc. blocks
>>>
>>> Is this an intentional change?
>>
>> Well, yes, but apparently one with unintended side-effects. :)
>>
>> I pushed a proposed quick-fix here, test case pending, so it would be great
>> if you or Yaroslav could give it a try.
>>
>> https://github.com/cython/cython/commit/279efc20fb8fb0bbcba5922afd010bdd3424e3b5
>>
>> Stefan
>
>
> No, I still get the same error message with cython master
> dab03a3cb796646c80d2f which includes the fix attempt.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Tue Jan 14 07:18:24 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 13 Jan 2014 22:18:24 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D3BEC7.1050700@behnel.de>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu> <52D3BEC7.1050700@behnel.de>
Message-ID: <CADiQ+QABtuxDAknesDHj=ZxV0kEO3ucBTuRTyYPiK8uKW4R3nQ@mail.gmail.com>

I fixed one of the tests that was trying to test using symlink:

https://github.com/cython/cython/commit/f236077786203f5f393dab3a707b82a62b4f4155

Is the list of failures notably worse than 0.19.x?

- Robert



On Mon, Jan 13, 2014 at 2:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Christoph,
>
> thanks for testing!
>
> Christoph Gohlke, 13.01.2014 09:46:
>> I have trouble running the tests (`runtests.py`) on Windows:
>> [...]
>> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
>> many open files`.
>
> The tests start consistently failing at some point, which hints at a file
> handle leak, either in the test runner or in the compiler.
>
>
>> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
>> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.
>
> Yes, lots of problems and warnings. Might take a bit to look through all of
> them.
>
> Regarding this one:
>
> """
> compiling (cpp) and running final_method_T586 ... final_method_T586.c
> final_method_T586.c(8) : fatal error C1083: Cannot open include file:
> 'pyconfig.h': No such file or directory
> ERROR
> """
>
> Is there no "pyconfig.h" amongst the CPython header files on Windows?
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Tue Jan 14 07:19:51 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 13 Jan 2014 22:19:51 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D2503B.7030107@behnel.de>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D2503B.7030107@behnel.de>
Message-ID: <CADiQ+QDq5pP3yNQRpVv=h7N=NMggamnB-M8LaQUYUkc8Y3eSsA@mail.gmail.com>

On Sun, Jan 12, 2014 at 12:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 12.01.2014 08:53:
>> There was a lot of bugfixes and other good stuff on master, so I
>> pulled it in and am releasing another beta. You can find it at
>> http://cython.org/release/Cython-0.20b2.tar.gz . I don't anticipate
>> merging master again before the release, so this is nearly a release
>> candidate, but please let us know if you run into any regressions.
>
> The only thing that is still a bit unclear to me is whether calling
> object.__new__() from extension types was a good idea:
>
> https://github.com/cython/cython/commit/ddaffbe78c06e580ca0d9cd334c7a28a4de5c40b
>
> It means that we are now running a couple of safety checks inside of
> CPython on object instantiation that validate the arguments being passed
> into __init__() and __new__(), because object() doesn't expect any.
>
> http://thread.gmane.org/gmane.comp.python.cython.user/10267/focus=10519
>
> If we want to play safe for the release, we could restrict this change only
> to abstract classes for now (for which we know that CPython will generate
> an error). That was the original intention of the change anyway.

Yes, I'm going to "play it safe" and include the restrict-to-abstract change.

From robertwb at gmail.com  Tue Jan 14 09:20:40 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 14 Jan 2014 00:20:40 -0800
Subject: [Cython] libimobiledevice build broken by 0.20b2
In-Reply-To: <52D2EE9D.4050702@googlemail.com>
References: <52D2EE9D.4050702@googlemail.com>
Message-ID: <CADiQ+QA8_sqbnyxE7WCyUf4fYUe6Lyk+nr=XS-oH1MdcR89YMg@mail.gmail.com>

On Sun, Jan 12, 2014 at 11:35 AM, Julian Taylor
<jtaylor.debian at googlemail.com> wrote:
> Hi,
> libimobiledevice cython file fails to build with cython git head and 0.20b2.
> It works with older versions, the bad commit in cython is:
> commit db5d5a41b852fa32876688198e3d2d46f12de858
> Author: Robert Bradshaw <robertwb at gmail.com>
> Date:   Mon Dec 30 22:24:04 2013 -0800
>
>     Fix bug when base classes were declared out-of-order.
>
>
> its parent commit succeeds the build.
>
>
> the file is:
> https://github.com/libimobiledevice/libimobiledevice/blob/master/cython/imobiledevice.pyx
>
> build can be run with
> ./autogen.sh
> ./configure --disable-openssl
> make CYTHON=/path/to/cython
>
> imobiledevice.c:13521:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:13644:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:13783:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:13906:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:14029:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:14345:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:14468:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:14591:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:14791:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:15046:88: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
> imobiledevice.c:26762:89: error: 'struct
> __pyx_obj_13imobiledevice_ScreenshotrClient' has no member named
> '__pyx_vtab'
> imobiledevice.c:53310:4: error: 'struct
> __pyx_obj_13imobiledevice_ScreenshotrClient' has no member named
> '__pyx_vtab'
> imobiledevice.c:53733:4: error: 'struct
> __pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
>
>
> Note, I know neither cython nor imobiledevice, Yaroslav Halchenko just
> encountered this issue while testing packages with 0.20b2 in Debian.
> Let me know if you need more information.

Thanks for the report! Looks like it's a bunch of pxi files with class
definitions that get included, fixed via
https://github.com/cython/cython/commit/27a95b9bdcdc45e95209ec92e546c0e439ecd2c9

(FWIW, I never was able to get this package to compile, but the
generated code looks better.)

From jtaylor.debian at googlemail.com  Tue Jan 14 09:58:37 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Tue, 14 Jan 2014 09:58:37 +0100
Subject: [Cython] h5py build broken by 0.20b2
In-Reply-To: <CADiQ+QB9dcJoGa_MjFo2ercOjOv7q4WBawh1Gd4vmiLy3AHrOQ@mail.gmail.com>
References: <52D2F722.2000703@googlemail.com> <52D3A31C.9060708@behnel.de>
 <52D4351E.3080602@googlemail.com>
 <CADiQ+QB9dcJoGa_MjFo2ercOjOv7q4WBawh1Gd4vmiLy3AHrOQ@mail.gmail.com>
Message-ID: <CAK5FAtFn-tCm=eBQk8PyieAGjjqwYyt+P2P3AL7cEGdv0KT6xw@mail.gmail.com>

On Tue, Jan 14, 2014 at 6:32 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

> I've verified that Stefan's latest fix lets h5py compile.
>
>
confirmed, that the new fix works, I guess it is:
https://github.com/cython/cython/commit/714f5e86b23fee295bb6a05b83a6d1031e0439d7
thanks

there are a bunch of uninitialized use and set-but-unused warnings, but
they are also present in 0.19.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140114/85b6b590/attachment.html>

From jtaylor.debian at googlemail.com  Tue Jan 14 09:59:59 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Tue, 14 Jan 2014 09:59:59 +0100
Subject: [Cython] libimobiledevice build broken by 0.20b2
In-Reply-To: <CADiQ+QA8_sqbnyxE7WCyUf4fYUe6Lyk+nr=XS-oH1MdcR89YMg@mail.gmail.com>
References: <52D2EE9D.4050702@googlemail.com>
 <CADiQ+QA8_sqbnyxE7WCyUf4fYUe6Lyk+nr=XS-oH1MdcR89YMg@mail.gmail.com>
Message-ID: <CAK5FAtGv3tw6QTjeYxKGG+BxwSfPFe8ghbtk_t3iLMRPfkkEiQ@mail.gmail.com>

On Tue, Jan 14, 2014 at 9:20 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

>
>
> Thanks for the report! Looks like it's a bunch of pxi files with class
> definitions that get included, fixed via
>
> https://github.com/cython/cython/commit/27a95b9bdcdc45e95209ec92e546c0e439ecd2c9
>
>
>
confirmed that the cython build now works, thanks
This fix is not in master only the 0.20.x branch. Currently master still
fails.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140114/3bc105c6/attachment.html>

From stefan_ml at behnel.de  Tue Jan 14 10:05:01 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Jan 2014 10:05:01 +0100
Subject: [Cython] libimobiledevice build broken by 0.20b2
In-Reply-To: <CAK5FAtGv3tw6QTjeYxKGG+BxwSfPFe8ghbtk_t3iLMRPfkkEiQ@mail.gmail.com>
References: <52D2EE9D.4050702@googlemail.com>
 <CADiQ+QA8_sqbnyxE7WCyUf4fYUe6Lyk+nr=XS-oH1MdcR89YMg@mail.gmail.com>
 <CAK5FAtGv3tw6QTjeYxKGG+BxwSfPFe8ghbtk_t3iLMRPfkkEiQ@mail.gmail.com>
Message-ID: <52D4FDBD.5090302@behnel.de>

Julian Taylor, 14.01.2014 09:59:
> On Tue, Jan 14, 2014 at 9:20 AM, Robert Bradshaw wrote:
>> Thanks for the report! Looks like it's a bunch of pxi files with class
>> definitions that get included, fixed via
>>
>> https://github.com/cython/cython/commit/27a95b9bdcdc45e95209ec92e546c0e439ecd2c9
>
> confirmed that the cython build now works, thanks

Thanks for testing.


> This fix is not in master only the 0.20.x branch. Currently master still
> fails.

I think the idea is that there will be no more merges from master to 0.20
before the release, but 0.20 will end up getting merged back into master.

Stefan


From robertwb at gmail.com  Thu Jan 16 10:40:46 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 16 Jan 2014 01:40:46 -0800
Subject: [Cython] Cython 0.20 release candidates
Message-ID: <CADiQ+QAyDTL1nn4j_pVzhcM6wA5GsHEccTayXMtA3xWq4J0AUQ@mail.gmail.com>

Up at http://cython.org/release/Cython-0.20rc1.tar.gz , to be released shortly.

From byrnes at wildpumpkin.net  Thu Jan 16 18:43:58 2014
From: byrnes at wildpumpkin.net (Robert Byrnes)
Date: Thu, 16 Jan 2014 12:43:58 -0500
Subject: [Cython] assignment to reference argument bug, revisited
Message-ID: <2fb6ca219914f9e71dcc5e5313153d92.squirrel@emailmg.dotster.com>

Are there any plans to fix the "Assignment to reference" bug mentioned here?

https://groups.google.com/forum/?fromgroups=#!topic/cython-users/j58Sp3QMrD4

> Oh, yes, of course. Good old C++, where local T& x is unassignable but
> assigning to an argument T& x is a common idiom. This is a bug in Cython.

This is still broken in 0.19.2, and a casual inspection of 0.20rc1
suggests that it is broken there as well.

Is the fix as simple as this?

--- ExprNodes.py~       2014-01-16 12:31:08.377573000 -0500
+++ ExprNodes.py        2014-01-16 12:30:56.945501000 -0500
@@ -1605,7 +1605,7 @@ class NameNode(AtomicExprNode):

         if self.type.is_const:
             error(self.pos, "Assignment to const '%s'" % self.name)
-        if self.type.is_reference:
+        if self.type.is_reference and not self.entry.is_arg:
             error(self.pos, "Assignment to reference '%s'" % self.name)
         if not self.is_lvalue():
             error(self.pos, "Assignment to non-lvalue '%s'"

This seems to work for me, but ...

I wonder if it is also correct to set self.entry.cf_unused = True
for the case of assignment to reference arguments, to avoid CYTHON_UNUSED
annotations in the function definitions and unused variable warnings.

--
Bob



From cgohlke at uci.edu  Fri Jan 17 03:21:34 2014
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Thu, 16 Jan 2014 18:21:34 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D3BEC7.1050700@behnel.de>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu> <52D3BEC7.1050700@behnel.de>
Message-ID: <52D893AE.2020609@uci.edu>

On 1/13/2014 2:24 AM, Stefan Behnel wrote:
> Hi Christoph,
>
> thanks for testing!
>
> Christoph Gohlke, 13.01.2014 09:46:
>> I have trouble running the tests (`runtests.py`) on Windows:
>> [...]
>> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
>> many open files`.
>
> The tests start consistently failing at some point, which hints at a file
> handle leak, either in the test runner or in the compiler.
>
>
>> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
>> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.
>
> Yes, lots of problems and warnings. Might take a bit to look through all of
> them.
>
> Regarding this one:
>
> """
> compiling (cpp) and running final_method_T586 ... final_method_T586.c
> final_method_T586.c(8) : fatal error C1083: Cannot open include file:
> 'pyconfig.h': No such file or directory
> ERROR
> """
>
> Is there no "pyconfig.h" amongst the CPython header files on Windows?
>
> Stefan
>

According to Process Explorer 
<http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx>, 
`\Device\ConDrv` (the Windows Console Driver) file handles are leaked 
during `Doctest`. When the number of open file handles reaches about 
2000, all kind of things start to fail, including the compiler not being 
able to open/find files such as 'pyconfig.h'. No file handles are leaked 
when testing Cython 0.19.2.

Christoph

From robertwb at gmail.com  Fri Jan 17 05:53:29 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 16 Jan 2014 20:53:29 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D893AE.2020609@uci.edu>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu> <52D3BEC7.1050700@behnel.de>
 <52D893AE.2020609@uci.edu>
Message-ID: <CADiQ+QAaoUpeLBsACgz+Fe1HAq_HEJDWnMv2YeNQ6cEVk18kOw@mail.gmail.com>

On Thu, Jan 16, 2014 at 6:21 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
> On 1/13/2014 2:24 AM, Stefan Behnel wrote:
>>
>> Hi Christoph,
>>
>> thanks for testing!
>>
>> Christoph Gohlke, 13.01.2014 09:46:
>>>
>>> I have trouble running the tests (`runtests.py`) on Windows:
>>> [...]
>>> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
>>> many open files`.
>>
>>
>> The tests start consistently failing at some point, which hints at a file
>> handle leak, either in the test runner or in the compiler.
>>
>>
>>> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
>>>
>>> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.
>>
>>
>> Yes, lots of problems and warnings. Might take a bit to look through all
>> of
>> them.
>>
>> Regarding this one:
>>
>> """
>> compiling (cpp) and running final_method_T586 ... final_method_T586.c
>> final_method_T586.c(8) : fatal error C1083: Cannot open include file:
>> 'pyconfig.h': No such file or directory
>> ERROR
>> """
>>
>> Is there no "pyconfig.h" amongst the CPython header files on Windows?
>>
>> Stefan
>>
>
> According to Process Explorer
> <http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx>,
> `\Device\ConDrv` (the Windows Console Driver) file handles are leaked during
> `Doctest`. When the number of open file handles reaches about 2000, all kind
> of things start to fail, including the compiler not being able to open/find
> files such as 'pyconfig.h'. No file handles are leaked when testing Cython
> 0.19.2.

No file handles appear to be leaked running under OS X or Linux. Could
you perhaps do a binary search on the git repo to locate the
incriminating CL?

From stefan_ml at behnel.de  Fri Jan 17 07:24:32 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jan 2014 07:24:32 +0100
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <52D893AE.2020609@uci.edu>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu> <52D3BEC7.1050700@behnel.de>
 <52D893AE.2020609@uci.edu>
Message-ID: <52D8CCA0.301@behnel.de>

Christoph Gohlke, 17.01.2014 03:21:
> On 1/13/2014 2:24 AM, Stefan Behnel wrote:
>> Christoph Gohlke, 13.01.2014 09:46:
>>> I have trouble running the tests (`runtests.py`) on Windows:
>>> [...]
>>> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
>>> many open files`.
>>
>> The tests start consistently failing at some point, which hints at a file
>> handle leak, either in the test runner or in the compiler.
>>
>>> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
>>> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.
>
> According to Process Explorer
> <http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx>,
> `\Device\ConDrv` (the Windows Console Driver) file handles are leaked
> during `Doctest`. When the number of open file handles reaches about 2000,
> all kind of things start to fail, including the compiler not being able to
> open/find files such as 'pyconfig.h'. No file handles are leaked when
> testing Cython 0.19.2.

That sounds like it's a problem in the test runner, maybe leaking
subprocesses. On unixish systems, it forks out for each test, but not on
Windows. That makes it likely that you wouldn't notice the problem under
Unix, because the ending child process would clean up the leak. On Windows,
it would add up instead as everything is run by the mainprocess. And since
there are more tests now than in Cython 0.19.x, also more "srctree" tests
that unfold whole directories and run multiple shell commands on them, it's
not impossible that we simply reached the maximum number of file handles
now and stayed a bit below before.

I pushed a couple of fixes to the master branch, but I don't think I found
the problem yet. If you say it's in the doctests, then we'll have to dig
over there a bit more.

Stefan


From robertwb at gmail.com  Sun Jan 19 00:16:27 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 18 Jan 2014 15:16:27 -0800
Subject: [Cython] Cython 0.20 release
Message-ID: <CADiQ+QD4dbrjJV-ZDiKgSm=MXE+xMZGYpnfihtk1C6rwiP2U6w@mail.gmail.com>

I'm happy to announce the release of Cython 0.20.

You can download it directly from the Cython site at
http://cython.org/ or from PyPI at
https://pypi.python.org/pypi/Cython/0.20 .

There are a significant number of new features and bug fixes in this
release, for a summary see
https://github.com/cython/cython/blob/8d4d615c49e858fc61663bd3b335ffde593c2840/CHANGES.rst
. Documentation is up at http://docs.cython.org/0.20/.

Many thanks to all those who contributed.

- Robert

From cgohlke at uci.edu  Sun Jan 19 00:46:20 2014
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 18 Jan 2014 15:46:20 -0800
Subject: [Cython] Cython 0.20 beta 2
In-Reply-To: <CADiQ+QAaoUpeLBsACgz+Fe1HAq_HEJDWnMv2YeNQ6cEVk18kOw@mail.gmail.com>
References: <CADiQ+QA_kaH4d8bR9v1h2dfYHbLoh5bJaqiFmGZfOSUjk6+S2g@mail.gmail.com>
 <52D3A7E6.3050109@uci.edu> <52D3BEC7.1050700@behnel.de>
 <52D893AE.2020609@uci.edu>
 <CADiQ+QAaoUpeLBsACgz+Fe1HAq_HEJDWnMv2YeNQ6cEVk18kOw@mail.gmail.com>
Message-ID: <52DB124C.3050205@uci.edu>

On 1/16/2014 8:53 PM, Robert Bradshaw wrote:
> On Thu, Jan 16, 2014 at 6:21 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>> On 1/13/2014 2:24 AM, Stefan Behnel wrote:
>>>
>>> Hi Christoph,
>>>
>>> thanks for testing!
>>>
>>> Christoph Gohlke, 13.01.2014 09:46:
>>>>
>>>> I have trouble running the tests (`runtests.py`) on Windows:
>>>> [...]
>>>> There are hundreds of test errors, most of kind `IOError: [Errno 24] Too
>>>> many open files`.
>>>
>>>
>>> The tests start consistently failing at some point, which hints at a file
>>> handle leak, either in the test runner or in the compiler.
>>>
>>>
>>>> The test output for 64 bit Python 2.7 with numpy 1.7.2 and msvc9 is at
>>>>
>>>> <http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/cython/Cython-0.20.x-win-amd64-py2.7_test.txt>.
>>>
>>>
>>> Yes, lots of problems and warnings. Might take a bit to look through all
>>> of
>>> them.
>>>
>>> Regarding this one:
>>>
>>> """
>>> compiling (cpp) and running final_method_T586 ... final_method_T586.c
>>> final_method_T586.c(8) : fatal error C1083: Cannot open include file:
>>> 'pyconfig.h': No such file or directory
>>> ERROR
>>> """
>>>
>>> Is there no "pyconfig.h" amongst the CPython header files on Windows?
>>>
>>> Stefan
>>>
>>
>> According to Process Explorer
>> <http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx>,
>> `\Device\ConDrv` (the Windows Console Driver) file handles are leaked during
>> `Doctest`. When the number of open file handles reaches about 2000, all kind
>> of things start to fail, including the compiler not being able to open/find
>> files such as 'pyconfig.h'. No file handles are leaked when testing Cython
>> 0.19.2.
>
> No file handles appear to be leaked running under OS X or Linux. Could
> you perhaps do a binary search on the git repo to locate the
> incriminating CL?


I did run a git bisect, which came out inconclusive because somehow the 
doctests failed to find/use msvc when run in the git bash environment 
(non-doctests did use msvc).

Christoph

From njs at pobox.com  Sun Jan 19 01:04:26 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 19 Jan 2014 00:04:26 +0000
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C911C3.5070609@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
 <52C884BA.4040500@behnel.de>
 <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>
 <52C911C3.5070609@behnel.de>
Message-ID: <CAPJVwBnrA41YGk9a1m=iva2OdXtNLJHPoMnyyqjedOQkatMACA@mail.gmail.com>

On Sun, Jan 5, 2014 at 8:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Nathaniel Smith, 05.01.2014 02:07:
>> On 4 Jan 2014 22:01, "Stefan Behnel" wrote:
>>> Stefan Behnel, 04.01.2014 22:51:
>>>> Stefan Behnel, 04.01.2014 22:47:
>>>>> Nathaniel Smith, 04.01.2014 18:36:
>>>>>>
>>>>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>>>>>   # numpy without relaxed stride support
>>>>>>   raise SkipTest
>>>>>
>>>>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
>>>>
>>>> Hmm, but this doesn't seem to work for me in older NumPy versions, although
>>>> the original test used to work there. Should we explicitly test for NumPy
>>>> 1.8+ as well?
>>>
>>> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>>
>> No, I'm missing something now; AFAIK there are only two numpy behaviors:
>> with relaxed strides and without relaxed strides, and version number should
>> be irrelevant beyond that. What's different between
>> 1.8-without-relaxed-strides and 1.7 that makes the test break?
>
> Mark would certainly know better than me.
>
> In any case, the test works with both NumPy 1.7 (tested with Py 2.x and
> 3.[12] on Jenkins) and NumPy 1.8 with relaxed strides support, but not with
> NumPy 1.8 without relaxed strides. The last two were tested in Py3.3 only,
> in case that matters. I also tested it locally now (in 3.3) and your
> snippet successfully distinguishes the two builds for me, but the test
> starts to fail when I disable relaxed strides in NumPy and works when it's
> enabled.

Then either there's a bug in numpy or in cython -- 1.7 and
1.8-without-relaxed-strides are supposed to be compatible, so if your
test works on one and not on the other, that's a bug in the libraries
somewhere, not in the test suite. Adding version checks will just hide
this bug, not fix anything.

(Personally I'd double-check whether the cython memoryview code is
doing its own version check, and wrongly assuming that if numpy
version >= 1.8, then relaxed strides = True...)

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From sebastian at sipsolutions.net  Sun Jan 19 00:41:26 2014
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sun, 19 Jan 2014 00:41:26 +0100
Subject: [Cython] "relaxed_strides" test broken with NumPy 1.8
In-Reply-To: <52C9174B.3070000@behnel.de>
References: <52C6FFE0.9060600@behnel.de>
 <CANg26EU9qAFtzTWNDi7kyqJD9v+LeN_sSWiH=eB-W6F8JNYygw@mail.gmail.com>
 <52C7F63C.9090707@behnel.de>
 <CAPJVwB=mJAJboKWVUJXdip-dMAN6kdKUYmgcAByhi41fOSrRCA@mail.gmail.com>
 <CADiQ+QDtywimFJv_GaohO8xWBsivNqGCke6McuZK3gsFHF0NUw@mail.gmail.com>
 <CAPJVwBmVKZq2XinMe2WomnOCRidhf=_9Z8QfMft2FK5YDPyMHg@mail.gmail.com>
 <52C88186.30603@behnel.de> <52C88262.5030900@behnel.de>
 <52C884BA.4040500@behnel.de>
 <CAPJVwBmV_CMNuWsgf9+kKt7iKNdKX4NJHGZ0i_ZL=0ebf-Ydfg@mail.gmail.com>
 <52C911C3.5070609@behnel.de> <52C9174B.3070000@behnel.de>
Message-ID: <1390088486.4059.3.camel@sebastian-laptop>

On Sun, 2014-01-05 at 09:26 +0100, Stefan Behnel wrote:
> Stefan Behnel, 05.01.2014 09:03:
> > Nathaniel Smith, 05.01.2014 02:07:
> >> On 4 Jan 2014 22:01, "Stefan Behnel" wrote:
> >>> Stefan Behnel, 04.01.2014 22:51:
> >>>> Stefan Behnel, 04.01.2014 22:47:
> >>>>> Nathaniel Smith, 04.01.2014 18:36:
> >>>>>>
> >>>>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
> >>>>>>   # numpy without relaxed stride support
> >>>>>>   raise SkipTest
> >>>>>
> >>>>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
> >>>>
> >>>> Hmm, but this doesn't seem to work for me in older NumPy versions, although
> >>>> the original test used to work there. Should we explicitly test for NumPy
> >>>> 1.8+ as well?
> >>>
> >>> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
> >>
> >> No, I'm missing something now; AFAIK there are only two numpy behaviors:
> >> with relaxed strides and without relaxed strides, and version number should
> >> be irrelevant beyond that. What's different between
> >> 1.8-without-relaxed-strides and 1.7 that makes the test break?
> > 
> > Mark would certainly know better than me.
> > 
> > In any case, the test works with both NumPy 1.7 (tested with Py 2.x and
> > 3.[12] on Jenkins) and NumPy 1.8 with relaxed strides support, but not with
> > NumPy 1.8 without relaxed strides. The last two were tested in Py3.3 only,
> > in case that matters. I also tested it locally now (in 3.3) and your
> > snippet successfully distinguishes the two builds for me, but the test
> > starts to fail when I disable relaxed strides in NumPy and works when it's
> > enabled.
> 
> I should add that this
> 
> 	np.ones((10, 1), order="C").flags.f_contiguous
> 
> returns False on NumPy 1.7 and only True on 1.8 with relaxed strides, thus
> the additional version test for <1.8.
> 

I didn't really follow this (only saw it now). Just to add...

Numpy 1.7. does not have relaxed strides. However, in some cases 1.8.
strides checks are more strict (and consistent) when not compiled with
relaxed strides. So it is plausible that the tests simply run into one
of those cases, maybe it even just doesn't matter if a test is
unnecessarily skipped in 1.7...

- Sebastian


> Stefan
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
> 



From arfrever.fta at gmail.com  Sun Jan 19 14:41:39 2014
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Sun, 19 Jan 2014 14:41:39 +0100
Subject: [Cython] Cython 0.20 release
In-Reply-To: <CADiQ+QD4dbrjJV-ZDiKgSm=MXE+xMZGYpnfihtk1C6rwiP2U6w@mail.gmail.com>
References: <CADiQ+QD4dbrjJV-ZDiKgSm=MXE+xMZGYpnfihtk1C6rwiP2U6w@mail.gmail.com>
Message-ID: <201401191441.41594.Arfrever.FTA@gmail.com>

2 tests fail with recent snapshot of Python 3.4.
All tests pass with Python 2.6, 2.7, 3.1, 3.2 and 3.3.
NumPy 1.8.0 is installed for each Python version.


======================================================================
FAIL: acquire_release_cycle (numpy_memoryview)
Doctest: numpy_memoryview.acquire_release_cycle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.acquire_release_cycle
  File "/var/tmp/portage/dev-python/cython-0.20-r1000/work/Cython-0.20/tests-3.4/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34.so", line unknown line number, in acquire_release_cycle

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.20-r1000/work/Cython-0.20/tests-3.4/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34.so", line ?, in numpy_memoryview.acquire_release_cycle
Failed example:
    acquire_release_cycle(a)
Expected:
    deallocated!
Got nothing


======================================================================
FAIL: acquire_release_cycle (numpy_memoryview)
Doctest: numpy_memoryview.acquire_release_cycle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.acquire_release_cycle
  File "/var/tmp/portage/dev-python/cython-0.20-r1000/work/Cython-0.20/tests-3.4/memoryview/cpp/numpy_memoryview/numpy_memoryview.cpython-34.so", line unknown line number, in acquire_release_cycle

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.20-r1000/work/Cython-0.20/tests-3.4/memoryview/cpp/numpy_memoryview/numpy_memoryview.cpython-34.so", line ?, in numpy_memoryview.acquire_release_cycle
Failed example:
    acquire_release_cycle(a)
Expected:
    deallocated!
Got nothing


----------------------------------------------------------------------
Ran 8732 tests in 3267.635s

FAILED (failures=2)
ALL DONE


--
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140119/eb11d73f/attachment.sig>

From stefan_ml at behnel.de  Sun Jan 19 14:54:37 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Jan 2014 14:54:37 +0100
Subject: [Cython] Cython 0.20 release
In-Reply-To: <201401191441.41594.Arfrever.FTA@gmail.com>
References: <CADiQ+QD4dbrjJV-ZDiKgSm=MXE+xMZGYpnfihtk1C6rwiP2U6w@mail.gmail.com>
 <201401191441.41594.Arfrever.FTA@gmail.com>
Message-ID: <52DBD91D.1020401@behnel.de>

Arfrever Frehtes Taifersar Arahesis, 19.01.2014 14:41:
> 2 tests fail with recent snapshot of Python 3.4.
> All tests pass with Python 2.6, 2.7, 3.1, 3.2 and 3.3.
> NumPy 1.8.0 is installed for each Python version.
> 
> ======================================================================
> FAIL: acquire_release_cycle (numpy_memoryview)
> Doctest: numpy_memoryview.acquire_release_cycle
> ----------------------------------------------------------------------
> Failed example:
>     acquire_release_cycle(a)
> Expected:
>     deallocated!
> Got nothing

Thanks for testing. The bug here is not that the test fails but that it is
being executed. Py3.4 has better support for docstrings in C functions and
finds this test which was explicitly disabled for previous CPython versions.

Stefan


From andriy.kornatskyy at live.com  Sun Jan 19 21:00:14 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Sun, 19 Jan 2014 22:00:14 +0200
Subject: [Cython] Compiler crash in RemoveUnreachableCode
Message-ID: <BLU0-SMTP206D43DF7EBFA5CAA6B012191BA0@phx.gbl>

The cython compiler crash report below. Steps to reproduce:

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install wheezy.http

The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.

Thanks.

Andriy Kornatskyy


  File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
  File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
Cython.Compiler.Errors.CompilerCrash:
Error compiling Cython file:
------------------------------------------------------------
...

"""
"""

__version__ = '0.1.129'
             ^
------------------------------------------------------------

src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode

ModuleNode.body = StatListNode(__init__.py:5:14)

Compiler crash traceback from this point on:
  File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
  File ?/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode
    if not self.current_directives['remove_unreachable']:
TypeError: 'NoneType' object has no attribute '__getitem__'

From stefan_ml at behnel.de  Mon Jan 20 21:19:07 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jan 2014 21:19:07 +0100
Subject: [Cython] Compiler crash in RemoveUnreachableCode
In-Reply-To: <BLU0-SMTP206D43DF7EBFA5CAA6B012191BA0@phx.gbl>
References: <BLU0-SMTP206D43DF7EBFA5CAA6B012191BA0@phx.gbl>
Message-ID: <52DD84BB.1080100@behnel.de>

Hi,

thanks for the report, I can reproduce this.

Andriy Kornatskyy, 19.01.2014 21:00:
> The cython compiler crash report below. Steps to reproduce:
> 
> 1. virtualenv env
> 2. env/bin/easy_install cython
> 3. env/bin/easy_install wheezy.http
>
> The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.

Yes, that's rather surprising.


>   File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
>   File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
> Cython.Compiler.Errors.CompilerCrash:
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> 
> """
> """
> 
> __version__ = '0.1.129'
>              ^
> ------------------------------------------------------------
> 
> src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode
> 
> ModuleNode.body = StatListNode(__init__.py:5:14)
> 
> Compiler crash traceback from this point on:
>   File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
>   File ?/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode
>     if not self.current_directives['remove_unreachable']:
> TypeError: 'NoneType' object has no attribute '__getitem__'

I only get this when Cython is compiled. When I run it from the source
tree, it works.

Debugging into Visitor.py's CythonTransform shows that this check fails:

        if isinstance(node, ModuleNode.ModuleNode):
            self.current_directives = node.directives

"node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode.

My guess is that there's something wrong with the import mechanism that
leads to the module being imported twice. Relative versus absolute imports,
most likely. I also see a couple of weird modules in sys.modules, e.g.
"Cython.Compiler.re", so there are some more bits going generally wrong here.

I also tried it in Py3, but that even gives me a hard crash. Not really better.

I might be able to take a deeper look this weekend, but wouldn't mind if
others beat me to it.

Stefan


From andriy.kornatskyy at live.com  Mon Jan 20 22:16:07 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Mon, 20 Jan 2014 23:16:07 +0200
Subject: [Cython] Compiler crash in RemoveUnreachableCode
In-Reply-To: <52DD84BB.1080100@behnel.de>
References: <BLU0-SMTP206D43DF7EBFA5CAA6B012191BA0@phx.gbl>
 <52DD84BB.1080100@behnel.de>
Message-ID: <BLU0-SMTP183B6F9D62EEC3DAFD0E5F191A50@phx.gbl>

Stefan,

Thank you for the comments back.

May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.

Here is another bug (details below):

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install lxml wheezy.core

It seems to have an issue while trying to install 2 or more libs at once.

Andriy Kornatskyy

...
Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg
Processing dependencies for lxml
Finished processing dependencies for lxml
Searching for wheezy.core
Reading https://pypi.python.org/simple/wheezy.core/
Best match: wheezy.core 0.1.129
Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d
Processing wheezy.core-0.1.129.tar.gz
Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg
Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1
Traceback (most recent call last):
...
  File ?env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation
    self.generate_c_code(env, options, result)
  File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code
    rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback)
  File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)

  File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__
    stream = StringIO()
TypeError: 'NoneType' object is not callable



On Jan 20, 2014, at 10:19 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi,
> 
> thanks for the report, I can reproduce this.
> 
> Andriy Kornatskyy, 19.01.2014 21:00:
>> The cython compiler crash report below. Steps to reproduce:
>> 
>> 1. virtualenv env
>> 2. env/bin/easy_install cython
>> 3. env/bin/easy_install wheezy.http
>> 
>> The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.
> 
> Yes, that's rather surprising.
> 
> 
>>  File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
>>  File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
>> Cython.Compiler.Errors.CompilerCrash:
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> 
>> """
>> """
>> 
>> __version__ = '0.1.129'
>>             ^
>> ------------------------------------------------------------
>> 
>> src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode
>> 
>> ModuleNode.body = StatListNode(__init__.py:5:14)
>> 
>> Compiler crash traceback from this point on:
>>  File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
>>  File ?/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode
>>    if not self.current_directives['remove_unreachable']:
>> TypeError: 'NoneType' object has no attribute '__getitem__'
> 
> I only get this when Cython is compiled. When I run it from the source
> tree, it works.
> 
> Debugging into Visitor.py's CythonTransform shows that this check fails:
> 
>        if isinstance(node, ModuleNode.ModuleNode):
>            self.current_directives = node.directives
> 
> "node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode.
> 
> My guess is that there's something wrong with the import mechanism that
> leads to the module being imported twice. Relative versus absolute imports,
> most likely. I also see a couple of weird modules in sys.modules, e.g.
> "Cython.Compiler.re", so there are some more bits going generally wrong here.
> 
> I also tried it in Py3, but that even gives me a hard crash. Not really better.
> 
> I might be able to take a deeper look this weekend, but wouldn't mind if
> others beat me to it.
> 
> Stefan
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From stefan_ml at behnel.de  Tue Jan 21 09:16:46 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Jan 2014 09:16:46 +0100
Subject: [Cython] Compiler crash in RemoveUnreachableCode
In-Reply-To: <BLU0-SMTP183B6F9D62EEC3DAFD0E5F191A50@phx.gbl>
References: <BLU0-SMTP206D43DF7EBFA5CAA6B012191BA0@phx.gbl>
 <52DD84BB.1080100@behnel.de> <BLU0-SMTP183B6F9D62EEC3DAFD0E5F191A50@phx.gbl>
Message-ID: <52DE2CEE.7030906@behnel.de>

Andriy Kornatskyy, 20.01.2014 22:16:
> May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.

No, I think it's a problem with importing in Cython compiled modules. ISTM
that some extension modules got "reloaded", i.e. their module dict got
cleared and they got reinitialised, thus recreating all global objects that
other modules had already imported (and still keep a reference to).


> Here is another bug (details below):
> 
> 1. virtualenv env
> 2. env/bin/easy_install cython
> 3. env/bin/easy_install lxml wheezy.core
> 
> It seems to have an issue while trying to install 2 or more libs at once.
> ...
> Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg
> Processing dependencies for lxml
> Finished processing dependencies for lxml
> Searching for wheezy.core
> Reading https://pypi.python.org/simple/wheezy.core/
> Best match: wheezy.core 0.1.129
> Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d
> Processing wheezy.core-0.1.129.tar.gz
> Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg
> Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1
> Traceback (most recent call last):
> ...
>   File ?env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation
>     self.generate_c_code(env, options, result)
>   File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code
>     rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback)
>   File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)
> 
>   File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__
>     stream = StringIO()
> TypeError: 'NoneType' object is not callable

Looks like the same thing.

Stefan


From jtaylor.debian at googlemail.com  Wed Jan 22 00:03:20 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 22 Jan 2014 00:03:20 +0100
Subject: [Cython] bytearray tests fail with default unsigned char
Message-ID: <52DEFCB8.8010700@googlemail.com>

hi,
the bytearray tests are broken when chars are unsigned.

tests/run/bytearraymethods.pyx defines following function:
def bytearray_append(b, char c, int i, object o):

this gets converted to an effective __Pyx_PyInt_AsUnsignedChar which
then errors out when -1 is passed in.
chars are unsigned like they are by default on arm, s390x and powerpc.
This causes a couple build failures in debian:
https://buildd.debian.org/status/package.php?p=cython
https://buildd.debian.org/status/fetch.php?pkg=cython&arch=armel&ver=0.20-1&stamp=1390316252

the tests can be fixed by adding signed char to the interface.

to reproduce on x86 with gcc (note the -funsigned-char to change the
default):

cython tests/run/bytearraymethods.pyx
gcc -funsigned-char bytearraymethods.c -fPIC $(python-config --includes)
$(python-config --libs) -shared -O2 -o bytearraymethods.so

python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'bytearraymethods' is not defined
root at ubuntu:/# python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"
**********************************************************************
File "bytearraymethods.so", line ?, in
bytearraymethods.__test__.bytearray_append (line 202)
Failed example:
    b = bytearray_append(b, -1, ord('y'), ord('z'))  # doctest: +ELLIPSIS
Expected:
    Traceback (most recent call last):
    ValueError: ...
Got:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest bytearraymethods.__test__.bytearray_append (line
202)[14]>", line 1, in <module>
        b = bytearray_append(b, -1, ord('y'), ord('z'))  # doctest:
+ELLIPSIS
      File "bytearraymethods.pyx", line 202, in
bytearraymethods.bytearray_append (bytearraymethods.c:1339)
        def bytearray_append(bytearray b, char c, int i, object o):
    OverflowError: can't convert negative value to char
**********************************************************************


From syam.gadde at duke.edu  Tue Jan 21 23:00:06 2014
From: syam.gadde at duke.edu (Syam Gadde)
Date: Tue, 21 Jan 2014 17:00:06 -0500
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52DECCD2.1030708@duke.edu>
References: <52DECCD2.1030708@duke.edu>
Message-ID: <52DEEDE6.8090102@duke.edu>

(apologies to the moderator for multiple postings -- I'm trying to figure out which of my email addresses is actually being exported to the public!)

Hi,

It seems that cython's static code analysis is failing on the following
code.  It thinks that a particular variable (myobject2) must not be
NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
from third-party code, so I could fix it with an explicit initialization
to None, but I'd love to have Cython deal with it directly.  Thanks for
your help!

#<<<BEGIN tmpnone.py>>>
# When compiled with Cython 0.19.2 or 0.20dev, the following code
# segfaults on import

import random

def myfunc():
      ## uncommenting the following fixes things
      #myobject2 = None

      myfalse = random.random() > 2

      if myfalse:
          myobject = None
          myobject2 = None

      if not myfalse:
          # here Cython uses __Pyx_XDECREF_SET
          myobject = None

      print "Made it past myobject assignment"

      if not myfalse or myobject2 is None:
          # here Cython uses Pyx_DECREF_SET because it has determined
          # that __pyx_v_myobject2 can't be null, but it really, really can!
          # (if no one assigned myobject2 yet)
          myobject2 = None

      print "Made it past myobject2 assignment"

myfunc()
#<<<END tmpnone.py>>>

#<<BEGIN setup.py>>>
#<from distutils.core import setup
from Cython.Build import cythonize

setup(
      ext_modules = cythonize("tmpnone.py")
)
#<<END setup.py>>>

[prompt]$ python setup.py build_ext --inplace
[prompt]$ ls tmpnone.*
tmpnone.c  tmpnone.py  tmpnone.so
[prompt]$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
  >>> import tmpnone
Made it past myobject assignment
Segmentation fault





From stefan_ml at behnel.de  Thu Jan 23 18:58:10 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Jan 2014 18:58:10 +0100
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52DEEDE6.8090102@duke.edu>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
Message-ID: <52E15832.90506@behnel.de>

Syam Gadde, 21.01.2014 23:00:
> It seems that cython's static code analysis is failing on the following
> code.  It thinks that a particular variable (myobject2) must not be
> NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
> from third-party code, so I could fix it with an explicit initialization
> to None, but I'd love to have Cython deal with it directly.  Thanks for
> your help!
> 
> # When compiled with Cython 0.19.2 or 0.20dev, the following code
> # segfaults on import
> 
> import random
> 
> def myfunc():
>      ## uncommenting the following fixes things
>      #myobject2 = None
> 
>      myfalse = random.random() > 2
> 
>      if myfalse:
>          myobject = None
>          myobject2 = None
> 
>      if not myfalse:
>          # here Cython uses __Pyx_XDECREF_SET
>          myobject = None
> 
>      print "Made it past myobject assignment"
> 
>      if not myfalse or myobject2 is None:
>          # here Cython uses Pyx_DECREF_SET because it has determined
>          # that __pyx_v_myobject2 can't be null, but it really, really can!
>          # (if no one assigned myobject2 yet)
>          myobject2 = None
> 
>      print "Made it past myobject2 assignment"
> 
> myfunc()

Thanks for the report. The generated C code currently looks like this:

===================
/*
 *      if not myfalse or myobject2 is None:             # <<<<<<<<<<<<<<
 */
  __pyx_t_4 = __Pyx_PyObject_IsTrue(__pyx_v_myfalse); if /*error*/
  __pyx_t_3 = ((!__pyx_t_4) != 0);
  if (!__pyx_t_3) {
    if (unlikely(!__pyx_v_myobject2)) {
                     __Pyx_RaiseUnboundLocalError("myobject2"); {...} }
    __pyx_t_4 = (__pyx_v_myobject2 == Py_None);
    __pyx_t_5 = (__pyx_t_4 != 0);
  } else {
    __pyx_t_5 = __pyx_t_3;
  }
  if (__pyx_t_5) {

/*
 *          myobject2 = None             # <<<<<<<<<<<<<<
 */
    __Pyx_INCREF(Py_None);
    __Pyx_DECREF_SET(__pyx_v_myobject2, Py_None);
===================

So, it's smart enough to figure out that it has to raise an
UnboundLocalError on the "if" test if the variable hasn't been set yet, but
then fails to see that in the case that the condition short-circuits, this
test hasn't been executed yet.

BTW, thanks for cutting down the code to a short and easy to analyse test
case, but if the original code really has a control flow like that, there
might be some space left for improved clarity.

Stefan


From stefan_ml at behnel.de  Thu Jan 23 19:19:21 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Jan 2014 19:19:21 +0100
Subject: [Cython] bytearray tests fail with default unsigned char
In-Reply-To: <52DEFCB8.8010700@googlemail.com>
References: <52DEFCB8.8010700@googlemail.com>
Message-ID: <52E15D29.90705@behnel.de>

Julian Taylor, 22.01.2014 00:03:
> the bytearray tests are broken when chars are unsigned.
> 
> tests/run/bytearraymethods.pyx defines following function:
> def bytearray_append(b, char c, int i, object o):
> 
> this gets converted to an effective __Pyx_PyInt_AsUnsignedChar which
> then errors out when -1 is passed in.
> chars are unsigned like they are by default on arm, s390x and powerpc.
> This causes a couple build failures in debian:
> https://buildd.debian.org/status/package.php?p=cython
> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=armel&ver=0.20-1&stamp=1390316252
> 
> the tests can be fixed by adding signed char to the interface.

Thanks for the report. You're right, testing with 'signed char' is better here.

https://github.com/cython/cython/commit/0b665f974669b7ad5c68c7fb3fcedad883c51426

Stefan


From syam.gadde at duke.edu  Thu Jan 23 19:25:06 2014
From: syam.gadde at duke.edu (Syam Gadde)
Date: Thu, 23 Jan 2014 13:25:06 -0500
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52E15832.90506@behnel.de>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de>
Message-ID: <52E15E82.9070306@duke.edu>

On 01/23/2014 01:04 PM, Stefan Behnel wrote:
> BTW, thanks for cutting down the code to a short and easy to analyse 
> test case, but if the original code really has a control flow like 
> that, there might be some space left for improved clarity.

Absolutely.  In the wild, it manifested itself in PyFFTW.  It turns out 
I will likely not be using it, but thought it was an interesting edge 
case, and was what looked like valid Python, though it plays hard and 
fast with Python's runtime binding...

-syam

On 01/23/2014 01:04 PM, Stefan Behnel wrote:
> Syam Gadde, 21.01.2014 23:00:
>> It seems that cython's static code analysis is failing on the following
>> code.  It thinks that a particular variable (myobject2) must not be
>> NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
>> from third-party code, so I could fix it with an explicit initialization
>> to None, but I'd love to have Cython deal with it directly.  Thanks for
>> your help!
>>
>> # When compiled with Cython 0.19.2 or 0.20dev, the following code
>> # segfaults on import
>>
>> import random
>>
>> def myfunc():
>>       ## uncommenting the following fixes things
>>       #myobject2 = None
>>
>>       myfalse = random.random() > 2
>>
>>       if myfalse:
>>           myobject = None
>>           myobject2 = None
>>
>>       if not myfalse:
>>           # here Cython uses __Pyx_XDECREF_SET
>>           myobject = None
>>
>>       print "Made it past myobject assignment"
>>
>>       if not myfalse or myobject2 is None:
>>           # here Cython uses Pyx_DECREF_SET because it has determined
>>           # that __pyx_v_myobject2 can't be null, but it really, really can!
>>           # (if no one assigned myobject2 yet)
>>           myobject2 = None
>>
>>       print "Made it past myobject2 assignment"
>>
>> myfunc()
> Thanks for the report. The generated C code currently looks like this:
>
> ===================
> /*
>   *      if not myfalse or myobject2 is None:             # <<<<<<<<<<<<<<
>   */
>    __pyx_t_4 = __Pyx_PyObject_IsTrue(__pyx_v_myfalse); if /*error*/
>    __pyx_t_3 = ((!__pyx_t_4) != 0);
>    if (!__pyx_t_3) {
>      if (unlikely(!__pyx_v_myobject2)) {
>                       __Pyx_RaiseUnboundLocalError("myobject2"); {...} }
>      __pyx_t_4 = (__pyx_v_myobject2 == Py_None);
>      __pyx_t_5 = (__pyx_t_4 != 0);
>    } else {
>      __pyx_t_5 = __pyx_t_3;
>    }
>    if (__pyx_t_5) {
>
> /*
>   *          myobject2 = None             # <<<<<<<<<<<<<<
>   */
>      __Pyx_INCREF(Py_None);
>      __Pyx_DECREF_SET(__pyx_v_myobject2, Py_None);
> ===================
>
> So, it's smart enough to figure out that it has to raise an
> UnboundLocalError on the "if" test if the variable hasn't been set yet, but
> then fails to see that in the case that the condition short-circuits, this
> test hasn't been executed yet.
>
> BTW, thanks for cutting down the code to a short and easy to analyse test
> case, but if the original code really has a control flow like that, there
> might be some space left for improved clarity.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>


From stefan_ml at behnel.de  Thu Jan 23 20:49:12 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Jan 2014 20:49:12 +0100
Subject: [Cython] C code churn and benchmark diffs
Message-ID: <52E17238.7050208@behnel.de>

Hi,

to get a better idea of what new optimisations (or C code changes in
general) actually bring, I've added a diffing step to the benchmark runner
jobs. The bigger ones now generate an additional output file
"cfiles.diff.gz" that contains a diff of the C files generated in the
current run against those of the last (successful) run. E.g. here:

https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-pybenchmarks-neu-py3k/

(That one should start looking a lot smaller tomorrow :)

While doing that, I noticed that there are a couple of unnecessary
differences in the C files that are due to dict iteration, which is
randomised in Py3.3+ (for security reasons). I fixed (most of?) them, so
repeated Cython runs should now generate pretty much identical output
files, if you strip the file header comment (diff can do that). The rest is
generated deterministically in source code order already.

It should be possible to do the same thing for the Sage build, I guess.
That would also be interesting to look at from time to time, especially
when things start failing for some reason.

Stefan

From vitja.makarov at gmail.com  Thu Jan 23 21:02:03 2014
From: vitja.makarov at gmail.com (Victor Makarov)
Date: Fri, 24 Jan 2014 00:02:03 +0400
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52E15E82.9070306@duke.edu>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
Message-ID: <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>

2014/1/23 Syam Gadde <syam.gadde at duke.edu>:
> On 01/23/2014 01:04 PM, Stefan Behnel wrote:
>>
>> BTW, thanks for cutting down the code to a short and easy to analyse test
>> case, but if the original code really has a control flow like that, there
>> might be some space left for improved clarity.
>
>
> Absolutely.  In the wild, it manifested itself in PyFFTW.  It turns out I
> will likely not be using it, but thought it was an interesting edge case,
> and was what looked like valid Python, though it plays hard and fast with
> Python's runtime binding...
>
> -syam
>
>
> On 01/23/2014 01:04 PM, Stefan Behnel wrote:
>>
>> Syam Gadde, 21.01.2014 23:00:
>>>
>>> It seems that cython's static code analysis is failing on the following
>>> code.  It thinks that a particular variable (myobject2) must not be
>>> NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
>>> from third-party code, so I could fix it with an explicit initialization
>>> to None, but I'd love to have Cython deal with it directly.  Thanks for
>>> your help!
>>>
>>> # When compiled with Cython 0.19.2 or 0.20dev, the following code
>>> # segfaults on import
>>>
>>> import random
>>>
>>> def myfunc():
>>>       ## uncommenting the following fixes things
>>>       #myobject2 = None
>>>
>>>       myfalse = random.random() > 2
>>>
>>>       if myfalse:
>>>           myobject = None
>>>           myobject2 = None
>>>
>>>       if not myfalse:
>>>           # here Cython uses __Pyx_XDECREF_SET
>>>           myobject = None
>>>
>>>       print "Made it past myobject assignment"
>>>
>>>       if not myfalse or myobject2 is None:
>>>           # here Cython uses Pyx_DECREF_SET because it has determined
>>>           # that __pyx_v_myobject2 can't be null, but it really, really
>>> can!
>>>           # (if no one assigned myobject2 yet)
>>>           myobject2 = None
>>>
>>>       print "Made it past myobject2 assignment"
>>>
>>> myfunc()
>>
>> Thanks for the report. The generated C code currently looks like this:
>>
>> ===================
>> /*
>>   *      if not myfalse or myobject2 is None:             # <<<<<<<<<<<<<<
>>   */
>>    __pyx_t_4 = __Pyx_PyObject_IsTrue(__pyx_v_myfalse); if /*error*/
>>    __pyx_t_3 = ((!__pyx_t_4) != 0);
>>    if (!__pyx_t_3) {
>>      if (unlikely(!__pyx_v_myobject2)) {
>>                       __Pyx_RaiseUnboundLocalError("myobject2"); {...} }
>>      __pyx_t_4 = (__pyx_v_myobject2 == Py_None);
>>      __pyx_t_5 = (__pyx_t_4 != 0);
>>    } else {
>>      __pyx_t_5 = __pyx_t_3;
>>    }
>>    if (__pyx_t_5) {
>>
>> /*
>>   *          myobject2 = None             # <<<<<<<<<<<<<<
>>   */
>>      __Pyx_INCREF(Py_None);
>>      __Pyx_DECREF_SET(__pyx_v_myobject2, Py_None);
>> ===================
>>
>> So, it's smart enough to figure out that it has to raise an
>> UnboundLocalError on the "if" test if the variable hasn't been set yet,
>> but
>> then fails to see that in the case that the condition short-circuits, this
>> test hasn't been executed yet.
>>
>> BTW, thanks for cutting down the code to a short and easy to analyse test
>> case, but if the original code really has a control flow like that, there
>> might be some space left for improved clarity.
>>
>> Stefan
>>


Hi!

It's a bug! Being referenced in expression varaiable is marked as not
null. But it doesn't take into account that some subexpressions wasn't
evaluated. Here is another example:

def crashme(a=True):
    if not a:
        b = []
    (a or b) and (b or a)
crashme()

It seems to me that reference shouldn't affect variable's flags since
we don't have a simple way to track evaluation order.

I'd like to fix this if you don't mind.


-- 
vitja.

From stefan_ml at behnel.de  Thu Jan 23 21:10:30 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Jan 2014 21:10:30 +0100
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
 <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
Message-ID: <52E17736.5020908@behnel.de>

Victor Makarov, 23.01.2014 21:02:
> I'd like to fix this if you don't mind.

Oh, I certainly don't mind. :)

Stefan


From jtaylor.debian at googlemail.com  Fri Jan 24 22:26:20 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 24 Jan 2014 22:26:20 +0100
Subject: [Cython] 0.20 tests fail with python3.4b2
Message-ID: <52E2DA7C.4050506@googlemail.com>

hi,

with python3.4b2 numpy_memoryview.acquire_release_cycle fails
looking at the source the test should be disabled, but its still run.

This might be a py3.4 change to doctest, but I know nothing about
doctest, so I wanted to check first if you know something about this.
Interestingly py3.4 runs 64 tests, while python3.3 only runs 32.
the __test__ variable contains the same functions in both python
versions (acquire_release is not in it)

======================================================================
FAIL: acquire_release_cycle (numpy_memoryview)
Doctest: numpy_memoryview.acquire_release_cycle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
numpy_memoryview.acquire_release_cycle
  File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line unknown line number, in acquire_release_cycle

----------------------------------------------------------------------
File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line ?, in numpy_memoryview.acquire_release_cycle
Failed example:
    acquire_release_cycle(a)
Expected:
    deallocated!
Got nothing

From stefan_ml at behnel.de  Sat Jan 25 07:07:08 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Jan 2014 07:07:08 +0100
Subject: [Cython] 0.20 tests fail with python3.4b2
In-Reply-To: <52E2DA7C.4050506@googlemail.com>
References: <52E2DA7C.4050506@googlemail.com>
Message-ID: <52E3548C.9060507@behnel.de>

Julian Taylor, 24.01.2014 22:26:
> with python3.4b2 numpy_memoryview.acquire_release_cycle fails
> looking at the source the test should be disabled, but its still run.

Yes, that's a known problem, the test is fixed in master. In short, Py3.4
has support for finding doctests in the docstrings of C implemented
functions, and thus doesn't need Cython's work-arounds for this anymore.

http://thread.gmane.org/gmane.comp.python.cython.devel/15111

Stefan


From vitja.makarov at gmail.com  Sun Jan 26 20:15:06 2014
From: vitja.makarov at gmail.com (Victor Makarov)
Date: Sun, 26 Jan 2014 23:15:06 +0400
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52E17736.5020908@behnel.de>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
 <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
 <52E17736.5020908@behnel.de>
Message-ID: <CAKGHGPQmtCbVq8_QvU3MQiU4o4ocSrYA--0hzLVg=EbKNL_Suw@mail.gmail.com>

2014/1/24 Stefan Behnel <stefan_ml at behnel.de>:
> Victor Makarov, 23.01.2014 21:02:
>> I'd like to fix this if you don't mind.
>
> Oh, I certainly don't mind. :)
>

Fixed here https://github.com/cython/cython/commit/9f7256fd07961b912d2224d995533c821441381b



-- 
vitja.

From syam.gadde at duke.edu  Tue Jan 28 19:41:35 2014
From: syam.gadde at duke.edu (Syam Gadde)
Date: Tue, 28 Jan 2014 13:41:35 -0500
Subject: [Cython] memoryview refcount error
Message-ID: <52E7F9DF.8020909@duke.edu>

Hi again,

I'm exploring the typed memoryview feature with numpy and encountered a 
refcount error.  It seems to happen when I create a slice of a 
memoryview, and transpose it.  If I don't do both, I don't get the 
refcount error.  It happens in the error cleanup code.  I suspect that 
any error that causes the function to jump to the error cleanup code 
will do the same thing as the Exception below -- but that seemed the 
easiest way to demonstrate the problem. However, if there truly is an 
error in my code, please let me know! Thanks,

-syam

# BEGIN CODE
import numpy
cimport numpy

class MyException(Exception):
     def __init__(self):
         Exception.__init__(self)

def testfunc():
    a = numpy.arange(12).reshape([3,4])

    cdef numpy.int_t[:,:] a_view = a

    ## here is a slice followed by a transpose
    cdef numpy.int_t[:,:] b = a_view[:,:].T
    ## same thing happens with a more realistic slicing:
    #cdef numpy.int_t[:,:] b = a_view[1:2,:].T
    ## however, there is no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view.T
    ## also no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view[:,:]

    # The exception below is just to force the function to abort
    # and run the extraneous __PYX_XDEC_MEMVIEW in the error
    # cleanup code:
    #   Fatal Python error: Acquisition count is 0 (line XXXX)
    # Comment out this exception and we don't get the error
    raise MyException()

try:
     testfunc()
except MyException:
     pass
# END CODE




From andrew.ohana at gmail.com  Tue Jan 28 23:34:10 2014
From: andrew.ohana at gmail.com (R. Andrew Ohana)
Date: Tue, 28 Jan 2014 14:34:10 -0800
Subject: [Cython] Broken list multiplication
Message-ID: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>

It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very
simple) instances of list multiplication when there are at least 3 terms in
the product. e.g. on my system I'm getting these results:

L = [None] * 2 * 3
print(len(L))
3

L = 2 * [None] * 3
print(len(L))
3

L = [None] * 2 * 3 * 5
print(len(L))
5

-- 
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140128/6d237f27/attachment.html>

From stefan_ml at behnel.de  Wed Jan 29 18:03:16 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Jan 2014 18:03:16 +0100
Subject: [Cython] Broken list multiplication
In-Reply-To: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
References: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
Message-ID: <52E93454.9030707@behnel.de>

R. Andrew Ohana, 28.01.2014 23:34:
> It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very
> simple) instances of list multiplication when there are at least 3 terms in
> the product. e.g. on my system I'm getting these results:
> 
> L = [None] * 2 * 3
> print(len(L))
> 3
> 
> L = 2 * [None] * 3
> print(len(L))
> 3
> 
> L = [None] * 2 * 3 * 5
> print(len(L))
> 5

Thanks for the report.

https://github.com/cython/cython/commit/bfe7e78355629577b3cc92d58d6e02f27ee82406

Stefan


From stefan_ml at behnel.de  Wed Jan 29 18:11:30 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Jan 2014 18:11:30 +0100
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <CAKGHGPQmtCbVq8_QvU3MQiU4o4ocSrYA--0hzLVg=EbKNL_Suw@mail.gmail.com>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
 <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
 <52E17736.5020908@behnel.de>
 <CAKGHGPQmtCbVq8_QvU3MQiU4o4ocSrYA--0hzLVg=EbKNL_Suw@mail.gmail.com>
Message-ID: <52E93642.7010301@behnel.de>

Victor Makarov, 26.01.2014 20:15:
> Fixed here https://github.com/cython/cython/commit/9f7256fd07961b912d2224d995533c821441381b

Hmm, it's unfortunate that this needs to be disabled.

I guess the correct way to eventually implement this would be by creating a
new control flow block also for RHS of the BoolBinopNode, right?

Stefan


From vitja.makarov at gmail.com  Wed Jan 29 18:35:42 2014
From: vitja.makarov at gmail.com (Victor Makarov)
Date: Wed, 29 Jan 2014 21:35:42 +0400
Subject: [Cython] segfault due to using DECREF instead of XDECREF
In-Reply-To: <52E93642.7010301@behnel.de>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
 <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
 <52E17736.5020908@behnel.de>
 <CAKGHGPQmtCbVq8_QvU3MQiU4o4ocSrYA--0hzLVg=EbKNL_Suw@mail.gmail.com>
 <52E93642.7010301@behnel.de>
Message-ID: <CAKGHGPSFp8Eqn+pHWuNqLNUgAgFtkRg9_DJ65ioHvhZhmiLR=A@mail.gmail.com>

2014/1/29 Stefan Behnel <stefan_ml at behnel.de>:
> Victor Makarov, 26.01.2014 20:15:
>> Fixed here https://github.com/cython/cython/commit/9f7256fd07961b912d2224d995533c821441381b
>
> Hmm, it's unfortunate that this needs to be disabled.
>
> I guess the correct way to eventually implement this would be by creating a
> new control flow block also for RHS of the BoolBinopNode, right?
>

I don't think it's a big deal because It only affects 'may be null'
variables that aren't common in python world.

OTOH it would be great to fix this. Easy way is to create a new CF
block for each reference. I'm not sure that BoolBinopNode is enough.

-- 
vitja.

From robertwb at gmail.com  Wed Jan 29 19:10:20 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 29 Jan 2014 10:10:20 -0800
Subject: [Cython] Broken list multiplication
In-Reply-To: <52E93454.9030707@behnel.de>
References: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
 <52E93454.9030707@behnel.de>
Message-ID: <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>

On Wed, Jan 29, 2014 at 9:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> R. Andrew Ohana, 28.01.2014 23:34:
>> It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very
>> simple) instances of list multiplication when there are at least 3 terms in
>> the product. e.g. on my system I'm getting these results:
>>
>> L = [None] * 2 * 3
>> print(len(L))
>> 3
>>
>> L = 2 * [None] * 3
>> print(len(L))
>> 3
>>
>> L = [None] * 2 * 3 * 5
>> print(len(L))
>> 5
>
> Thanks for the report.
>
> https://github.com/cython/cython/commit/bfe7e78355629577b3cc92d58d6e02f27ee82406

Ouch. Between this and the other Sage issue, I feel a small point
release coming up. Any other bug fixes we should get in?

- Robert

From robertwb at gmail.com  Wed Jan 29 19:13:22 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 29 Jan 2014 10:13:22 -0800
Subject: [Cython] Broken list multiplication
In-Reply-To: <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>
References: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
 <52E93454.9030707@behnel.de>
 <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>
Message-ID: <CADiQ+QC0cZUpwBMsUKf2vZX-qkCOrmFa3be4P3W3ELKJTTDb-Q@mail.gmail.com>

On Wed, Jan 29, 2014 at 10:10 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Wed, Jan 29, 2014 at 9:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> R. Andrew Ohana, 28.01.2014 23:34:
>>> It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very
>>> simple) instances of list multiplication when there are at least 3 terms in
>>> the product. e.g. on my system I'm getting these results:
>>>
>>> L = [None] * 2 * 3
>>> print(len(L))
>>> 3
>>>
>>> L = 2 * [None] * 3
>>> print(len(L))
>>> 3
>>>
>>> L = [None] * 2 * 3 * 5
>>> print(len(L))
>>> 5
>>
>> Thanks for the report.
>>
>> https://github.com/cython/cython/commit/bfe7e78355629577b3cc92d58d6e02f27ee82406
>
> Ouch. Between this and the other Sage issue, I feel a small point
> release coming up. Any other bug fixes we should get in?

Oh, the issues are one and the same. I should read all my email before
replying...

Still, anything else we should try to get in?

- Robert

From stefan_ml at behnel.de  Thu Jan 30 11:13:04 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 30 Jan 2014 11:13:04 +0100
Subject: [Cython] "embedsignature" and "autotestdict" features in Py3.4
In-Reply-To: <5298BAB8.3080909@behnel.de>
References: <5298BAB8.3080909@behnel.de>
Message-ID: <52EA25B0.9000403@behnel.de>

Stefan Behnel, 29.11.2013 17:03:
> recent changes in Py3.4 affect the functionality of the two directives
> "embedsignature" and "autotestdict".
> 
> The so-called "argument clinic" extracts any embedded signatures from
> docstrings and moves them into a new property "__text_signature__". This is
> done at runtime, i.e. when either "__doc__" or "__text_signature__" are
> requested.
> 
> http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182
> 
> I personally consider this a feature (and it works nicely with the
> signatures that Cython embeds), but you may or may not agree. It broke some
> of our own doctests, at least, because the "__doc__" value that we tested
> for was no longer the same.

This feature has essentially been reverted and can thus be expected to not
go into the final CPython 3.4 release. The reason was that it interfered
with existing docstrings (as noted above), so they invented a new scheme
"sig=..." for generated signatures to only extract those created by the
argument clinic itself.

For Cython, this means that CPython no longer touches the docstrings, so
the embedded signature string continues to appear where users can see it
and tools like Sphinx or epydoc parse it.

Eventually, Cython should generate a real Python level representation of
the signature, but that's not easy to get "right" due to the C-to-Python
mapping, and because the "Signature" class is not available in older Python
versions and generally not meant to be used manually. See PEP 362 for some
details.

http://www.python.org/dev/peps/pep-0362/

Stefan


From stefan_ml at behnel.de  Thu Jan 30 17:40:16 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 30 Jan 2014 17:40:16 +0100
Subject: [Cython] Broken list multiplication
In-Reply-To: <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>
References: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
 <52E93454.9030707@behnel.de>
 <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>
Message-ID: <52EA8070.7090108@behnel.de>

Robert Bradshaw, 29.01.2014 19:10:
> Ouch. Between this and the other Sage issue, I feel a small point
> release coming up. Any other bug fixes we should get in?

I have a couple of pending requests for others to test lxml on Windows and
maybe even PyPy, so let's wait another couple of days for them to report
back. I don't see any notable changes coming.

Stefan


From stefan_ml at behnel.de  Fri Jan 31 12:49:05 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Jan 2014 12:49:05 +0100
Subject: [Cython] 0.20.1 bug fix release (was: Broken list multiplication)
In-Reply-To: <52EA8070.7090108@behnel.de>
References: <CANUPmSZKMecDZLiJzQ5UcF5vVpNU7zK9Jysiur=0E5z0rw8VnA@mail.gmail.com>
 <52E93454.9030707@behnel.de>
 <CADiQ+QDx88uV=voOTvndG4v0AikSpgnPhRFUCkEAKcP12Ppv5w@mail.gmail.com>
 <52EA8070.7090108@behnel.de>
Message-ID: <52EB8DB1.1070708@behnel.de>

Stefan Behnel, 30.01.2014 17:40:
> Robert Bradshaw, 29.01.2014 19:10:
>> Ouch. Between this and the other Sage issue, I feel a small point
>> release coming up. Any other bug fixes we should get in?
> 
> I have a couple of pending requests for others to test lxml on Windows and
> maybe even PyPy, so let's wait another couple of days for them to report
> back. I don't see any notable changes coming.

All confirmed that it works now, so I'm fine with getting the bug fix
release rolling.

The Sage builds looks a bit broken currently, though.

Stefan


From stefan_ml at behnel.de  Fri Jan 31 12:59:32 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Jan 2014 12:59:32 +0100
Subject: [Cython] flow control analysis of expressions (was: segfault due to
 using DECREF instead of XDECREF)
In-Reply-To: <CAKGHGPSFp8Eqn+pHWuNqLNUgAgFtkRg9_DJ65ioHvhZhmiLR=A@mail.gmail.com>
References: <52DECCD2.1030708@duke.edu> <52DEEDE6.8090102@duke.edu>
 <52E15832.90506@behnel.de> <52E15E82.9070306@duke.edu>
 <CAKGHGPSOsYS5A=BZKtMcR5Ne__xF_WuFbfN-qa6oz-e4c-ddEg@mail.gmail.com>
 <52E17736.5020908@behnel.de>
 <CAKGHGPQmtCbVq8_QvU3MQiU4o4ocSrYA--0hzLVg=EbKNL_Suw@mail.gmail.com>
 <52E93642.7010301@behnel.de>
 <CAKGHGPSFp8Eqn+pHWuNqLNUgAgFtkRg9_DJ65ioHvhZhmiLR=A@mail.gmail.com>
Message-ID: <52EB9024.20908@behnel.de>

[looks like the mailing list server failed to accept this, so here it is again]

Victor Makarov, 29.01.2014 18:35:
> 2014/1/29 Stefan Behnel:
>> Victor Makarov, 26.01.2014 20:15:
>>> Fixed here https://github.com/cython/cython/commit/9f7256fd07961b912d2224d995533c821441381b
>>
>> Hmm, it's unfortunate that this needs to be disabled.
>>
>> I guess the correct way to eventually implement this would be by creating a
>> new control flow block also for RHS of the BoolBinopNode, right?
> 
> I don't think it's a big deal because It only affects 'may be null'
> variables that aren't common in python world.

Agreed.


> OTOH it would be great to fix this. Easy way is to create a new CF
> block for each reference. I'm not sure that BoolBinopNode is enough.

Well, then anything that short-circuits, i.e. that has an actual control
flow impact. In expressions like

    x = a + b - c

it's clear that if no exception is raised, all variables will have been
evaluated, whereas in

    x = a + b or c

the variable c may or may not have been touched, but a and b definitely
were. The same applies to conditional "x if y else z" expressions, but I
don't think there are so many more. Anything scoped (genexprs, listcomps,
lambdas) obviously doesn't fit here and can simply be handled as before.

Stefan


From stefan_ml at behnel.de  Fri Jan 31 12:55:13 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Jan 2014 12:55:13 +0100
Subject: [Cython] Sage build problem
Message-ID: <52EB8F21.1090801@behnel.de>

Hi,

There's some kind of C++ enum issue in the Sage build.

Generated code in sage/libs/ppl.cpp:

"""
static PyObject *__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(
             struct __pyx_obj_4sage_4libs_3ppl_MIP_Problem *__pyx_v_self) {
  enum Parma_Polyhedra_Library::PPL_MIP_Problem_Status __pyx_v_tmp;
"""

gcc error:

"""
sage/libs/ppl.cpp: In function 'PyObject*
__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(__pyx_obj_4sage_4libs_3ppl_MIP_Problem*)':

sage/libs/ppl.cpp:5514:33: error: 'PPL_MIP_Problem_Status' in namespace
'Parma_Polyhedra_Library' does not name a type
sage/libs/ppl.cpp:5514:67: error: invalid type in declaration before ';' token
"""

The type of "tmp" seems to be inferred. That might be a hint that it could
be a problem in Cython.

The Cython code in question is also a bit funny:

sage/libs/ppl.pyx:1076
"""
       sig_on()
       try:
           tmp = self.thisptr.solve()             # <<<<<<<<<<<<<<
       finally:
           sig_off()
"""

The try-finally is useless here, since the call to solve() cannot raise
Python exceptions. The new try-finally code generation shows this clearly,
it does not generate an exception handling block. The idea might have been
that it could raise C++ exceptions, but they don't seem to be declared, so
Cython won't do anything about them.

Might be a hint that Cython could do more to make C++ exception handling
safer by default.

Stefan


From syam.gadde at duke.edu  Fri Jan 31 20:11:49 2014
From: syam.gadde at duke.edu (Syam Gadde)
Date: Fri, 31 Jan 2014 14:11:49 -0500
Subject: [Cython] memoryview refcount error
In-Reply-To: <52E7F9DF.8020909@duke.edu>
References: <52E7F9DF.8020909@duke.edu>
Message-ID: <52EBF575.1010500@duke.edu>

I've done some more digging here. It turns out the test case for this 
bug is simpler than I thought:

import numpy
cimport numpy

def testfunc():
    a = numpy.arange(12).reshape([3,4])
    cdef numpy.int_t[:,:] a_view = a
    cdef numpy.int_t[:,:] b = a_view[:,:].T

testfunc()


The core issue is that a transpose of a slice of a memoryview results in 
bad reference counts on the memview (comments interspersed below to 
indicate refcount and who has references to the base memoryview):

   /* "testtranspose.pyx":6
  * def testfunc():
  *    a = numpy.arange(12).reshape([3,4])
  *    cdef numpy.int_t[:,:] a_view = a             # <<<<<<<<<<<<<<
  *    cdef numpy.int_t[:,:] b = a_view[:,:].T
  *
  */
   __pyx_t_4 = 
__Pyx_PyObject_to_MemoryviewSlice_dsds_nn___pyx_t_5numpy_int_t(__pyx_v_a);
   if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; 
__pyx_lineno = 6; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
   __pyx_v_a_view = __pyx_t_4;
   __pyx_t_4.memview = NULL;
   __pyx_t_4.data = NULL;
// >>> [SG] REFCOUNT=1 (pointed to by __pyx_v_a_view)


   /* "testtranspose.pyx":7
  *    a = numpy.arange(12).reshape([3,4])
  *    cdef numpy.int_t[:,:] a_view = a
  *    cdef numpy.int_t[:,:] b = a_view[:,:].T # <<<<<<<<<<<<<<
  *
  * testfunc()
  */
   __pyx_t_6 = -1;
   __pyx_t_5.data = __pyx_v_a_view.data;
   __pyx_t_5.memview = __pyx_v_a_view.memview;
   __PYX_INC_MEMVIEW(&__pyx_t_5, 0);
// >>> [SG] REFCOUNT=2 (pointed to by __pyx_v_a_view, __pyx_t_5)
   __pyx_t_5.shape[0] = __pyx_v_a_view.shape[0];
__pyx_t_5.strides[0] = __pyx_v_a_view.strides[0];
     __pyx_t_5.suboffsets[0] = -1;

__pyx_t_5.shape[1] = __pyx_v_a_view.shape[1];
__pyx_t_5.strides[1] = __pyx_v_a_view.strides[1];
     __pyx_t_5.suboffsets[1] = -1;

__pyx_t_7 = __pyx_t_5;
// >>> [SG] REFCOUNT=2 (pointed to by __pyx_v_a_view, __pyx_t_5, __pyx_t_7)
   if (unlikely(__pyx_memslice_transpose(&__pyx_t_7) == 0)) 
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 7; __pyx_clineno = 
__LINE__; goto __pyx_L1_error;}
   __PYX_XDEC_MEMVIEW(&__pyx_t_5, 1);
// >>> [SG] REFCOUNT=1 (pointed to by __pyx_v_a_view, __pyx_t_5, __pyx_t_7)
   __pyx_v_b = __pyx_t_7;
// >>> [SG] REFCOUNT=1 (pointed to by __pyx_v_a_view, __pyx_t_5, 
__pyx_t_7, __pyx_v_b)
   __pyx_t_7.memview = NULL;
   __pyx_t_7.data = NULL;
// >>> [SG] REFCOUNT=1 (pointed to by __pyx_v_a_view, __pyx_t_5, __pyx_v_b)

Note that now __pyx_v_a_view, __pyx_t_b and __pyx_t_5 retain references 
to the same memview, but the memview's reference count is only 1.  At 
the end of the function, the memviews of both __pyx_v_a_view and 
__pyx_v_b will both get decremented, causing a segfault.  If any error 
happens before that, then it will also attempt a decrement of __pyx_t_5 
(which is really, really annoying because the resulting segfault loses 
the actual error).  I think __pyx_t_7 needs an increment before the call 
to memslice_transpose, and that __pyx_t_5.memview needs to be set to 
NULL after its count is decremented.  The following patch fixes the 
issue, but as I'm not very familiar with Cython internals, it may be 
doing too much, or doing it in the wrong place.

-syam


diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index 315d6bb..631db8b 100644
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
@@ -614,6 +614,8 @@ class ExprNode(Node):
                  elif self.type.is_memoryviewslice:
                      code.put_xdecref_memoryviewslice(
                              self.result(), have_gil=not 
self.in_nogil_context)
+                    code.putln("%s.memview = NULL;" % self.result())
+                    code.putln("%s.data = NULL;" % self.result())
          else:
              # Already done if self.is_temp
              self.generate_subexpr_disposal_code(code)
@@ -5561,9 +5563,10 @@ class AttributeNode(ExprNode):
                          return

                  code.putln("%s = %s;" % (self.result(), 
self.obj.result()))
-                if self.obj.is_name or (self.obj.is_attribute and
- self.obj.is_memslice_transpose):
- code.put_incref_memoryviewslice(self.result(), have_gil=True)
+                #if self.obj.is_name or (self.obj.is_attribute and
+                # self.obj.is_memslice_transpose):
+                # code.put_incref_memoryviewslice(self.result(), 
have_gil=True)
+                code.put_incref_memoryviewslice(self.result(), 
have_gil=True)

                  T = "__pyx_memslice_transpose(&%s) == 0"
                  code.putln(code.error_goto_if(T % self.result(), 
self.pos))
@@ -5586,10 +5589,14 @@ class AttributeNode(ExprNode):
      def generate_disposal_code(self, code):
          if self.is_temp and self.type.is_memoryviewslice and 
self.is_memslice_transpose:
              # mirror condition for putting the memview incref here:
-            if self.obj.is_name or (self.obj.is_attribute and
- self.obj.is_memslice_transpose):
-                code.put_xdecref_memoryviewslice(
+            #if self.obj.is_name or (self.obj.is_attribute and
+            # self.obj.is_memslice_transpose):
+            #    code.put_xdecref_memoryviewslice(
+            #            self.result(), have_gil=True)
+            code.put_xdecref_memoryviewslice(
                          self.result(), have_gil=True)
+            code.putln("%s.memview = NULL;" % self.result())
+            code.putln("%s.data = NULL;" % self.result())
          else:
              ExprNode.generate_disposal_code(self, code)



On 01/28/2014 01:47 PM, Syam Gadde wrote:
> Hi again,
>
> I'm exploring the typed memoryview feature with numpy and encountered a
> refcount error.  It seems to happen when I create a slice of a
> memoryview, and transpose it.  If I don't do both, I don't get the
> refcount error.  It happens in the error cleanup code.  I suspect that
> any error that causes the function to jump to the error cleanup code
> will do the same thing as the Exception below -- but that seemed the
> easiest way to demonstrate the problem. However, if there truly is an
> error in my code, please let me know! Thanks,
>
> -syam
>
> # BEGIN CODE
> import numpy
> cimport numpy
>
> class MyException(Exception):
>       def __init__(self):
>           Exception.__init__(self)
>
> def testfunc():
>      a = numpy.arange(12).reshape([3,4])
>
>      cdef numpy.int_t[:,:] a_view = a
>
>      ## here is a slice followed by a transpose
>      cdef numpy.int_t[:,:] b = a_view[:,:].T
>      ## same thing happens with a more realistic slicing:
>      #cdef numpy.int_t[:,:] b = a_view[1:2,:].T
>      ## however, there is no error if I do this instead:
>      #cdef numpy.int_t[:,:] b = a_view.T
>      ## also no error if I do this instead:
>      #cdef numpy.int_t[:,:] b = a_view[:,:]
>
>      # The exception below is just to force the function to abort
>      # and run the extraneous __PYX_XDEC_MEMVIEW in the error
>      # cleanup code:
>      #   Fatal Python error: Acquisition count is 0 (line XXXX)
>      # Comment out this exception and we don't get the error
>      raise MyException()
>
> try:
>       testfunc()
> except MyException:
>       pass
> # END CODE
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>


From stefan_ml at behnel.de  Fri Jan 31 21:01:47 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Jan 2014 21:01:47 +0100
Subject: [Cython] breaking news in Py3.4: inspect.isfunction(cyfunction) ==
	True
Message-ID: <52EC012B.9040907@behnel.de>

Hi,

Yury Selivanov just committed a revised version of a patch I wrote for
CPython's inspect module last year. It now accepts Cython's function type
as Python function, based on the function-like attributes that it exports.

http://hg.python.org/cpython/rev/32a660a52aae

That means that things like inspect.signature(cyfunction) will also work
now, and finally have a reason to be further improved, including parameter
introspection, annotations, etc. :)

Stefan