From markflorisson88 at gmail.com  Thu Mar  1 13:03:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 1 Mar 2012 12:03:28 +0000
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4E66F9.9000303@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no> <4F4E6383.8030908@behnel.de>
	<4F4E66F9.9000303@astro.uio.no>
Message-ID: <CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>

On 29 February 2012 17:57, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/29/2012 09:42 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 29.02.2012 18:06:
>>>
>>> I'm wondering what the best course of action for deprecating the shape
>>> field in numpy.pxd is.
>>>
>>> The thing is, currently "shape" really gets in the way. In most
>>> situations
>>> it is OK with slow access to shape through the Python layer, and
>>> "arr.shape[0]" is often just fine, but currently one is in a situation
>>> where one must either write "(<object>arr).shape[0])" or
>>> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
>>> forward-compatible with NumPy.
>>
>>
>> Can Cython emulate this at the C layer? And even your work-around for the
>> Python object access looks more like a Cython bug to me. I wouldn't know
>> why that can't "just work". It usually works for other undeclared Python
>> attributes of "anything", so it might just as well be made to work here.
>
>
> Well, the problem is that shape is currently declared as a C field. It is
> also available as a Python attribute. Usually the user doesn't care which
> one is used, but the C field is declared for the few cases where access is
> speed-critical.
>
> Though even with current NumPy, I find myself doing "print
> (<object>arr).shape" in order to get a tuple rather than a Py_ssize_t*...
>
>
>>> It would really be good to do the transition as fast as possible, so that
>>> all Cython code eventually becomes ready for upcoming NumPy releases.
>>
>>
>> But it previously worked, right? It's just no longer supported in newer
>> NumPy versions IIUC? If that's the case, deleting it would break otherwise
>> working code. No-one forces you to switch to the latest NumPy version,
>> after all, and certainly not right now. A warning is much better.
>
>
> It previously worked, but it turns out that it was always frowned-upon. I
> didn't know that when I added the fields, and it was a convenient way of
> speeding things up...

I would personally prefer either cdef nogil extension class properties
(needs compiler support) or just special-casing in the compiler, which
shouldn't be too hard I think. Warnings would be a first step, but the
linkage of ndarray attributes to C attributes is really an
implementation detail, so it's better to keep supporting the
attributes correctly.

> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Thu Mar  1 17:18:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 01 Mar 2012 08:18:47 -0800
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
References: <4F4E5B22.8010701@astro.uio.no>
	<4F4E6383.8030908@behnel.de>	<4F4E66F9.9000303@astro.uio.no>
	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
Message-ID: <4F4FA167.5040500@astro.uio.no>

On 03/01/2012 04:03 AM, mark florisson wrote:
> On 29 February 2012 17:57, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 02/29/2012 09:42 AM, Stefan Behnel wrote:
>>>
>>> Dag Sverre Seljebotn, 29.02.2012 18:06:
>>>>
>>>> I'm wondering what the best course of action for deprecating the shape
>>>> field in numpy.pxd is.
>>>>
>>>> The thing is, currently "shape" really gets in the way. In most
>>>> situations
>>>> it is OK with slow access to shape through the Python layer, and
>>>> "arr.shape[0]" is often just fine, but currently one is in a situation
>>>> where one must either write "(<object>arr).shape[0])" or
>>>> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
>>>> forward-compatible with NumPy.
>>>
>>>
>>> Can Cython emulate this at the C layer? And even your work-around for the
>>> Python object access looks more like a Cython bug to me. I wouldn't know
>>> why that can't "just work". It usually works for other undeclared Python
>>> attributes of "anything", so it might just as well be made to work here.
>>
>>
>> Well, the problem is that shape is currently declared as a C field. It is
>> also available as a Python attribute. Usually the user doesn't care which
>> one is used, but the C field is declared for the few cases where access is
>> speed-critical.
>>
>> Though even with current NumPy, I find myself doing "print
>> (<object>arr).shape" in order to get a tuple rather than a Py_ssize_t*...
>>
>>
>>>> It would really be good to do the transition as fast as possible, so that
>>>> all Cython code eventually becomes ready for upcoming NumPy releases.
>>>
>>>
>>> But it previously worked, right? It's just no longer supported in newer
>>> NumPy versions IIUC? If that's the case, deleting it would break otherwise
>>> working code. No-one forces you to switch to the latest NumPy version,
>>> after all, and certainly not right now. A warning is much better.
>>
>>
>> It previously worked, but it turns out that it was always frowned-upon. I
>> didn't know that when I added the fields, and it was a convenient way of
>> speeding things up...
>
> I would personally prefer either cdef nogil extension class properties
> (needs compiler support) or just special-casing in the compiler, which
> shouldn't be too hard I think. Warnings would be a first step, but the
> linkage of ndarray attributes to C attributes is really an
> implementation detail, so it's better to keep supporting the
> attributes correctly.

So you are saying we (somehow) stick with supporting "arr.shape[0]" in 
the future, and perhaps even support "print arr.shape"? (+ arr.dim, 
arr.strides). Exactly how we could figure out at PyCon.

I'm anyway leaning towards deprecating arr.data, as it's too different 
from what the Python attribute does.

Reason I'm asking is that I'm giving a talk on Saturday, and I don't 
want to teach people bad habits -- so we must figure out what the bad 
habits are :-) (I think this applies for the PyCon poster as well...)

[1] PyData workshop at Google's offices in Mountain View; the event was 
open for all but now it is full with a long waiting list, which is why I 
didn't announce it. http://pydataworkshop.eventbrite.com/

Dag

From sturla at molden.no  Thu Mar  1 18:29:16 2012
From: sturla at molden.no (Sturla Molden)
Date: Thu, 01 Mar 2012 18:29:16 +0100
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FA167.5040500@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no>	<4F4E6383.8030908@behnel.de>	<4F4E66F9.9000303@astro.uio.no>	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
	<4F4FA167.5040500@astro.uio.no>
Message-ID: <4F4FB1EC.4020908@molden.no>

On 01.03.2012 17:18, Dag Sverre Seljebotn wrote:

> are saying we (somehow) stick with supporting "arr.shape[0]" in
> the future, and perhaps even support "print arr.shape"? (+ arr.dim,
> arr.strides).

What if you just deprecate ndarray support completely, and just focus on 
memory views?

Yes, you will break all Cython code in the world depending on ndarrays. 
But you will do that anyway by tempering with the interface. And as 
changes to the NumPy C API mandates a change to the interface, I see no 
reason to keep it. If you are going to break all code, then just do it 
completely. It is worse to stick to syntax bloat. There should not be 
multiple ways to do the same (like the zen of Python).


Sturla

From sturla at molden.no  Thu Mar  1 18:21:29 2012
From: sturla at molden.no (Sturla Molden)
Date: Thu, 01 Mar 2012 18:21:29 +0100
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FA167.5040500@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no>	<4F4E6383.8030908@behnel.de>	<4F4E66F9.9000303@astro.uio.no>	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
	<4F4FA167.5040500@astro.uio.no>
Message-ID: <4F4FB019.60800@molden.no>

On 01.03.2012 17:18, Dag Sverre Seljebotn wrote:

> I'm anyway leaning towards deprecating arr.data, as it's too different
> from what the Python attribute does.


This should be preferred, I think

    &arr[0]

or

    <char*> &arr[0]

The latter is exacty what arr.data will currently do in Cython (but not 
in Python).

But there is code in SciPy that depends on the arr.data attribute in 
Cython, such as cKDTree.


Sturla









From d.s.seljebotn at astro.uio.no  Thu Mar  1 19:33:07 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 01 Mar 2012 10:33:07 -0800
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FB1EC.4020908@molden.no>
References: <4F4E5B22.8010701@astro.uio.no>	<4F4E6383.8030908@behnel.de>	<4F4E66F9.9000303@astro.uio.no>	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>	<4F4FA167.5040500@astro.uio.no>
	<4F4FB1EC.4020908@molden.no>
Message-ID: <4F4FC0E3.2020900@astro.uio.no>

On 03/01/2012 09:29 AM, Sturla Molden wrote:
> On 01.03.2012 17:18, Dag Sverre Seljebotn wrote:
>
>> are saying we (somehow) stick with supporting "arr.shape[0]" in
>> the future, and perhaps even support "print arr.shape"? (+ arr.dim,
>> arr.strides).
>
> What if you just deprecate ndarray support completely, and just focus on
> memory views?
>
> Yes, you will break all Cython code in the world depending on ndarrays.
> But you will do that anyway by tempering with the interface. And as
> changes to the NumPy C API mandates a change to the interface, I see no
> reason to keep it. If you are going to break all code, then just do it
> completely. It is worse to stick to syntax bloat. There should not be
> multiple ways to do the same (like the zen of Python).
>

Yeah, I proposed this on another thread as one of the options, but the 
support wasn't overwhelming at the time...

About the scipy kcdTree issue, the SciPy process of generating Cython 
code manually when the code is written makes the problem slightly smaller...

Dag

From sturla at molden.no  Thu Mar  1 20:16:18 2012
From: sturla at molden.no (Sturla Molden)
Date: Thu, 01 Mar 2012 20:16:18 +0100
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FC0E3.2020900@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no>	<4F4E6383.8030908@behnel.de>	<4F4E66F9.9000303@astro.uio.no>	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>	<4F4FA167.5040500@astro.uio.no>	<4F4FB1EC.4020908@molden.no>
	<4F4FC0E3.2020900@astro.uio.no>
Message-ID: <4F4FCB02.5050105@molden.no>

On 01.03.2012 19:33, Dag Sverre Seljebotn wrote:
>
> Yeah, I proposed this on another thread as one of the options, but the
> support wasn't overwhelming at the time...

I think it is worse to break parts of it, thus introducing bugs that 
might go silent for a long time.

Rather deprecate the whole ndarray interface.


Sturla

From richsharp at stanford.edu  Fri Mar  2 07:29:25 2012
From: richsharp at stanford.edu (Richard Sharp)
Date: Thu, 1 Mar 2012 22:29:25 -0800
Subject: [Cython] RuntimeWarning: numpy.dtype size changed when cimporting
	numpy
Message-ID: <CAGuVLy0BRGbi9L4KzNMGuEa5HhPG_d_7U0mEX0xEWHg_EEjz-w@mail.gmail.com>

I have a largish cython module that outputs the following warnings
every time I import the compiled module.

__main__:1: RuntimeWarning: numpy.dtype size changed, may indicate
binary incompatibility
__main__:1: RuntimeWarning: numpy.flatiter size changed, may indicate
binary incompatibility

Is this something I should worry about?  I've traced this warning gets
generated when a `cimport numpy as np` line is processed in my module.
 When I comment out that line I don't get any warning.  To recreate
this error I've created a cython with only that line as follows:

numpy_test.pyx:

cimport numpy as np

A barebones setup in `setup.py`:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(name='numpy_test',
      version='tip',
      cmdclass={'build_ext': build_ext},
      ext_modules=[Extension(name="numpy_test", sources=['numpy_test.pyx'])]
      )

Then I do a local build like this:
python setup.py build

cd into the build directory:
cd build/lib.linux-x86_64-2.6/

verify the .so is there:
$ ls
numpy_test.so

and import it and get the runtime warning:
$ python -c "import numpy_test"
-c:1: RuntimeWarning: numpy.dtype size changed, may indicate binary
incompatibility
-c:1: RuntimeWarning: numpy.flatiter size changed, may indicate binary
incompatibility

I'm running 64 bit Ubuntu 10.04, Python 2.6.6, and Cython 0.15.1.

Any help would be greatly appreciated!

Rich

--
Richard P. Sharp Jr.
Lead Software Developer
Natural Capital Project
Stanford University, U Minnesota, TNC, WWF
371 Serra Mall
Stanford, CA 94305
http://www.stanford.edu/~rpsharp/

From stefan_ml at behnel.de  Fri Mar  2 10:45:02 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 02 Mar 2012 10:45:02 +0100
Subject: [Cython] Problem with Py_buffer struct definition in Builtin.py
Message-ID: <4F50969E.9060103@behnel.de>

Hi,

the builtin Py_buffer struct type is currently defined as follows:

"""
builtin_structs_table = [
    ('Py_buffer', 'Py_buffer',
     [("buf",        PyrexTypes.c_void_ptr_type),
      ("obj",        PyrexTypes.py_object_type),
      ("len",        PyrexTypes.c_py_ssize_t_type),
      ...
"""

I hadn't noticed this before, but when you actually use it in a
"__getbuffer__()" special method, you have to assign the buffer owner (i.e.
self) to the .obj field, which is currently defined as "object". Meaning,
Cython will do reference counting for it. That's fine for the self
reference being assigned to it. However, when "__getbuffer__()" gets
called, .obj is not assigned, i.e. it may be NULL, or it may contain
garbage, depending on the caller. So, Cython will generate broken code for
it when trying to DECREF the original value as part of the assignment.

"""
 *         buffer.obj = self             # <<<<<<<<<<<<<<
 */
  __Pyx_INCREF(((PyObject *)__pyx_v_self));
  __Pyx_GIVEREF(((PyObject *)__pyx_v_self));
  __Pyx_GOTREF(__pyx_v_buffer->obj);
  __Pyx_DECREF(__pyx_v_buffer->obj);           // please crash for me
  __pyx_v_buffer->obj = ((PyObject *)__pyx_v_self);
"""

I see a couple of ways to fix this:

1) Change the field to PyObject* instead of object (and finally make
PyObject a known type in Cython). Problem: may break existing code,
although that's currently broken anyway. Also makes it more annoying for
users who then have to do the manual cast+INCREF dance.

2) Special case this field and do not do any reference counting for its
current value. PyBuffer_Release() will properly do the XDECREF cleanup
anyway, and it's unlikely that user code will assign to it more than once.
Problem: very unexpected behaviour for users in case they ever *do*
multiple assignments.

3) Wrap the user provided "__getbuffer__()" special method in code that
always assigns "self" to the .obj field before entering the user code and
also clears it on error return. That would mean that it will always contain
a valid value, so that ref-counting will work as expected. Problem: this
complicates the implementation and may not be obvious for users (who don't
read the documentation). It also won't fix any code that uses Py_buffer
outside of "__getbuffer__()" or that calls "__releasebuffer__()" directly
without going through PyBuffer_Release(). However, this actually makes it
easier for users, who can then even skip setting the value in all but some
very rare cases and just care about the actual buffer setup in their
"__getbuffer__()" code.

I'm leaning towards 3). What do you think?

I also think that this is worth fixing for 0.16 if possible, because it
currently generates crash code from innocent and correct looking user code.

Stefan

From stefan_ml at behnel.de  Fri Mar  2 11:09:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 02 Mar 2012 11:09:14 +0100
Subject: [Cython] Problem with Py_buffer struct definition in Builtin.py
In-Reply-To: <4F50969E.9060103@behnel.de>
References: <4F50969E.9060103@behnel.de>
Message-ID: <4F509C4A.2000708@behnel.de>

Stefan Behnel, 02.03.2012 10:45:
> the builtin Py_buffer struct type is currently defined as follows:
> 
> """
> builtin_structs_table = [
>     ('Py_buffer', 'Py_buffer',
>      [("buf",        PyrexTypes.c_void_ptr_type),
>       ("obj",        PyrexTypes.py_object_type),
>       ("len",        PyrexTypes.c_py_ssize_t_type),
>       ...
> """
> 
> I hadn't noticed this before, but when you actually use it in a
> "__getbuffer__()" special method, you have to assign the buffer owner (i.e.
> self) to the .obj field, which is currently defined as "object".

Oh, well, I should really learn to read code before composing a lengthy
e-mail...

"__getbuffer__()" is already special-cased and sets the value to None. I
think I even recall that we discussed this back when Dag implemented
support for buffers. The reason I had originally noticed this was this
recent change in Py3.3:

"""
static PyObject *
_PyManagedBuffer_FromObject(PyObject *base)
{
    _PyManagedBufferObject *mbuf;

    mbuf = mbuf_alloc();
    if (mbuf == NULL)
        return NULL;

    if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) {
        /* mbuf->master.obj must be NULL. */
        Py_DECREF(mbuf);
        return NULL;
    }

    /* Assume that master.obj is a new reference to base. */
    assert(mbuf->master.obj == base);

    return (PyObject *)mbuf;
}
"""

Note the assertion (which is unreleased as of now, so it may still be
subject to changes). Is there any reason the value should be set to None by
Cython's special casing code instead of self?

Stefan

From markflorisson88 at gmail.com  Fri Mar  2 17:29:46 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 2 Mar 2012 16:29:46 +0000
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FA167.5040500@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no> <4F4E6383.8030908@behnel.de>
	<4F4E66F9.9000303@astro.uio.no>
	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
	<4F4FA167.5040500@astro.uio.no>
Message-ID: <CANg26EUtbJeMzpLpM+6f9zkbui-k-8wh0HJmu+fTpgnaeBVTAg@mail.gmail.com>

On 1 March 2012 16:18, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 03/01/2012 04:03 AM, mark florisson wrote:
>>
>> On 29 February 2012 17:57, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 02/29/2012 09:42 AM, Stefan Behnel wrote:
>>>>
>>>>
>>>> Dag Sverre Seljebotn, 29.02.2012 18:06:
>>>>>
>>>>>
>>>>> I'm wondering what the best course of action for deprecating the shape
>>>>> field in numpy.pxd is.
>>>>>
>>>>> The thing is, currently "shape" really gets in the way. In most
>>>>> situations
>>>>> it is OK with slow access to shape through the Python layer, and
>>>>> "arr.shape[0]" is often just fine, but currently one is in a situation
>>>>> where one must either write "(<object>arr).shape[0])" or
>>>>> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
>>>>> forward-compatible with NumPy.
>>>>
>>>>
>>>>
>>>> Can Cython emulate this at the C layer? And even your work-around for
>>>> the
>>>> Python object access looks more like a Cython bug to me. I wouldn't know
>>>> why that can't "just work". It usually works for other undeclared Python
>>>> attributes of "anything", so it might just as well be made to work here.
>>>
>>>
>>>
>>> Well, the problem is that shape is currently declared as a C field. It is
>>> also available as a Python attribute. Usually the user doesn't care which
>>> one is used, but the C field is declared for the few cases where access
>>> is
>>> speed-critical.
>>>
>>> Though even with current NumPy, I find myself doing "print
>>> (<object>arr).shape" in order to get a tuple rather than a Py_ssize_t*...
>>>
>>>
>>>>> It would really be good to do the transition as fast as possible, so
>>>>> that
>>>>> all Cython code eventually becomes ready for upcoming NumPy releases.
>>>>
>>>>
>>>>
>>>> But it previously worked, right? It's just no longer supported in newer
>>>> NumPy versions IIUC? If that's the case, deleting it would break
>>>> otherwise
>>>> working code. No-one forces you to switch to the latest NumPy version,
>>>> after all, and certainly not right now. A warning is much better.
>>>
>>>
>>>
>>> It previously worked, but it turns out that it was always frowned-upon. I
>>> didn't know that when I added the fields, and it was a convenient way of
>>> speeding things up...
>>
>>
>> I would personally prefer either cdef nogil extension class properties
>> (needs compiler support) or just special-casing in the compiler, which
>> shouldn't be too hard I think. Warnings would be a first step, but the
>> linkage of ndarray attributes to C attributes is really an
>> implementation detail, so it's better to keep supporting the
>> attributes correctly.
>
>
> So you are saying we (somehow) stick with supporting "arr.shape[0]" in the
> future, and perhaps even support "print arr.shape"? (+ arr.dim,
> arr.strides). Exactly how we could figure out at PyCon.

To remain consistent with previous versions the former should be
supported and the latter would be a bonus (and it wouldn't be too hard
anyway).

> I'm anyway leaning towards deprecating arr.data, as it's too different from
> what the Python attribute does.

+1 for that, just write &arr[0] as Sturla mentioned. The transition
should be trivial.

> Reason I'm asking is that I'm giving a talk on Saturday, and I don't want to
> teach people bad habits -- so we must figure out what the bad habits are :-)
> (I think this applies for the PyCon poster as well...)
>
> [1] PyData workshop at Google's offices in Mountain View; the event was open
> for all but now it is full with a long waiting list, which is why I didn't
> announce it. http://pydataworkshop.eventbrite.com/
>
>
> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Fri Mar  2 17:33:45 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 2 Mar 2012 16:33:45 +0000
Subject: [Cython] Problem with Py_buffer struct definition in Builtin.py
In-Reply-To: <4F509C4A.2000708@behnel.de>
References: <4F50969E.9060103@behnel.de>
	<4F509C4A.2000708@behnel.de>
Message-ID: <CANg26EUTzDnXQ8K77KkJD9MWhV7R=Acg_Qpt-vO-tnO-xERwpw@mail.gmail.com>

On 2 March 2012 10:09, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 02.03.2012 10:45:
>> the builtin Py_buffer struct type is currently defined as follows:
>>
>> """
>> builtin_structs_table = [
>> ? ? ('Py_buffer', 'Py_buffer',
>> ? ? ?[("buf", ? ? ? ?PyrexTypes.c_void_ptr_type),
>> ? ? ? ("obj", ? ? ? ?PyrexTypes.py_object_type),
>> ? ? ? ("len", ? ? ? ?PyrexTypes.c_py_ssize_t_type),
>> ? ? ? ...
>> """
>>
>> I hadn't noticed this before, but when you actually use it in a
>> "__getbuffer__()" special method, you have to assign the buffer owner (i.e.
>> self) to the .obj field, which is currently defined as "object".
>
> Oh, well, I should really learn to read code before composing a lengthy
> e-mail...
>
> "__getbuffer__()" is already special-cased and sets the value to None. I
> think I even recall that we discussed this back when Dag implemented
> support for buffers. The reason I had originally noticed this was this
> recent change in Py3.3:
>
> """
> static PyObject *
> _PyManagedBuffer_FromObject(PyObject *base)
> {
> ? ?_PyManagedBufferObject *mbuf;
>
> ? ?mbuf = mbuf_alloc();
> ? ?if (mbuf == NULL)
> ? ? ? ?return NULL;
>
> ? ?if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) {
> ? ? ? ?/* mbuf->master.obj must be NULL. */
> ? ? ? ?Py_DECREF(mbuf);
> ? ? ? ?return NULL;
> ? ?}
>
> ? ?/* Assume that master.obj is a new reference to base. */
> ? ?assert(mbuf->master.obj == base);
>
> ? ?return (PyObject *)mbuf;
> }
> """
>
> Note the assertion (which is unreleased as of now, so it may still be
> subject to changes). Is there any reason the value should be set to None by
> Cython's special casing code instead of self?

It sets it to None to it can later reset it to NULL. Python 3.3 is the
first version to document the 'obj' Py_buffer attribute, and it
mentions that the exporter may set it to NULL. The code above is not
the PyObject_GetBuffer function, so setting it to NULL should still
work. Defaulting it to 'self' instead would work equally well I'd
think.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Fri Mar  2 17:29:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 2 Mar 2012 16:29:22 +0000
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4FCB02.5050105@molden.no>
References: <4F4E5B22.8010701@astro.uio.no> <4F4E6383.8030908@behnel.de>
	<4F4E66F9.9000303@astro.uio.no>
	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
	<4F4FA167.5040500@astro.uio.no> <4F4FB1EC.4020908@molden.no>
	<4F4FC0E3.2020900@astro.uio.no> <4F4FCB02.5050105@molden.no>
Message-ID: <CANg26EU74yOYkHP4=Ps4cDts-5nvqX4smWkeO=_eVOZ-vkRZcA@mail.gmail.com>

On 1 March 2012 19:16, Sturla Molden <sturla at molden.no> wrote:
> On 01.03.2012 19:33, Dag Sverre Seljebotn wrote:
>>
>>
>> Yeah, I proposed this on another thread as one of the options, but the
>> support wasn't overwhelming at the time...
>
>
> I think it is worse to break parts of it, thus introducing bugs that might
> go silent for a long time.

The point is that we would remain fully backwards compatible (except
for the data attribute perhaps) and support the attributes portably
across numpy versions, nothing would be broken or silent.

> Rather deprecate the whole ndarray interface.
>

As much as I would like that, you can't just break everyone's code in
a new release. I think the syntax should be removed several versions
later, or maybe even wait until Cython 1.0.

> Sturla
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From dtcaciuc at gmail.com  Fri Mar  2 17:59:30 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Fri, 2 Mar 2012 08:59:30 -0800
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <CANg26EUtbJeMzpLpM+6f9zkbui-k-8wh0HJmu+fTpgnaeBVTAg@mail.gmail.com>
References: <4F4E5B22.8010701@astro.uio.no> <4F4E6383.8030908@behnel.de>
	<4F4E66F9.9000303@astro.uio.no>
	<CANg26EWu=saT+hKgXn0c2jJvBbwW8xaCavU6pfDPyxfB=x2b4Q@mail.gmail.com>
	<4F4FA167.5040500@astro.uio.no>
	<CANg26EUtbJeMzpLpM+6f9zkbui-k-8wh0HJmu+fTpgnaeBVTAg@mail.gmail.com>
Message-ID: <CAL41sfT1vAOQA3RNo2A=RLi-uqytiDxGyS0asLuCRJLdn_koRA@mail.gmail.com>

On Fri, Mar 2, 2012 at 8:29 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 1 March 2012 16:18, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> On 03/01/2012 04:03 AM, mark florisson wrote:
>>>
>>> On 29 February 2012 17:57, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> On 02/29/2012 09:42 AM, Stefan Behnel wrote:
>>>>>
>>>>>
>>>>> Dag Sverre Seljebotn, 29.02.2012 18:06:
>>>>>>
>>>>>>
>>>>>> I'm wondering what the best course of action for deprecating the shape
>>>>>> field in numpy.pxd is.
>>>>>>
>>>>>> The thing is, currently "shape" really gets in the way. In most
>>>>>> situations
>>>>>> it is OK with slow access to shape through the Python layer, and
>>>>>> "arr.shape[0]" is often just fine, but currently one is in a situation
>>>>>> where one must either write "(<object>arr).shape[0])" or
>>>>>> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
>>>>>> forward-compatible with NumPy.
>>>>>
>>>>>
>>>>>
>>>>> Can Cython emulate this at the C layer? And even your work-around for
>>>>> the
>>>>> Python object access looks more like a Cython bug to me. I wouldn't know
>>>>> why that can't "just work". It usually works for other undeclared Python
>>>>> attributes of "anything", so it might just as well be made to work here.
>>>>
>>>>
>>>>
>>>> Well, the problem is that shape is currently declared as a C field. It is
>>>> also available as a Python attribute. Usually the user doesn't care which
>>>> one is used, but the C field is declared for the few cases where access
>>>> is
>>>> speed-critical.
>>>>
>>>> Though even with current NumPy, I find myself doing "print
>>>> (<object>arr).shape" in order to get a tuple rather than a Py_ssize_t*...
>>>>
>>>>
>>>>>> It would really be good to do the transition as fast as possible, so
>>>>>> that
>>>>>> all Cython code eventually becomes ready for upcoming NumPy releases.
>>>>>
>>>>>
>>>>>
>>>>> But it previously worked, right? It's just no longer supported in newer
>>>>> NumPy versions IIUC? If that's the case, deleting it would break
>>>>> otherwise
>>>>> working code. No-one forces you to switch to the latest NumPy version,
>>>>> after all, and certainly not right now. A warning is much better.
>>>>
>>>>
>>>>
>>>> It previously worked, but it turns out that it was always frowned-upon. I
>>>> didn't know that when I added the fields, and it was a convenient way of
>>>> speeding things up...
>>>
>>>
>>> I would personally prefer either cdef nogil extension class properties
>>> (needs compiler support) or just special-casing in the compiler, which
>>> shouldn't be too hard I think. Warnings would be a first step, but the
>>> linkage of ndarray attributes to C attributes is really an
>>> implementation detail, so it's better to keep supporting the
>>> attributes correctly.
>>
>>
>> So you are saying we (somehow) stick with supporting "arr.shape[0]" in the
>> future, and perhaps even support "print arr.shape"? (+ arr.dim,
>> arr.strides). Exactly how we could figure out at PyCon.
>
> To remain consistent with previous versions the former should be
> supported and the latter would be a bonus (and it wouldn't be too hard
> anyway).
>
>> I'm anyway leaning towards deprecating arr.data, as it's too different from
>> what the Python attribute does.
>
> +1 for that, just write &arr[0] as Sturla mentioned. The transition
> should be trivial.

If there's a confusion due to .data already having a certain meaning
with the python attribute, perhaps it would make sense to have an
attribute with a different name, eg. .ptr or .voidptr ? IMHO writing
&arr[0] looks like a workaround of some kind. Like, when in C you had
something like a 2d array and you'd need to interpret it as a 1d array
you'd write &arr[0][0], but C array syntax doesn't support attributes
which you can add here. Unless of course the idea is to make arrays to
behave and look exactly like C counterparts.

Dimitri.

>
>> Reason I'm asking is that I'm giving a talk on Saturday, and I don't want to
>> teach people bad habits -- so we must figure out what the bad habits are :-)
>> (I think this applies for the PyCon poster as well...)
>>
>> [1] PyData workshop at Google's offices in Mountain View; the event was open
>> for all but now it is full with a long waiting list, which is why I didn't
>> announce it. http://pydataworkshop.eventbrite.com/
>>
>>
>> Dag
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sat Mar  3 10:15:34 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 03 Mar 2012 10:15:34 +0100
Subject: [Cython] Problem with Py_buffer struct definition in Builtin.py
In-Reply-To: <CANg26EUTzDnXQ8K77KkJD9MWhV7R=Acg_Qpt-vO-tnO-xERwpw@mail.gmail.com>
References: <4F50969E.9060103@behnel.de> <4F509C4A.2000708@behnel.de>
	<CANg26EUTzDnXQ8K77KkJD9MWhV7R=Acg_Qpt-vO-tnO-xERwpw@mail.gmail.com>
Message-ID: <4F51E136.5010209@behnel.de>

mark florisson, 02.03.2012 17:33:
> On 2 March 2012 10:09, Stefan Behnel wrote:
>> Stefan Behnel, 02.03.2012 10:45:
>>> the builtin Py_buffer struct type is currently defined as follows:
>>>
>>> """
>>> builtin_structs_table = [
>>>     ('Py_buffer', 'Py_buffer',
>>>      [("buf",        PyrexTypes.c_void_ptr_type),
>>>       ("obj",        PyrexTypes.py_object_type),
>>>       ("len",        PyrexTypes.c_py_ssize_t_type),
>>>       ...
>>> """
>>>
>>> I hadn't noticed this before, but when you actually use it in a
>>> "__getbuffer__()" special method, you have to assign the buffer owner (i.e.
>>> self) to the .obj field, which is currently defined as "object".
>>
>> Oh, well, I should really learn to read code before composing a lengthy
>> e-mail...
>>
>> "__getbuffer__()" is already special-cased and sets the value to None. I
>> think I even recall that we discussed this back when Dag implemented
>> support for buffers. The reason I had originally noticed this was this
>> recent change in Py3.3:
>>
>> """
>> static PyObject *
>> _PyManagedBuffer_FromObject(PyObject *base)
>> {
>>    _PyManagedBufferObject *mbuf;
>>
>>    mbuf = mbuf_alloc();
>>    if (mbuf == NULL)
>>        return NULL;
>>
>>    if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) {
>>        /* mbuf->master.obj must be NULL. */
>>        Py_DECREF(mbuf);
>>        return NULL;
>>    }
>>
>>    /* Assume that master.obj is a new reference to base. */
>>    assert(mbuf->master.obj == base);

I asked on python-dev and I think this assert will be removed.

http://thread.gmane.org/gmane.comp.python.devel/130365

There is now a ticket to allow buffer redirection by "__getbuffer__()" as a
general feature.

http://bugs.python.org/issue14181


>>    return (PyObject *)mbuf;
>> }
>> """
>>
>> Note the assertion (which is unreleased as of now, so it may still be
>> subject to changes). Is there any reason the value should be set to None by
>> Cython's special casing code instead of self?
> 
> It sets it to None to it can later reset it to NULL. Python 3.3 is the
> first version to document the 'obj' Py_buffer attribute, and it
> mentions that the exporter may set it to NULL.

I don't find it very clear on that part, though. The way I read it, NULL is
only to be set on error (which Cython does already).


> The code above is not
> the PyObject_GetBuffer function, so setting it to NULL should still
> work. Defaulting it to 'self' instead would work equally well I'd
> think.

I'll wait and see what the discussion on python-dev brings up. If the
outcome is that NULL is for errors and that it should have a value
otherwise, I think 'self' is a better default.

Stefan

From arfrever at gentoo.org  Mon Mar  5 05:41:51 2012
From: arfrever at gentoo.org (Arfrever Frehtes Taifersar Arahesis)
Date: Mon, 5 Mar 2012 05:41:51 +0100
Subject: [Cython] Cython 0.16 Beta 0
In-Reply-To: <CANg26EXhkDzL6=RXFhgqyVJMFTfoSmKpra4HGsv+e9f58MKaJQ@mail.gmail.com>
References: <CANg26EXhkDzL6=RXFhgqyVJMFTfoSmKpra4HGsv+e9f58MKaJQ@mail.gmail.com>
Message-ID: <201203050541.53665.Arfrever.FTA@gmail.com>

All tests pass with Python 2.6 (2.6.7 release).
All tests pass with Python 2.7 (snapshot of 2.7 branch, revision 52ecec12c0ed).
10 failures with Python 3.1 (3.1.4 release).
14 failures with Python 3.2 (snapshot of 3.2 branch, revision 4966907d3661).

Failures with Python 3.2:

======================================================================
FAIL: test_copy_to (memoryviewattrs.__test__)
Doctest: memoryviewattrs.__test__.test_copy_to
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for memoryviewattrs.__test__.test_copy_to
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/memoryviewattrs.cpython-32.so", line unknown line number, in test_copy_to

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/memoryviewattrs.cpython-32.so", line ?, in memoryviewattrs.__test__.test_copy_to
Failed example:
    test_copy_to()
Expected:
    0 1 2 3 4 5 6 7
    0 1 2 3 4 5 6 7
    0 1 2 3 4 5 6 7
Got:
    0 1 2 3 4 5 6 7 
    0 1 2 3 4 5 6 7 
    0 1 2 3 4 5 6 7 


======================================================================
FAIL: test_copy_to (memoryviewattrs.__test__)
Doctest: memoryviewattrs.__test__.test_copy_to
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for memoryviewattrs.__test__.test_copy_to
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/memoryviewattrs.cpython-32.so", line unknown line number, in test_copy_to

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/memoryviewattrs.cpython-32.so", line ?, in memoryviewattrs.__test__.test_copy_to
Failed example:
    test_copy_to()
Expected:
    0 1 2 3 4 5 6 7
    0 1 2 3 4 5 6 7
    0 1 2 3 4 5 6 7
Got:
    0 1 2 3 4 5 6 7 
    0 1 2 3 4 5 6 7 
    0 1 2 3 4 5 6 7 


======================================================================
FAIL: test_coerce_to_numpy (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_coerce_to_numpy
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_coerce_to_numpy
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_coerce_to_numpy

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_coerce_to_numpy
Failed example:
    test_coerce_to_numpy()
Expected:
    (97, 98, 600L, 700, 800)
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.3
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.6+17.7j)
    deallocating...
    (18.8+19.9j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...
Got:
    (97, 98, 600, 700, 800)
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.3
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.6+17.7j)
    deallocating...
    (18.8+19.9j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...


======================================================================
FAIL: test_memslice_structarray (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_memslice_structarray
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_memslice_structarray
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_memslice_structarray

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
Failed example:
    test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_memslice_structarray[2]>", line 1, in <module>
        test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
      File "numpy_memoryview.pyx", line 464, in numpy_memoryview.test_memslice_structarray (numpy_memoryview.c:15496)
    TypeError: int() argument must be a string or a number, not 'map'
----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
Failed example:
    test_memslice_structarray(data, dtype)
Expected:
    0
    1
    2
    3
    spam
    4
    5
    6
    7
    ham
    8
    9
    10
    11
    eggs
Got:
    0
    1
    2
    3
    b'spam'
    4
    5
    6
    7
    b'ham'
    8
    9
    10
    11
    b'eggs'


======================================================================
FAIL: test_string_invalid_dims (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_string_invalid_dims
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_string_invalid_dims
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_string_invalid_dims

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_string_invalid_dims
Failed example:
    stringstructtest(np.array(data, dtype=dtype))
Expected:
    Traceback (most recent call last):
       ...
    ValueError: Expected 2 dimensions, got 1
Got:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_string_invalid_dims[2]>", line 1, in <module>
        stringstructtest(np.array(data, dtype=dtype))
    TypeError: expected an object with a buffer interface


======================================================================
FAIL: test_struct_attributes (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_struct_attributes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_struct_attributes
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_struct_attributes

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_struct_attributes
Failed example:
    test_struct_attributes()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_struct_attributes[0]>", line 1, in <module>
        test_struct_attributes()
      File "numpy_memoryview.pyx", line 554, in numpy_memoryview.test_struct_attributes (numpy_memoryview.c:15960)
    IndexError: invalid index


======================================================================
FAIL: test_coerce_to_numpy (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_coerce_to_numpy
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_coerce_to_numpy
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_coerce_to_numpy

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_coerce_to_numpy
Failed example:
    test_coerce_to_numpy()
Expected:
    (97, 98, 600L, 700, 800)
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.3
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.6+17.7j)
    deallocating...
    (18.8+19.9j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...
Got:
    (97, 98, 600, 700, 800)
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.3
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.6+17.7j)
    deallocating...
    (18.8+19.9j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...


======================================================================
FAIL: test_memslice_structarray (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_memslice_structarray
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_memslice_structarray
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_memslice_structarray

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
Failed example:
    test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_memslice_structarray[2]>", line 1, in <module>
        test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
      File "numpy_memoryview.pyx", line 464, in numpy_memoryview.test_memslice_structarray (numpy_memoryview.cpp:15496)
    TypeError: int() argument must be a string or a number, not 'map'
----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
Failed example:
    test_memslice_structarray(data, dtype)
Expected:
    0
    1
    2
    3
    spam
    4
    5
    6
    7
    ham
    8
    9
    10
    11
    eggs
Got:
    0
    1
    2
    3
    b'spam'
    4
    5
    6
    7
    b'ham'
    8
    9
    10
    11
    b'eggs'


======================================================================
FAIL: test_string_invalid_dims (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_string_invalid_dims
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_string_invalid_dims
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_string_invalid_dims

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_string_invalid_dims
Failed example:
    stringstructtest(np.array(data, dtype=dtype))
Expected:
    Traceback (most recent call last):
       ...
    ValueError: Expected 2 dimensions, got 1
Got:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_string_invalid_dims[2]>", line 1, in <module>
        stringstructtest(np.array(data, dtype=dtype))
    TypeError: expected an object with a buffer interface


======================================================================
FAIL: test_struct_attributes (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_struct_attributes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_struct_attributes
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_struct_attributes

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_struct_attributes
Failed example:
    test_struct_attributes()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest numpy_memoryview.__test__.test_struct_attributes[0]>", line 1, in <module>
        test_struct_attributes()
      File "numpy_memoryview.pyx", line 554, in numpy_memoryview.test_struct_attributes (numpy_memoryview.cpp:15960)
    IndexError: invalid index


======================================================================
FAIL: NestedWith (withstat)
Doctest: withstat.NestedWith
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for withstat.NestedWith
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat.cpython-32.so", line unknown line number, in NestedWith

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat.cpython-32.so", line ?, in withstat.NestedWith
Failed example:
    NestedWith().runTest()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest withstat.NestedWith[0]>", line 1, in <module>
        NestedWith().runTest()
      File "withstat.pyx", line 183, in withstat.NestedWith.runTest (withstat.c:5544)
      File "withstat.pyx", line 222, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:8071)
      File "withstat.pyx", line 223, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:7959)
      File "withstat.pyx", line 224, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:7808)
      File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
        DeprecationWarning, 2)
      File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
        file.write(formatwarning(message, category, filename, lineno, line))
      File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
        line = linecache.getline(filename, lineno) if line is None else line
      File "/usr/lib64/python3.2/linecache.py", line 15, in getline
        lines = getlines(filename, module_globals)
      File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
        return self.save_linecache_getlines(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
        return updatecache(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
        lines = fp.readlines()
      File "/usr/lib64/python3.2/codecs.py", line 300, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 40: invalid start byte


======================================================================
FAIL: NestedWith (withstat)
Doctest: withstat.NestedWith
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for withstat.NestedWith
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat.cpython-32.so", line unknown line number, in NestedWith

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat.cpython-32.so", line ?, in withstat.NestedWith
Failed example:
    NestedWith().runTest()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest withstat.NestedWith[0]>", line 1, in <module>
        NestedWith().runTest()
      File "withstat.pyx", line 183, in withstat.NestedWith.runTest (withstat.cpp:5544)
      File "withstat.pyx", line 222, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:8071)
      File "withstat.pyx", line 223, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:7959)
      File "withstat.pyx", line 224, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:7808)
      File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
        DeprecationWarning, 2)
      File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
        file.write(formatwarning(message, category, filename, lineno, line))
      File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
        line = linecache.getline(filename, lineno) if line is None else line
      File "/usr/lib64/python3.2/linecache.py", line 15, in getline
        lines = getlines(filename, module_globals)
      File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
        return self.save_linecache_getlines(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
        return updatecache(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
        lines = fp.readlines()
      File "/usr/lib64/python3.2/codecs.py", line 300, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 24: invalid continuation byte


======================================================================
FAIL: NestedWith (withstat_py)
Doctest: withstat_py.NestedWith
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for withstat_py.NestedWith
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat_py.cpython-32.so", line unknown line number, in NestedWith

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat_py.cpython-32.so", line ?, in withstat_py.NestedWith
Failed example:
    NestedWith().runTest()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest withstat_py.NestedWith[0]>", line 1, in <module>
        NestedWith().runTest()
      File "withstat_py.py", line 250, in withstat_py.NestedWith.runTest (withstat_py.c:7232)
      File "withstat_py.py", line 289, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9759)
      File "withstat_py.py", line 290, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9647)
      File "withstat_py.py", line 291, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9496)
      File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
        DeprecationWarning, 2)
      File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
        file.write(formatwarning(message, category, filename, lineno, line))
      File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
        line = linecache.getline(filename, lineno) if line is None else line
      File "/usr/lib64/python3.2/linecache.py", line 15, in getline
        lines = getlines(filename, module_globals)
      File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
        return self.save_linecache_getlines(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
        return updatecache(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
        lines = fp.readlines()
      File "/usr/lib64/python3.2/codecs.py", line 300, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 24: invalid start byte


======================================================================
FAIL: NestedWith (withstat_py)
Doctest: withstat_py.NestedWith
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for withstat_py.NestedWith
  File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat_py.cpython-32.so", line unknown line number, in NestedWith

----------------------------------------------------------------------
File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat_py.cpython-32.so", line ?, in withstat_py.NestedWith
Failed example:
    NestedWith().runTest()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest withstat_py.NestedWith[0]>", line 1, in <module>
        NestedWith().runTest()
      File "withstat_py.py", line 250, in withstat_py.NestedWith.runTest (withstat_py.cpp:7232)
      File "withstat_py.py", line 289, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9759)
      File "withstat_py.py", line 290, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9647)
      File "withstat_py.py", line 291, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9496)
      File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
        DeprecationWarning, 2)
      File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
        file.write(formatwarning(message, category, filename, lineno, line))
      File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
        line = linecache.getline(filename, lineno) if line is None else line
      File "/usr/lib64/python3.2/linecache.py", line 15, in getline
        lines = getlines(filename, module_globals)
      File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
        return self.save_linecache_getlines(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
        return updatecache(filename, module_globals)
      File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
        lines = fp.readlines()
      File "/usr/lib64/python3.2/codecs.py", line 300, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 24: invalid start byte


----------------------------------------------------------------------
Ran 6457 tests in 2232.009s

FAILED (failures=14)
ALL DONE

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120305/9a7c3118/attachment-0001.pgp>

From markflorisson88 at gmail.com  Mon Mar  5 14:46:25 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 5 Mar 2012 14:46:25 +0100
Subject: [Cython] Cython 0.16 Beta 0
In-Reply-To: <201203050541.53665.Arfrever.FTA@gmail.com>
References: <CANg26EXhkDzL6=RXFhgqyVJMFTfoSmKpra4HGsv+e9f58MKaJQ@mail.gmail.com>
	<201203050541.53665.Arfrever.FTA@gmail.com>
Message-ID: <CANg26EWb+RG0TS7TSyWGV7qTG-A3myk1LyjYXNhmifHNJmk2=g@mail.gmail.com>

On 5 March 2012 05:41, Arfrever Frehtes Taifersar Arahesis
<arfrever at gentoo.org> wrote:
> All tests pass with Python 2.6 (2.6.7 release).
> All tests pass with Python 2.7 (snapshot of 2.7 branch, revision 52ecec12c0ed).
> 10 failures with Python 3.1 (3.1.4 release).
> 14 failures with Python 3.2 (snapshot of 3.2 branch, revision 4966907d3661).
>
> Failures with Python 3.2:
>
> ======================================================================
> FAIL: test_copy_to (memoryviewattrs.__test__)
> Doctest: memoryviewattrs.__test__.test_copy_to
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for memoryviewattrs.__test__.test_copy_to
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/memoryviewattrs.cpython-32.so", line unknown line number, in test_copy_to
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/memoryviewattrs.cpython-32.so", line ?, in memoryviewattrs.__test__.test_copy_to
> Failed example:
> ? ?test_copy_to()
> Expected:
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> Got:
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
>
>
> ======================================================================
> FAIL: test_copy_to (memoryviewattrs.__test__)
> Doctest: memoryviewattrs.__test__.test_copy_to
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for memoryviewattrs.__test__.test_copy_to
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/memoryviewattrs.cpython-32.so", line unknown line number, in test_copy_to
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/memoryviewattrs.cpython-32.so", line ?, in memoryviewattrs.__test__.test_copy_to
> Failed example:
> ? ?test_copy_to()
> Expected:
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> Got:
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
> ? ?0 1 2 3 4 5 6 7
>
>
> ======================================================================
> FAIL: test_coerce_to_numpy (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_coerce_to_numpy
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_coerce_to_numpy
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_coerce_to_numpy
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_coerce_to_numpy
> Failed example:
> ? ?test_coerce_to_numpy()
> Expected:
> ? ?(97, 98, 600L, 700, 800)
> ? ?deallocating...
> ? ?(600, 700)
> ? ?deallocating...
> ? ?((100, 200), (300, 400), 500)
> ? ?deallocating...
> ? ?(97, 900)
> ? ?deallocating...
> ? ?99
> ? ?deallocating...
> ? ?111
> ? ?deallocating...
> ? ?222
> ? ?deallocating...
> ? ?333
> ? ?deallocating...
> ? ?11.1
> ? ?deallocating...
> ? ?12.2
> ? ?deallocating...
> ? ?13.3
> ? ?deallocating...
> ? ?(14.4+15.5j)
> ? ?deallocating...
> ? ?(16.6+17.7j)
> ? ?deallocating...
> ? ?(18.8+19.9j)
> ? ?deallocating...
> ? ?22
> ? ?deallocating...
> ? ?33.33
> ? ?deallocating...
> ? ?44
> ? ?deallocating...
> Got:
> ? ?(97, 98, 600, 700, 800)
> ? ?deallocating...
> ? ?(600, 700)
> ? ?deallocating...
> ? ?((100, 200), (300, 400), 500)
> ? ?deallocating...
> ? ?(97, 900)
> ? ?deallocating...
> ? ?99
> ? ?deallocating...
> ? ?111
> ? ?deallocating...
> ? ?222
> ? ?deallocating...
> ? ?333
> ? ?deallocating...
> ? ?11.1
> ? ?deallocating...
> ? ?12.2
> ? ?deallocating...
> ? ?13.3
> ? ?deallocating...
> ? ?(14.4+15.5j)
> ? ?deallocating...
> ? ?(16.6+17.7j)
> ? ?deallocating...
> ? ?(18.8+19.9j)
> ? ?deallocating...
> ? ?22
> ? ?deallocating...
> ? ?33.33
> ? ?deallocating...
> ? ?44
> ? ?deallocating...
>
>
> ======================================================================
> FAIL: test_memslice_structarray (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_memslice_structarray
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_memslice_structarray
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_memslice_structarray
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
> Failed example:
> ? ?test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_memslice_structarray[2]>", line 1, in <module>
> ? ? ? ?test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
> ? ? ?File "numpy_memoryview.pyx", line 464, in numpy_memoryview.test_memslice_structarray (numpy_memoryview.c:15496)
> ? ?TypeError: int() argument must be a string or a number, not 'map'
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
> Failed example:
> ? ?test_memslice_structarray(data, dtype)
> Expected:
> ? ?0
> ? ?1
> ? ?2
> ? ?3
> ? ?spam
> ? ?4
> ? ?5
> ? ?6
> ? ?7
> ? ?ham
> ? ?8
> ? ?9
> ? ?10
> ? ?11
> ? ?eggs
> Got:
> ? ?0
> ? ?1
> ? ?2
> ? ?3
> ? ?b'spam'
> ? ?4
> ? ?5
> ? ?6
> ? ?7
> ? ?b'ham'
> ? ?8
> ? ?9
> ? ?10
> ? ?11
> ? ?b'eggs'
>
>
> ======================================================================
> FAIL: test_string_invalid_dims (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_string_invalid_dims
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_string_invalid_dims
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_string_invalid_dims
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_string_invalid_dims
> Failed example:
> ? ?stringstructtest(np.array(data, dtype=dtype))
> Expected:
> ? ?Traceback (most recent call last):
> ? ? ? ...
> ? ?ValueError: Expected 2 dimensions, got 1
> Got:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_string_invalid_dims[2]>", line 1, in <module>
> ? ? ? ?stringstructtest(np.array(data, dtype=dtype))
> ? ?TypeError: expected an object with a buffer interface
>
>
> ======================================================================
> FAIL: test_struct_attributes (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_struct_attributes
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_struct_attributes
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line unknown line number, in test_struct_attributes
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_struct_attributes
> Failed example:
> ? ?test_struct_attributes()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_struct_attributes[0]>", line 1, in <module>
> ? ? ? ?test_struct_attributes()
> ? ? ?File "numpy_memoryview.pyx", line 554, in numpy_memoryview.test_struct_attributes (numpy_memoryview.c:15960)
> ? ?IndexError: invalid index
>
>
> ======================================================================
> FAIL: test_coerce_to_numpy (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_coerce_to_numpy
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_coerce_to_numpy
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_coerce_to_numpy
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_coerce_to_numpy
> Failed example:
> ? ?test_coerce_to_numpy()
> Expected:
> ? ?(97, 98, 600L, 700, 800)
> ? ?deallocating...
> ? ?(600, 700)
> ? ?deallocating...
> ? ?((100, 200), (300, 400), 500)
> ? ?deallocating...
> ? ?(97, 900)
> ? ?deallocating...
> ? ?99
> ? ?deallocating...
> ? ?111
> ? ?deallocating...
> ? ?222
> ? ?deallocating...
> ? ?333
> ? ?deallocating...
> ? ?11.1
> ? ?deallocating...
> ? ?12.2
> ? ?deallocating...
> ? ?13.3
> ? ?deallocating...
> ? ?(14.4+15.5j)
> ? ?deallocating...
> ? ?(16.6+17.7j)
> ? ?deallocating...
> ? ?(18.8+19.9j)
> ? ?deallocating...
> ? ?22
> ? ?deallocating...
> ? ?33.33
> ? ?deallocating...
> ? ?44
> ? ?deallocating...
> Got:
> ? ?(97, 98, 600, 700, 800)
> ? ?deallocating...
> ? ?(600, 700)
> ? ?deallocating...
> ? ?((100, 200), (300, 400), 500)
> ? ?deallocating...
> ? ?(97, 900)
> ? ?deallocating...
> ? ?99
> ? ?deallocating...
> ? ?111
> ? ?deallocating...
> ? ?222
> ? ?deallocating...
> ? ?333
> ? ?deallocating...
> ? ?11.1
> ? ?deallocating...
> ? ?12.2
> ? ?deallocating...
> ? ?13.3
> ? ?deallocating...
> ? ?(14.4+15.5j)
> ? ?deallocating...
> ? ?(16.6+17.7j)
> ? ?deallocating...
> ? ?(18.8+19.9j)
> ? ?deallocating...
> ? ?22
> ? ?deallocating...
> ? ?33.33
> ? ?deallocating...
> ? ?44
> ? ?deallocating...
>
>
> ======================================================================
> FAIL: test_memslice_structarray (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_memslice_structarray
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_memslice_structarray
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_memslice_structarray
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
> Failed example:
> ? ?test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_memslice_structarray[2]>", line 1, in <module>
> ? ? ? ?test_memslice_structarray([(L, map(ord, s)) for L, s in data], dtype)
> ? ? ?File "numpy_memoryview.pyx", line 464, in numpy_memoryview.test_memslice_structarray (numpy_memoryview.cpp:15496)
> ? ?TypeError: int() argument must be a string or a number, not 'map'
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_memslice_structarray
> Failed example:
> ? ?test_memslice_structarray(data, dtype)
> Expected:
> ? ?0
> ? ?1
> ? ?2
> ? ?3
> ? ?spam
> ? ?4
> ? ?5
> ? ?6
> ? ?7
> ? ?ham
> ? ?8
> ? ?9
> ? ?10
> ? ?11
> ? ?eggs
> Got:
> ? ?0
> ? ?1
> ? ?2
> ? ?3
> ? ?b'spam'
> ? ?4
> ? ?5
> ? ?6
> ? ?7
> ? ?b'ham'
> ? ?8
> ? ?9
> ? ?10
> ? ?11
> ? ?b'eggs'
>
>
> ======================================================================
> FAIL: test_string_invalid_dims (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_string_invalid_dims
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_string_invalid_dims
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_string_invalid_dims
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_string_invalid_dims
> Failed example:
> ? ?stringstructtest(np.array(data, dtype=dtype))
> Expected:
> ? ?Traceback (most recent call last):
> ? ? ? ...
> ? ?ValueError: Expected 2 dimensions, got 1
> Got:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_string_invalid_dims[2]>", line 1, in <module>
> ? ? ? ?stringstructtest(np.array(data, dtype=dtype))
> ? ?TypeError: expected an object with a buffer interface
>
>
> ======================================================================
> FAIL: test_struct_attributes (numpy_memoryview.__test__)
> Doctest: numpy_memoryview.__test__.test_struct_attributes
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for numpy_memoryview.__test__.test_struct_attributes
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line unknown line number, in test_struct_attributes
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/numpy_memoryview.cpython-32.so", line ?, in numpy_memoryview.__test__.test_struct_attributes
> Failed example:
> ? ?test_struct_attributes()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest numpy_memoryview.__test__.test_struct_attributes[0]>", line 1, in <module>
> ? ? ? ?test_struct_attributes()
> ? ? ?File "numpy_memoryview.pyx", line 554, in numpy_memoryview.test_struct_attributes (numpy_memoryview.cpp:15960)
> ? ?IndexError: invalid index
>
>
> ======================================================================
> FAIL: NestedWith (withstat)
> Doctest: withstat.NestedWith
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for withstat.NestedWith
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat.cpython-32.so", line unknown line number, in NestedWith
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat.cpython-32.so", line ?, in withstat.NestedWith
> Failed example:
> ? ?NestedWith().runTest()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest withstat.NestedWith[0]>", line 1, in <module>
> ? ? ? ?NestedWith().runTest()
> ? ? ?File "withstat.pyx", line 183, in withstat.NestedWith.runTest (withstat.c:5544)
> ? ? ?File "withstat.pyx", line 222, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:8071)
> ? ? ?File "withstat.pyx", line 223, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:7959)
> ? ? ?File "withstat.pyx", line 224, in withstat.NestedWith.testEnterReturnsTuple (withstat.c:7808)
> ? ? ?File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
> ? ? ? ?DeprecationWarning, 2)
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
> ? ? ? ?file.write(formatwarning(message, category, filename, lineno, line))
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
> ? ? ? ?line = linecache.getline(filename, lineno) if line is None else line
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 15, in getline
> ? ? ? ?lines = getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
> ? ? ? ?return self.save_linecache_getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
> ? ? ? ?return updatecache(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
> ? ? ? ?lines = fp.readlines()
> ? ? ?File "/usr/lib64/python3.2/codecs.py", line 300, in decode
> ? ? ? ?(result, consumed) = self._buffer_decode(data, self.errors, final)
> ? ?UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 40: invalid start byte
>
>
> ======================================================================
> FAIL: NestedWith (withstat)
> Doctest: withstat.NestedWith
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for withstat.NestedWith
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat.cpython-32.so", line unknown line number, in NestedWith
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat.cpython-32.so", line ?, in withstat.NestedWith
> Failed example:
> ? ?NestedWith().runTest()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest withstat.NestedWith[0]>", line 1, in <module>
> ? ? ? ?NestedWith().runTest()
> ? ? ?File "withstat.pyx", line 183, in withstat.NestedWith.runTest (withstat.cpp:5544)
> ? ? ?File "withstat.pyx", line 222, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:8071)
> ? ? ?File "withstat.pyx", line 223, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:7959)
> ? ? ?File "withstat.pyx", line 224, in withstat.NestedWith.testEnterReturnsTuple (withstat.cpp:7808)
> ? ? ?File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
> ? ? ? ?DeprecationWarning, 2)
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
> ? ? ? ?file.write(formatwarning(message, category, filename, lineno, line))
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
> ? ? ? ?line = linecache.getline(filename, lineno) if line is None else line
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 15, in getline
> ? ? ? ?lines = getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
> ? ? ? ?return self.save_linecache_getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
> ? ? ? ?return updatecache(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
> ? ? ? ?lines = fp.readlines()
> ? ? ?File "/usr/lib64/python3.2/codecs.py", line 300, in decode
> ? ? ? ?(result, consumed) = self._buffer_decode(data, self.errors, final)
> ? ?UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 24: invalid continuation byte
>
>
> ======================================================================
> FAIL: NestedWith (withstat_py)
> Doctest: withstat_py.NestedWith
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for withstat_py.NestedWith
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat_py.cpython-32.so", line unknown line number, in NestedWith
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/c/withstat_py.cpython-32.so", line ?, in withstat_py.NestedWith
> Failed example:
> ? ?NestedWith().runTest()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest withstat_py.NestedWith[0]>", line 1, in <module>
> ? ? ? ?NestedWith().runTest()
> ? ? ?File "withstat_py.py", line 250, in withstat_py.NestedWith.runTest (withstat_py.c:7232)
> ? ? ?File "withstat_py.py", line 289, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9759)
> ? ? ?File "withstat_py.py", line 290, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9647)
> ? ? ?File "withstat_py.py", line 291, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.c:9496)
> ? ? ?File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
> ? ? ? ?DeprecationWarning, 2)
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
> ? ? ? ?file.write(formatwarning(message, category, filename, lineno, line))
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
> ? ? ? ?line = linecache.getline(filename, lineno) if line is None else line
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 15, in getline
> ? ? ? ?lines = getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
> ? ? ? ?return self.save_linecache_getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
> ? ? ? ?return updatecache(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
> ? ? ? ?lines = fp.readlines()
> ? ? ?File "/usr/lib64/python3.2/codecs.py", line 300, in decode
> ? ? ? ?(result, consumed) = self._buffer_decode(data, self.errors, final)
> ? ?UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 24: invalid start byte
>
>
> ======================================================================
> FAIL: NestedWith (withstat_py)
> Doctest: withstat_py.NestedWith
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib64/python3.2/doctest.py", line 2153, in runTest
> ? ?raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for withstat_py.NestedWith
> ?File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat_py.cpython-32.so", line unknown line number, in NestedWith
>
> ----------------------------------------------------------------------
> File "/var/tmp/portage/dev-python/cython-0.16_beta0/work/Cython-0.16.beta0/tests-3.2/run/cpp/withstat_py.cpython-32.so", line ?, in withstat_py.NestedWith
> Failed example:
> ? ?NestedWith().runTest()
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
> ? ? ? ?compileflags, 1), test.globs)
> ? ? ?File "<doctest withstat_py.NestedWith[0]>", line 1, in <module>
> ? ? ? ?NestedWith().runTest()
> ? ? ?File "withstat_py.py", line 250, in withstat_py.NestedWith.runTest (withstat_py.cpp:7232)
> ? ? ?File "withstat_py.py", line 289, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9759)
> ? ? ?File "withstat_py.py", line 290, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9647)
> ? ? ?File "withstat_py.py", line 291, in withstat_py.NestedWith.testEnterReturnsTuple (withstat_py.cpp:9496)
> ? ? ?File "/usr/lib64/python3.2/unittest/case.py", line 1169, in deprecated_func
> ? ? ? ?DeprecationWarning, 2)
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 18, in showwarning
> ? ? ? ?file.write(formatwarning(message, category, filename, lineno, line))
> ? ? ?File "/usr/lib64/python3.2/warnings.py", line 25, in formatwarning
> ? ? ? ?line = linecache.getline(filename, lineno) if line is None else line
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 15, in getline
> ? ? ? ?lines = getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/doctest.py", line 1372, in __patched_linecache_getlines
> ? ? ? ?return self.save_linecache_getlines(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 41, in getlines
> ? ? ? ?return updatecache(filename, module_globals)
> ? ? ?File "/usr/lib64/python3.2/linecache.py", line 127, in updatecache
> ? ? ? ?lines = fp.readlines()
> ? ? ?File "/usr/lib64/python3.2/codecs.py", line 300, in decode
> ? ? ? ?(result, consumed) = self._buffer_decode(data, self.errors, final)
> ? ?UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 24: invalid start byte
>
>
> ----------------------------------------------------------------------
> Ran 6457 tests in 2232.009s
>
> FAILED (failures=14)
> ALL DONE
>
> --
> Arfrever Frehtes Taifersar Arahesis
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Hey Arfrever, thanks for the report, we will look into it. Are you on Gentoo?

From stefan_ml at behnel.de  Mon Mar  5 15:26:05 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 05 Mar 2012 15:26:05 +0100
Subject: [Cython] PEP 380 approved: syntax for generator delegation
In-Reply-To: <BANLkTi=zjaS43x8F_UvGk_2ecCtgwp9Y9Q@mail.gmail.com>
References: <4E083A72.3050309@behnel.de>
	<BANLkTi=zjaS43x8F_UvGk_2ecCtgwp9Y9Q@mail.gmail.com>
Message-ID: <4F54CCFD.40600@behnel.de>

Vitja Makarov, 27.06.2011 22:31:
> 2011/6/27 Stefan Behnel:
>> PEP 380 ("yield from") recently went through the approval gate.
>>
>> http://www.python.org/dev/peps/pep-0380/
>>
>> The implementation in CPython isn't rounded up yet and lots of tests are
>> missing from the test suite. But it's agreed that it'll make it into 3.3,
>> and we should try to support it as well. It's a really nice feature.
> 
> Wow! Interesting thing.
> I think that could be implemented but I think we should try to
> optimize cython generators first.

Coming back to this, the way it's now implemented in CPython is using a new
field "f_yieldfrom" in the frame that the coroutine implementation
automatically delegates to. Cython's coroutine implementation could do the
same thing, and even copy much of the original implementation, by adding a
new field to the coroutine class and updating the send(), throw() and
close() methods accordingly. The interesting changes are those in
Objects/genobject.c and Python/ceval.c in this patch:

http://bugs.python.org/file24214/f8349cbc1b26.diff

That makes it look like most of the work is done for us already.

Stefan

From d.s.seljebotn at astro.uio.no  Mon Mar  5 20:01:49 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 05 Mar 2012 11:01:49 -0800
Subject: [Cython] NumFOCUS
Message-ID: <4F550D9D.1080300@astro.uio.no>

To whoever hasn't heard the news already, there's now a NumPy Foundation 
for Open Code for Usable Science:

http://numfocus.org/

Cython is on the "looking for representatives" list (my English fails me 
as to what exactly that means though).

They are seeking US tax-exempt status. So:

  a) Cython users who wants to sponsor development may be able to 
contribute through this organizatin

  b) It's a place Cython devs should keep in mind as a possible source 
of funding for good projects or workshop opportunities (if money can 
help -- which is often not the case, but sometimes)

Dag

From wstein at gmail.com  Mon Mar  5 21:22:52 2012
From: wstein at gmail.com (William Stein)
Date: Mon, 5 Mar 2012 12:22:52 -0800
Subject: [Cython] NumFOCUS
In-Reply-To: <4F550D9D.1080300@astro.uio.no>
References: <4F550D9D.1080300@astro.uio.no>
Message-ID: <CACLE5GBX6MStF81P_L-d8sOYM5sp_-z-8jun65JvEf49ZRUfbQ@mail.gmail.com>

On Mar 5, 2012 11:02 AM, "Dag Sverre Seljebotn" <d.s.seljebotn at astro.uio.no>
wrote:
>
> To whoever hasn't heard the news already, there's now a NumPy Foundation
for Open Code for Usable Science:
>
> http://numfocus.org/
>
> Cython is on the "looking for representatives" list (my English fails me
as to what exactly that means though).
>
> They are seeking US tax-exempt status. So:
>
>  a) Cython users who wants to sponsor development may be able to
contribute through this organizatin
>
>  b) It's a place Cython devs should keep in mind as a possible source of
funding for good projects or workshop opportunities (if money can help --
which is often not the case, but sometimes)
>

On this topic, keep me in mind too, as I have an NSF grant right now that
can be used for such things.

> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120305/a00e8105/attachment.html>

From stefan_ml at behnel.de  Tue Mar  6 11:03:40 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 06 Mar 2012 11:03:40 +0100
Subject: [Cython] PEP 380 approved: syntax for generator delegation
In-Reply-To: <4F54CCFD.40600@behnel.de>
References: <4E083A72.3050309@behnel.de>
	<BANLkTi=zjaS43x8F_UvGk_2ecCtgwp9Y9Q@mail.gmail.com>
	<4F54CCFD.40600@behnel.de>
Message-ID: <4F55E0FC.6000007@behnel.de>

Stefan Behnel, 05.03.2012 15:26:
> Vitja Makarov, 27.06.2011 22:31:
>> 2011/6/27 Stefan Behnel:
>>> PEP 380 ("yield from") recently went through the approval gate.
>>>
>>> http://www.python.org/dev/peps/pep-0380/
>>>
>>> The implementation in CPython isn't rounded up yet and lots of tests are
>>> missing from the test suite. But it's agreed that it'll make it into 3.3,
>>> and we should try to support it as well. It's a really nice feature.
>>
>> Wow! Interesting thing.
>> I think that could be implemented but I think we should try to
>> optimize cython generators first.
> 
> Coming back to this, the way it's now implemented in CPython is using a new
> field "f_yieldfrom" in the frame that the coroutine implementation
> automatically delegates to. Cython's coroutine implementation could do the
> same thing, and even copy much of the original implementation, by adding a
> new field to the coroutine class and updating the send(), throw() and
> close() methods accordingly. The interesting changes are those in
> Objects/genobject.c and Python/ceval.c in this patch:
> 
> http://bugs.python.org/file24214/f8349cbc1b26.diff
> 
> That makes it look like most of the work is done for us already.

Just to report some progress here: I got the implementation almost done and
it even seems to be quite portable (StopIteration propagates a return value
now, but that also works in older Pythons so far). There are a couple of
bugs that I'll have to look into, but I should have something ready in a
couple of days (or nights...). The CPython tests proved to be quite
helpful, but I'll need the usual Cython specific tests as well.

If anyone wants to write up some code that you think should work with
"yield from" in Cython, I'll happily take it. One additional feature that
would be really cool is support for delegating to C arrays and C++
iterators, so that you could wrap C++ iterators with simple values in
straight code like this:

  def gen():
     yield from some_cpp_iterable

Anyway, that's pretty far off for now as it requires specialised code for
the different C/C++ source types.

Stefan

From stefan_ml at behnel.de  Thu Mar  8 14:27:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 08 Mar 2012 14:27:54 +0100
Subject: [Cython] GSoC 2012
Message-ID: <4F58B3DA.5070602@behnel.de>

Hi,

I noticed that people start rushing for the next season on Python's GSoC
mailing lists. Do we have any interested developers here, or general ideas
about suitable topics? I would expect that we'll do as in the last years
and participate under Python's umbrella.

Please reply on the cython-devel mailing list.

Stefan

From dalcinl at gmail.com  Fri Mar  9 14:40:52 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 Mar 2012 16:40:52 +0300
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
	inside OpenMP blocks
In-Reply-To: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
Message-ID: <CAEcYPwBJMDNn1MUimRDvVQiQyOEYsyiE9TimTjZ9xUQhtXB=kw@mail.gmail.com>

On 9 March 2012 16:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>
> PS: This is the only serious failure I get on OS X Lion. The embed
> test is not working, but it is a Makefile issue.
>

UPDATE: The embed test do work with system Python 2.7. The failure is
with EPD Python 7.2, it is caused by a bad definition of LINKFORSHARED
in its installed config/Makefile.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Fri Mar  9 14:22:35 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 Mar 2012 16:22:35 +0300
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized inside
	OpenMP blocks
Message-ID: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>

I'm basically experiencing the issues here:
http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html

Can you imagine any way to workaround it?

PS: This is the only serious failure I get on OS X Lion. The embed
test is not working, but it is a Makefile issue.

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Sat Mar 10 01:41:34 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 10 Mar 2012 01:41:34 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
Message-ID: <CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>

On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> I'm basically experiencing the issues here:
> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>
> Can you imagine any way to workaround it?

What a lovely C compiler bug... Did you file a bug report with gcc or
xcode? What version of gcc does Lion ship with?

I suppose the macro would have to be disabled for OS X Lion inside
parallel sections (there seems to be __APPLE__ and __OSX__, I don't
know about detecting the OS X version), that's easy to do (undef and
redefine to no-ops before parallel section and redefine it again after
the section). I'll try fixing it during the sprints next week.

> PS: This is the only serious failure I get on OS X Lion. The embed
> test is not working, but it is a Makefile issue.
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From dalcinl at gmail.com  Sat Mar 10 10:51:51 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 10 Mar 2012 12:51:51 +0300
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
Message-ID: <CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>

On 10 March 2012 03:41, mark florisson <markflorisson88 at gmail.com> wrote:
> On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> I'm basically experiencing the issues here:
>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>
>> Can you imagine any way to workaround it?
>
> What a lovely C compiler bug... Did you file a bug report with gcc or
> xcode? What version of gcc does Lion ship with?
>

I'm using a Mac just by accident, no idea (nor motivation) to report
bugs to Apple.

$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
build 5658) (LLVM build 2336.9.00)

Could that be the the bug is actually in the LLVM backend?

> I suppose the macro would have to be disabled for OS X Lion inside
> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
> know about detecting the OS X version), that's easy to do (undef and
> redefine to no-ops before parallel section and redefine it again after
> the section). I'll try fixing it during the sprints next week.
>

Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
OpenMP blocks would be nicer than defining/undefining?

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Sat Mar 10 14:00:33 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 10 Mar 2012 14:00:33 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
Message-ID: <4F5B5071.5040105@behnel.de>

Lisandro Dalcin, 10.03.2012 10:51:
> On 10 March 2012 03:41, mark florisson wrote:
>> On 9 March 2012 14:22, Lisandro Dalcin wrote:
>>> I'm basically experiencing the issues here:
>>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>>
>>> Can you imagine any way to workaround it?
>>
>> What a lovely C compiler bug... Did you file a bug report with gcc or
>> xcode? What version of gcc does Lion ship with?
>>
> 
> I'm using a Mac just by accident, no idea (nor motivation) to report
> bugs to Apple.
> 
> $ gcc --version
> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
> build 5658) (LLVM build 2336.9.00)
> 
> Could that be the the bug is actually in the LLVM backend?
> 
>> I suppose the macro would have to be disabled for OS X Lion inside
>> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
>> know about detecting the OS X version), that's easy to do (undef and
>> redefine to no-ops before parallel section and redefine it again after
>> the section). I'll try fixing it during the sprints next week.
> 
> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
> OpenMP blocks would be nicer than defining/undefining?

Could that be coded into the macro or would it require to change the
generated code? But at least it sounds like it would not impact code in
functions that are being called from within the OpenMP blocks, would it?
Just the code straight inside the block. A work-around could still have a
substantial impact if it requires changes to the generated code.

Stefan

From markflorisson88 at gmail.com  Sun Mar 11 07:44:14 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 11 Mar 2012 07:44:14 +0100
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <4F58B3DA.5070602@behnel.de>
References: <4F58B3DA.5070602@behnel.de>
Message-ID: <CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>

On 8 March 2012 14:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I noticed that people start rushing for the next season on Python's GSoC
> mailing lists. Do we have any interested developers here, or general ideas
> about suitable topics? I would expect that we'll do as in the last years
> and participate under Python's umbrella.

I will likely be submitting a proposal for the OpenCL support CEP. As
for other proposals, anyone can come up with something themselves or
choose a suitable CEP. Some ideas:

- fused cdef classes (probably not as an entire gsoc project)
- profile guided optimizations (using python's profilers and/or a
custom profiler that collects data such as types etc, which can be
used to specialize variables inside loops (or entire functions) with a
fallback to normal mode in case the type changes)
- cython library to contain common functionality (although that might
be a bit boring and rather involved)
- better type inference, that would be enabled by default and again
handle thing like reassignments of variables and fallbacks to the
default object type. With entry caching Cython could build a database
of types ((extension) classes, functions, variables) used in the
modules and functions that are compiled (also def functions), and
infer the types used and specialize on those. Maybe a switch should be
added to cython to handle circular dependencies, or maybe with the
distutils preprocessing it can run all the type inference first and
keep track of unresolved entries, and try to fill those in after
building the database. For bonus points the user can be allowed to
write plugins to aid the process.
- llvm based JIT :), i.e. have Cython instrument the generated code to
record information and use that to create specializations at runtime
(probably far out for a gsoc)

I'd be willing to (co)mentor if wanted and possible within the
constraints of the gsoc program.

> Please reply on the cython-devel mailing list.
>
> Stefan

From markflorisson88 at gmail.com  Sun Mar 11 07:46:12 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 11 Mar 2012 07:46:12 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <4F5B5071.5040105@behnel.de>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
	<4F5B5071.5040105@behnel.de>
Message-ID: <CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>

On 10 March 2012 14:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 10.03.2012 10:51:
>> On 10 March 2012 03:41, mark florisson wrote:
>>> On 9 March 2012 14:22, Lisandro Dalcin wrote:
>>>> I'm basically experiencing the issues here:
>>>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>>>
>>>> Can you imagine any way to workaround it?
>>>
>>> What a lovely C compiler bug... Did you file a bug report with gcc or
>>> xcode? What version of gcc does Lion ship with?
>>>
>>
>> I'm using a Mac just by accident, no idea (nor motivation) to report
>> bugs to Apple.
>>
>> $ gcc --version
>> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
>> build 5658) (LLVM build 2336.9.00)
>>
>> Could that be the the bug is actually in the LLVM backend?
>>
>>> I suppose the macro would have to be disabled for OS X Lion inside
>>> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
>>> know about detecting the OS X version), that's easy to do (undef and
>>> redefine to no-ops before parallel section and redefine it again after
>>> the section). I'll try fixing it during the sprints next week.
>>
>> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
>> OpenMP blocks would be nicer than defining/undefining?
>
> Could that be coded into the macro or would it require to change the
> generated code? But at least it sounds like it would not impact code in
> functions that are being called from within the OpenMP blocks, would it?
> Just the code straight inside the block. A work-around could still have a
> substantial impact if it requires changes to the generated code.

Yeah, that's why I suggested the undef/re-def approach around OpenMP
blocks. It's some code bloat, but only for the C preprocessor, so it
should be fine.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Sun Mar 11 07:48:50 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 10 Mar 2012 22:48:50 -0800
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
Message-ID: <09222f23-af19-4c47-add2-55ff2fa88e05@email.android.com>

I'm pretty sure you can't be a student and mentor at the same time... Something to keep in mind...
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

mark florisson <markflorisson88 at gmail.com> wrote:

On 8 March 2012 14:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I noticed that people start rushing for the next season on Python's GSoC
> mailing lists. Do we have any interested developers here, or general ideas
> about suitable topics? I would expect that we'll do as in the last years
> and participate under Python's umbrella.

I will likely be submitting a proposal for the OpenCL support CEP. As
for other proposals, anyone can come up with something themselves or
choose a suitable CEP. Some ideas:

- fused cdef classes (probably not as an entire gsoc project)
- profile guided optimizations (using python's profilers and/or a
custom profiler that collects data such as types etc, which can be
used to specialize variables inside loops (or entire functions) with a
fallback to normal mode in case the type changes)
- cython library to contain common functionality (although that might
be a bit boring and rather involved)
- better type inference, that would be enabled by default and again
handle thing like reassignments of variables and fallbacks to the
default object type. With entry caching Cython could build a database
of types ((extension) classes, functions, variables) used in the
modules and functions that are compiled (also def functions), and
infer the types used and specialize on those. Maybe a switch should be
added to cython to handle circular dependencies, or maybe with the
distutils preprocessing it can run all the type inference first and
keep track of unresolved entries, and try to fill those in after
building the database. For bonus points the user can be allowed to
write plugins to aid the process.
- llvm based JIT :), i.e. have Cython instrument the generated code to
record information and use that to create specializations at runtime
(probably far out for a gsoc)

I'd be willing to (co)mentor if wanted and possible within the
constraints of the gsoc program.

> Please reply on the cython-devel mailing list.
>
> Stefan
_____________________________________________

cython-devel mailing list
cython-devel at python.org
http://mail.python.org/mailman/listinfo/cython-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120310/734b1ef9/attachment.html>

From stefan_ml at behnel.de  Sun Mar 11 09:08:02 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 Mar 2012 09:08:02 +0100
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
Message-ID: <4F5C5D62.20003@behnel.de>

mark florisson, 11.03.2012 07:44:
> - better type inference, that would be enabled by default and again
> handle thing like reassignments of variables and fallbacks to the
> default object type. With entry caching Cython could build a database
> of types ((extension) classes, functions, variables) used in the
> modules and functions that are compiled (also def functions), and
> infer the types used and specialize on those. Maybe a switch should be
> added to cython to handle circular dependencies, or maybe with the
> distutils preprocessing it can run all the type inference first and
> keep track of unresolved entries, and try to fill those in after
> building the database. For bonus points the user can be allowed to
> write plugins to aid the process.

That would be my favourite. We definitely need control flow driven type
inference, local type specialisation, variable renaming, etc. Maybe even
whole program (or at least module) analysis, like ShedSkin and PyPy do for
their restricted Python dialects. Any serious step towards that goal would
be a good outcome of a GSoC.

There's also better support for PyPy through its cpyext C-API layer, but
that currently involves much more work on PyPy than on Cython, including a
lot of performance optimisation on their side. And there doesn't seem to be
much interest in the PyPy project for doing this.

Stefan

From mathieu at mblondel.org  Sun Mar 11 09:19:52 2012
From: mathieu at mblondel.org (Mathieu Blondel)
Date: Sun, 11 Mar 2012 17:19:52 +0900
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <4F58B3DA.5070602@behnel.de>
References: <4F58B3DA.5070602@behnel.de>
Message-ID: <CAOKSrLziUiPBqk62ouCt_nJ1M89oOhN2xfMn9dSAyrtYe5QPTw@mail.gmail.com>

On Thu, Mar 8, 2012 at 10:27 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> I noticed that people start rushing for the next season on Python's GSoC
> mailing lists. Do we have any interested developers here, or general ideas
> about suitable topics? I would expect that we'll do as in the last years
> and participate under Python's umbrella.

Function overloading would be nice:

http://wiki.cython.org/enhancements/overloading

Mathieu

From dalcinl at gmail.com  Sun Mar 11 09:26:08 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 11 Mar 2012 11:26:08 +0300
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
	<4F5B5071.5040105@behnel.de>
	<CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>
Message-ID: <CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>

On 11 March 2012 09:46, mark florisson <markflorisson88 at gmail.com> wrote:
> On 10 March 2012 14:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Lisandro Dalcin, 10.03.2012 10:51:
>>> On 10 March 2012 03:41, mark florisson wrote:
>>>> On 9 March 2012 14:22, Lisandro Dalcin wrote:
>>>>> I'm basically experiencing the issues here:
>>>>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>>>>
>>>>> Can you imagine any way to workaround it?
>>>>
>>>> What a lovely C compiler bug... Did you file a bug report with gcc or
>>>> xcode? What version of gcc does Lion ship with?
>>>>
>>>
>>> I'm using a Mac just by accident, no idea (nor motivation) to report
>>> bugs to Apple.
>>>
>>> $ gcc --version
>>> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
>>> build 5658) (LLVM build 2336.9.00)
>>>
>>> Could that be the the bug is actually in the LLVM backend?
>>>
>>>> I suppose the macro would have to be disabled for OS X Lion inside
>>>> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
>>>> know about detecting the OS X version), that's easy to do (undef and
>>>> redefine to no-ops before parallel section and redefine it again after
>>>> the section). I'll try fixing it during the sprints next week.
>>>
>>> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
>>> OpenMP blocks would be nicer than defining/undefining?
>>
>> Could that be coded into the macro or would it require to change the
>> generated code? But at least it sounds like it would not impact code in
>> functions that are being called from within the OpenMP blocks, would it?
>> Just the code straight inside the block. A work-around could still have a
>> substantial impact if it requires changes to the generated code.
>
> Yeah, that's why I suggested the undef/re-def approach around OpenMP
> blocks. It's some code bloat, but only for the C preprocessor, so it
> should be fine.
>

I still feel bad about this. What about just disabling branch
prediction if OpenMP is ever used? Or perhaps just protect the
definitions of likely/unlikely with some guard, such as users can
disable them using a -D definition?


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Sun Mar 11 09:27:30 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 Mar 2012 09:27:30 +0100
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CAOKSrLziUiPBqk62ouCt_nJ1M89oOhN2xfMn9dSAyrtYe5QPTw@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CAOKSrLziUiPBqk62ouCt_nJ1M89oOhN2xfMn9dSAyrtYe5QPTw@mail.gmail.com>
Message-ID: <4F5C61F2.6010705@behnel.de>

Mathieu Blondel, 11.03.2012 09:19:
> On Thu, Mar 8, 2012 at 10:27 PM, Stefan Behnel wrote:
>> I noticed that people start rushing for the next season on Python's GSoC
>> mailing lists. Do we have any interested developers here, or general ideas
>> about suitable topics? I would expect that we'll do as in the last years
>> and participate under Python's umbrella.
> 
> Function overloading would be nice:
> 
> http://wiki.cython.org/enhancements/overloading

Don't we have (most of) that already? At least the complete infrastructure
should be there now, given that we have C++ support and fused types.
Doesn't sound non-trivial enough for a GSoC to me.

Stefan

From stefan_ml at behnel.de  Sun Mar 11 09:35:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 Mar 2012 09:35:14 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
	<4F5B5071.5040105@behnel.de>
	<CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>
	<CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>
Message-ID: <4F5C63C2.7030107@behnel.de>

Lisandro Dalcin, 11.03.2012 09:26:
> On 11 March 2012 09:46, mark florisson wrote:
>> On 10 March 2012 14:00, Stefan Behnel wrote:
>>> Lisandro Dalcin, 10.03.2012 10:51:
>>>> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
>>>> OpenMP blocks would be nicer than defining/undefining?
>>>
>>> Could that be coded into the macro or would it require to change the
>>> generated code? But at least it sounds like it would not impact code in
>>> functions that are being called from within the OpenMP blocks, would it?
>>> Just the code straight inside the block. A work-around could still have a
>>> substantial impact if it requires changes to the generated code.
>>
>> Yeah, that's why I suggested the undef/re-def approach around OpenMP
>> blocks. It's some code bloat, but only for the C preprocessor, so it
>> should be fine.
> 
> I still feel bad about this. What about just disabling branch
> prediction if OpenMP is ever used? Or perhaps just protect the
> definitions of likely/unlikely with some guard, such as users can
> disable them using a -D definition?

There's a Cython compiler switch for it already. The problem is that it's
not used everywhere where branch prediction hints are being generated.

Would be worth fixing IMHO. There's already an unlikely() method in the
code generator, a likely() method should be added and both should be used
appropriately. That would allow us to a) switch the hints on and off
properly in the compiler and b) generate different code for the hints based
on the code environment.

I'd still leave utility code out of this. So far, it sounds like it doesn't
matter if we have branch hints in utility code or not, so specialising all
utility code that could make use of branch hints would just be a waste of time.

Stefan

From vitja.makarov at gmail.com  Sun Mar 11 09:51:02 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 11 Mar 2012 12:51:02 +0400
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <4F5C5D62.20003@behnel.de>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
	<4F5C5D62.20003@behnel.de>
Message-ID: <CAKGHGPRhsC7vMe55JoP0EvbWyg1c__t4Frzepes2kqDE5EE6xQ@mail.gmail.com>

2012/3/11 Stefan Behnel <stefan_ml at behnel.de>:
> mark florisson, 11.03.2012 07:44:
>> - better type inference, that would be enabled by default and again
>> handle thing like reassignments of variables and fallbacks to the
>> default object type. With entry caching Cython could build a database
>> of types ((extension) classes, functions, variables) used in the
>> modules and functions that are compiled (also def functions), and
>> infer the types used and specialize on those. Maybe a switch should be
>> added to cython to handle circular dependencies, or maybe with the
>> distutils preprocessing it can run all the type inference first and
>> keep track of unresolved entries, and try to fill those in after
>> building the database. For bonus points the user can be allowed to
>> write plugins to aid the process.
>
> That would be my favourite. We definitely need control flow driven type
> inference, local type specialisation, variable renaming, etc. Maybe even
> whole program (or at least module) analysis, like ShedSkin and PyPy do for
> their restricted Python dialects. Any serious step towards that goal would
> be a good outcome of a GSoC.
>

I think we should be careful here and try to avoid making Cython code
more complicated.

> There's also better support for PyPy through its cpyext C-API layer, but
> that currently involves much more work on PyPy than on Cython, including a
> lot of performance optimisation on their side. And there doesn't seem to be
> much interest in the PyPy project for doing this.
>

I'm intrested in function/method call inlining based on CF analysis
and on generic cyfunction's signature.
I'll do some benchmarks to see how much do we get from this optimization.

-- 
vitja.

From markflorisson88 at gmail.com  Sun Mar 11 17:09:32 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 11 Mar 2012 17:09:32 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
	<4F5B5071.5040105@behnel.de>
	<CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>
	<CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>
Message-ID: <CANg26EX8LEsNjAf-+6rGxFMR0RYWhgF5NtboQeO0kdtsv5mhbg@mail.gmail.com>

On 11 March 2012 09:26, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 11 March 2012 09:46, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 10 March 2012 14:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Lisandro Dalcin, 10.03.2012 10:51:
>>>> On 10 March 2012 03:41, mark florisson wrote:
>>>>> On 9 March 2012 14:22, Lisandro Dalcin wrote:
>>>>>> I'm basically experiencing the issues here:
>>>>>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>>>>>
>>>>>> Can you imagine any way to workaround it?
>>>>>
>>>>> What a lovely C compiler bug... Did you file a bug report with gcc or
>>>>> xcode? What version of gcc does Lion ship with?
>>>>>
>>>>
>>>> I'm using a Mac just by accident, no idea (nor motivation) to report
>>>> bugs to Apple.
>>>>
>>>> $ gcc --version
>>>> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
>>>> build 5658) (LLVM build 2336.9.00)
>>>>
>>>> Could that be the the bug is actually in the LLVM backend?
>>>>
>>>>> I suppose the macro would have to be disabled for OS X Lion inside
>>>>> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
>>>>> know about detecting the OS X version), that's easy to do (undef and
>>>>> redefine to no-ops before parallel section and redefine it again after
>>>>> the section). I'll try fixing it during the sprints next week.
>>>>
>>>> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
>>>> OpenMP blocks would be nicer than defining/undefining?
>>>
>>> Could that be coded into the macro or would it require to change the
>>> generated code? But at least it sounds like it would not impact code in
>>> functions that are being called from within the OpenMP blocks, would it?
>>> Just the code straight inside the block. A work-around could still have a
>>> substantial impact if it requires changes to the generated code.
>>
>> Yeah, that's why I suggested the undef/re-def approach around OpenMP
>> blocks. It's some code bloat, but only for the C preprocessor, so it
>> should be fine.
>>
>
> I still feel bad about this. What about just disabling branch
> prediction if OpenMP is ever used? Or perhaps just protect the
> definitions of likely/unlikely with some guard, such as users can
> disable them using a -D definition?
>

Why? That would disable all compiler optimizations in all code that
could be gotten through the hints, due to a stupid C compiler bug when
using cython.parallel. It would be better to do an easy fix, a nasty
workaround for a nasty bug, but it shouldn't slow down other code. The
only problem with it is a couple of extra bytes in your .c file, and
the fact that it makes us feel less fuzzy inside because we realize
our world isn't what we want it to be.

> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Wed Mar 14 06:58:26 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 14 Mar 2012 06:58:26 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
Message-ID: <CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>

On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> I'm basically experiencing the issues here:
> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>
> Can you imagine any way to workaround it?
>
> PS: This is the only serious failure I get on OS X Lion. The embed
> test is not working, but it is a Makefile issue.

Could you give this fix a try?
https://github.com/markflorisson88/cython/commit/2bffde15edc66c7416716051959e3b0cf1d6b41b

> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Wed Mar 14 10:09:56 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 14 Mar 2012 10:09:56 +0100
Subject: [Cython] Jenkins down
Message-ID: <4F606064.3090206@behnel.de>

Hi,

the sage.math server is currently suffering from a power breakdown at the
university campus, so Jenkins is unreachable. I'll restart Jenkins as soon
as the server is accessible via ssh again.

Stefan

From wstein at gmail.com  Wed Mar 14 16:40:40 2012
From: wstein at gmail.com (William Stein)
Date: Wed, 14 Mar 2012 08:40:40 -0700
Subject: [Cython] Jenkins down
In-Reply-To: <4F606064.3090206@behnel.de>
References: <4F606064.3090206@behnel.de>
Message-ID: <CACLE5GC59GbUZ7eyw3g+CgMt_WMzO0PMf-nFoj=NzGXazK+e5w@mail.gmail.com>

On Wed, Mar 14, 2012 at 2:09 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> the sage.math server is currently suffering from a power breakdown at the
> university campus, so Jenkins is unreachable. I'll restart Jenkins as soon
> as the server is accessible via ssh again.

Thanks. This should be the case about 24 hours from right now, once I
am back in Seattle.

William

>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From dalcinl at gmail.com  Thu Mar 15 13:21:55 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 15 Mar 2012 15:21:55 +0300
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
Message-ID: <CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>

On 14 March 2012 08:58, mark florisson <markflorisson88 at gmail.com> wrote:
> On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>
> Could you give this fix a try?
> https://github.com/markflorisson88/cython/commit/2bffde15edc66c7416716051959e3b0cf1d6b41b
>

I'm still getting the linking error. Your fix does not resolve the issue :-(


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Fri Mar 16 04:00:34 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 16 Mar 2012 04:00:34 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
	<CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
Message-ID: <CANg26EX03=Ypc=5E9yKNgmUFt2T2RYf_ynprztA0zMMthz0nPw@mail.gmail.com>

On 15 March 2012 13:21, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 14 March 2012 08:58, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> Could you give this fix a try?
>> https://github.com/markflorisson88/cython/commit/2bffde15edc66c7416716051959e3b0cf1d6b41b
>>
>
> I'm still getting the linking error. Your fix does not resolve the issue :-(
>

Hm, what does this give you?

$ cpp
#if defined(__APPLE__) || defined(__OSX__)
    FOO
#endif
^D

> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Fri Mar 16 04:02:02 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 16 Mar 2012 04:02:02 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EX03=Ypc=5E9yKNgmUFt2T2RYf_ynprztA0zMMthz0nPw@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
	<CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
	<CANg26EX03=Ypc=5E9yKNgmUFt2T2RYf_ynprztA0zMMthz0nPw@mail.gmail.com>
Message-ID: <CANg26EVZhaO8S59Z8LJ2BA39qPa4amVOjQhjmm-KA4Cb1N3Wqw@mail.gmail.com>

On 16 March 2012 04:00, mark florisson <markflorisson88 at gmail.com> wrote:
> On 15 March 2012 13:21, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> On 14 March 2012 08:58, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>
>>> Could you give this fix a try?
>>> https://github.com/markflorisson88/cython/commit/2bffde15edc66c7416716051959e3b0cf1d6b41b
>>>
>>
>> I'm still getting the linking error. Your fix does not resolve the issue :-(
>>
>
> Hm, what does this give you?
>
> $ cpp
> #if defined(__APPLE__) || defined(__OSX__)
> ? ?FOO
> #endif
> ^D

Ah, maybe I should pull it out of nested prange constructs.

>> --
>> Lisandro Dalcin
>> ---------------
>> CIMEC (INTEC/CONICET-UNL)
>> Predio CONICET-Santa Fe
>> Colectora RN 168 Km 472, Paraje El Pozo
>> 3000 Santa Fe, Argentina
>> Tel: +54-342-4511594 (ext 1011)
>> Tel/Fax: +54-342-4511169
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sun Mar 18 11:58:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 18 Mar 2012 11:58:26 +0100
Subject: [Cython] some pull requests for 0.16?
Message-ID: <4F65BFD2.8010706@behnel.de>

Hi,

I put up two new pull requests on github:

Implementation of PEP 380 (yield from):
https://github.com/cython/cython/pull/96

Rewrite of dict iteration:
https://github.com/cython/cython/pull/95

Given that the release of 0.16 has currently slowed down a bit, and given
that these are really nice features, could someone (and especially Mark, as
the responsible release manager) take a look at them and give an opinion
regarding their suitability for 0.16?

Personally, I think they are ready, but they'd certainly require another
beta release for testing.

You may notice that I also added the PyPy related preprocessor defines
"CYTHON_COMPILING_IN_CPYTHON" and "CYTHON_COMPILING_IN_PYPY" to the current
master and used them in a couple of places to reduce Cython's dependency on
CPython details. That doesn't mean we already support PyPy (there are still
open issues, mostly on their side). I only pushed the changes that were
non-intrusive and helped me in preparing the two branches above in a clean
way. I have some more changes in my PyPy branch:

https://github.com/scoder/cython/tree/pypy

Stefan

From wardefar at iro.umontreal.ca  Tue Mar 20 06:44:31 2012
From: wardefar at iro.umontreal.ca (David Warde-Farley)
Date: Tue, 20 Mar 2012 01:44:31 -0400
Subject: [Cython] (bug?) "cdef public" and Python attribute access
Message-ID: <6A7A31B7-CBF9-466F-AE7C-A6CE780F5200@iro.umontreal.ca>

All of this refers to Cython 0.15.1:

I have a "cdef public void" function which takes several ndarrays, that segfaults when I call it from other C code. Inside I use the ".size" attribute, triggering a PyObject_GetAttr, which segfaults. Running in gdb and looking at the generated C code, I see it is because __pyx_n_s__size is NULL, presumably owing to __Pyx_InitStrings() never being called.

Am I to presume this is intended behaviour? i.e. code that generates Python C API (or more specifically GetAttr) calls simply doesn't work/isn't intended to work from "public" functions called from C code? I don't see mention of it in the documentation, so it's kind of a nasty surprise :)  (The "api" keyword mentions the import_modulename() business but no signature for that is generated in my header file.)

In this case I can work around it by multiplying the elements of array.shape.

Thanks,

David

From stefan_ml at behnel.de  Tue Mar 20 09:21:57 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 20 Mar 2012 09:21:57 +0100
Subject: [Cython] (bug?) "cdef public" and Python attribute access
In-Reply-To: <6A7A31B7-CBF9-466F-AE7C-A6CE780F5200@iro.umontreal.ca>
References: <6A7A31B7-CBF9-466F-AE7C-A6CE780F5200@iro.umontreal.ca>
Message-ID: <4F683E25.5080809@behnel.de>

Hi,

note that the cython-users mailing list would be a more appropriate place
to discuss this. Please reply over there.

David Warde-Farley, 20.03.2012 06:44:
> All of this refers to Cython 0.15.1:
> 
> I have a "cdef public void" function which takes several ndarrays, that
> segfaults when I call it from other C code. Inside I use the ".size"
> attribute, triggering a PyObject_GetAttr, which segfaults. Running in
> gdb and looking at the generated C code, I see it is because
> __pyx_n_s__size is NULL, presumably owing to __Pyx_InitStrings() never
> being called.

I take it you are embedding Cython code in a C program?

Note that the module must be imported in order to work properly. See here:

http://wiki.cython.org/EmbeddingCython


> Am I to presume this is intended behaviour?

Yes.


> i.e. code that generates
> Python C API (or more specifically GetAttr) calls simply doesn't
> work/isn't intended to work from "public" functions called from C code?

It does work, but it requires two things: a properly imported module and
(unless declared "nogil") the GIL.


> I don't see mention of it in the documentation, so it's kind of a nasty
> surprise :)

Yes, this can use some better documentation.


> (The "api" keyword mentions the import_modulename()
> business but no signature for that is generated in my header file.)

It has been discussed before that the concepts "api" and "public" are not
very clearly separated and should eventually be merged into one. Hasn't
been done yet.

Basically, the concepts are somewhat more focussed towards being used from
inside of CPython than towards embedding CPython in C code. Both work, but
embedding requires a little more manual setup.

Stefan

From markflorisson88 at gmail.com  Tue Mar 20 17:40:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 20 Mar 2012 17:40:28 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F65BFD2.8010706@behnel.de>
References: <4F65BFD2.8010706@behnel.de>
Message-ID: <CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>

On 18 March 2012 11:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I put up two new pull requests on github:
>
> Implementation of PEP 380 (yield from):
> https://github.com/cython/cython/pull/96
>
> Rewrite of dict iteration:
> https://github.com/cython/cython/pull/95
>
> Given that the release of 0.16 has currently slowed down a bit, and given
> that these are really nice features, could someone (and especially Mark, as
> the responsible release manager) take a look at them and give an opinion
> regarding their suitability for 0.16?

Sorry about the slowdown, my BT internet connection at home got closed
off, so working on a release is rather hard, hopefully that will be
fixed within the next few days. Anyway, during the sprints Dag worked
on the numpy attributes accesses to generate non-deprecated numpy
code, I'm not sure how far along that is and if we can get that in as
well, or have it wait until 0.16.1.
I worked on enhancing fused types runtime dispatching (e.g. select the
right specialization at runtime from the argument types), especially
on matching buffers. It's also a bit of a rewrite as the complexity
grew quite a bit and ndarray and fused types were broken in subtle
ways (I would do the pull request now if I hadn't left the code at
home :).

Finally, there is a serious bug in the reversed(range()) optimization
that Mike Graham discovered when working on the range() optimization
with a runtime step, which is basically wrong for any step that is not
1 or -1 (it simply swaps the bounds with an offset by 1, but it
actually has to figure out what the actual last value was in the range
and use that as the start instead). I personally consider this a
blocker, so I'll try contacting Mike and see if he is still
(interested in) working on that.

As for the two pull requests, both are quite large, but the dict
iteration rewrite is more like an enhancement whereas the 'yield from'
is really a new (somewhat big) feature. I would personally prefer to
merge only enhancements and bug fixes, but not new features. If you
feel confident enough that the dict iteration pull request is ready to
be merged, then please do so. We can do an 0.16.1 release with the
'yield from', the 'lnotab' line number mapping from code objects to
Cython source and probably the boundscheck and wraparound
optimizations I started working on (and other things if and when they
come up). We can release 0.16.1 somewhat shortly after releasing 0.16.
I think a shorter release cycle with less changes would be beneficial
and give us swifter feedback about bugs, missing features, wanted
features, etc.

So I propose that when we merged dict iteration, fused types
enhancements and the range bug fix, we do another beta release. We (I)
also need to look into the test failures on Gentoo.

> Personally, I think they are ready, but they'd certainly require another
> beta release for testing.
>
> You may notice that I also added the PyPy related preprocessor defines
> "CYTHON_COMPILING_IN_CPYTHON" and "CYTHON_COMPILING_IN_PYPY" to the current
> master and used them in a couple of places to reduce Cython's dependency on
> CPython details. That doesn't mean we already support PyPy (there are still
> open issues, mostly on their side). I only pushed the changes that were
> non-intrusive and helped me in preparing the two branches above in a clean
> way. I have some more changes in my PyPy branch:
>
> https://github.com/scoder/cython/tree/pypy
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Mar 20 18:51:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 20 Mar 2012 18:51:46 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
Message-ID: <4F68C3B2.7020206@behnel.de>

mark florisson, 20.03.2012 17:40:
> On 18 March 2012 11:58, Stefan Behnel wrote:
>> I put up two new pull requests on github:
>>
>> Implementation of PEP 380 (yield from):
>> https://github.com/cython/cython/pull/96
>>
>> Rewrite of dict iteration:
>> https://github.com/cython/cython/pull/95
>>
>> Given that the release of 0.16 has currently slowed down a bit, and given
>> that these are really nice features, could someone (and especially Mark, as
>> the responsible release manager) take a look at them and give an opinion
>> regarding their suitability for 0.16?
> 
> Sorry about the slowdown, my BT internet connection at home got closed
> off, so working on a release is rather hard, hopefully that will be
> fixed within the next few days. Anyway, during the sprints Dag worked
> on the numpy attributes accesses to generate non-deprecated numpy
> code, I'm not sure how far along that is and if we can get that in as
> well, or have it wait until 0.16.1.

I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
not too long from now. There will certainly be things to fix after the
release anyway.


> I worked on enhancing fused types runtime dispatching (e.g. select the
> right specialization at runtime from the argument types), especially
> on matching buffers. It's also a bit of a rewrite as the complexity
> grew quite a bit and ndarray and fused types were broken in subtle
> ways (I would do the pull request now if I hadn't left the code at
> home :).

I took a tiny look at that when I wrote a test case and was surprised to
see that it imports "sys" inside of the generated function. Looks like
there's a bit left to optimise.

I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
"generic functions" module. It uses type dispatching as well, and he wrote
about the ways he found to tune that. Might be an interesting read.


> Finally, there is a serious bug in the reversed(range()) optimization
> that Mike Graham discovered when working on the range() optimization
> with a runtime step, which is basically wrong for any step that is not
> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
> actually has to figure out what the actual last value was in the range
> and use that as the start instead). I personally consider this a
> blocker, so I'll try contacting Mike and see if he is still
> (interested in) working on that.

Interesting. Yes, I can well imagine that being wrong. These things are
exceedingly tricky.

However, it's easy to disable the optimisation for a specific case - likely
easier than fixing it.

Also, I don't consider this a blocker. It's not like it works in the
current release, so it's not a regression compared to that.


> As for the two pull requests, both are quite large, but the dict
> iteration rewrite is more like an enhancement whereas the 'yield from'
> is really a new (somewhat big) feature. I would personally prefer to
> merge only enhancements and bug fixes, but not new features.

Funny. I would have proposed the opposite, because I consider the "yield
from" implementation (as a new language feature) safer than the dict
iteration (which impacts existing code). As I said, I wouldn't mind
postponing both to 0.16.1, but I think it would be better to get them out
and have people use them, so that we can also fix the bugs earlier.


> If you
> feel confident enough that the dict iteration pull request is ready to
> be merged, then please do so.

I'll take another look over it and decide. Thanks.


> We can do an 0.16.1 release with the
> 'yield from', the 'lnotab' line number mapping from code objects to
> Cython source

Note that that's not useful yet - it requires changes to the code that
*uses* the code objects as well, to a) actually reuse the code objects (iff
C line numbers are disabled in traces) and b) write the proper relative
line into the byte code offset field. I just put it up to have a place
where we can discuss further changes.


> and probably the boundscheck and wraparound
> optimizations I started working on (and other things if and when they
> come up). We can release 0.16.1 somewhat shortly after releasing 0.16.
> I think a shorter release cycle with less changes would be beneficial
> and give us swifter feedback about bugs, missing features, wanted
> features, etc.

+1


> So I propose that when we merged dict iteration, fused types
> enhancements and the range bug fix, we do another beta release. We (I)
> also need to look into the test failures on Gentoo.

Whatever you choose.

In case the PyPy developers manage to get one change in that I'd like to
see on their side (two new C-API functions for exception handling), I'll
also push some more Cython changes based on it that I have lying around.
That would hugely improve the compatibility for both.

In general, I started using the "CYTHON_COMPILING_IN_..." #defines wherever
they made sense, and specifically where we rely on CPython internals and
also in some places where we use borrowed references, especially long-lived
ones. PyPy can't handle those very well (and certainly not efficiently) due
to its garbage collector and its emulated ref-counting. I can well imagine
that that's similar for IronPython.

Stefan

From robertwb at gmail.com  Wed Mar 21 02:30:13 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 20 Mar 2012 18:30:13 -0700
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EX8LEsNjAf-+6rGxFMR0RYWhgF5NtboQeO0kdtsv5mhbg@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EXKQDUnKawcXVU9LqBMHKPDocNew9UOhEp=NW=-4JK2HA@mail.gmail.com>
	<CAEcYPwDdgKfoKDgUoksUtpmzPLUyA2L_EgBpz76t2s8n67vX9Q@mail.gmail.com>
	<4F5B5071.5040105@behnel.de>
	<CANg26EWwJ5pCkQCgjfeTR1yDnXqU7RDLJSduDgDYRv+op6XyqQ@mail.gmail.com>
	<CAEcYPwBJOCi_2_q2rJmZaPxS72HJLRz2Edgo03b8GiwRUBbR9A@mail.gmail.com>
	<CANg26EX8LEsNjAf-+6rGxFMR0RYWhgF5NtboQeO0kdtsv5mhbg@mail.gmail.com>
Message-ID: <CADiQ+QA9Ar-MvVOkPvWdZangt1uTrfbL6UtvF9K8Etr9X9nVCQ@mail.gmail.com>

On Sun, Mar 11, 2012 at 9:09 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 11 March 2012 09:26, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> On 11 March 2012 09:46, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 10 March 2012 14:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Lisandro Dalcin, 10.03.2012 10:51:
>>>>> On 10 March 2012 03:41, mark florisson wrote:
>>>>>> On 9 March 2012 14:22, Lisandro Dalcin wrote:
>>>>>>> I'm basically experiencing the issues here:
>>>>>>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html
>>>>>>>
>>>>>>> Can you imagine any way to workaround it?
>>>>>>
>>>>>> What a lovely C compiler bug... Did you file a bug report with gcc or
>>>>>> xcode? What version of gcc does Lion ship with?
>>>>>>
>>>>>
>>>>> I'm using a Mac just by accident, no idea (nor motivation) to report
>>>>> bugs to Apple.
>>>>>
>>>>> $ gcc --version
>>>>> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc.
>>>>> build 5658) (LLVM build 2336.9.00)
>>>>>
>>>>> Could that be the the bug is actually in the LLVM backend?
>>>>>
>>>>>> I suppose the macro would have to be disabled for OS X Lion inside
>>>>>> parallel sections (there seems to be __APPLE__ and __OSX__, I don't
>>>>>> know about detecting the OS X version), that's easy to do (undef and
>>>>>> redefine to no-ops before parallel section and redefine it again after
>>>>>> the section). I'll try fixing it during the sprints next week.
>>>>>
>>>>> Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside
>>>>> OpenMP blocks would be nicer than defining/undefining?
>>>>
>>>> Could that be coded into the macro or would it require to change the
>>>> generated code? But at least it sounds like it would not impact code in
>>>> functions that are being called from within the OpenMP blocks, would it?
>>>> Just the code straight inside the block. A work-around could still have a
>>>> substantial impact if it requires changes to the generated code.
>>>
>>> Yeah, that's why I suggested the undef/re-def approach around OpenMP
>>> blocks. It's some code bloat, but only for the C preprocessor, so it
>>> should be fine.
>>>
>>
>> I still feel bad about this. What about just disabling branch
>> prediction if OpenMP is ever used? Or perhaps just protect the
>> definitions of likely/unlikely with some guard, such as users can
>> disable them using a -D definition?
>>
>
> Why? That would disable all compiler optimizations in all code that
> could be gotten through the hints, due to a stupid C compiler bug when
> using cython.parallel. It would be better to do an easy fix, a nasty
> workaround for a nasty bug, but it shouldn't slow down other code. The
> only problem with it is a couple of extra bytes in your .c file, and
> the fact that it makes us feel less fuzzy inside because we realize
> our world isn't what we want it to be.

+1

From stefan_ml at behnel.de  Wed Mar 21 08:16:59 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Mar 2012 08:16:59 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CANg26EX03=Ypc=5E9yKNgmUFt2T2RYf_ynprztA0zMMthz0nPw@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
	<CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
	<CANg26EX03=Ypc=5E9yKNgmUFt2T2RYf_ynprztA0zMMthz0nPw@mail.gmail.com>
Message-ID: <4F69806B.6060408@behnel.de>

mark florisson, 16.03.2012 04:00:
> $ cpp
> #if defined(__APPLE__) || defined(__OSX__)
>     FOO
> #endif
> ^D

Ah, hackers. Isn't that just lovely?

Stefan

From stefan_ml at behnel.de  Wed Mar 21 09:03:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Mar 2012 09:03:43 +0100
Subject: [Cython] Exception instantiation
Message-ID: <4F698B5F.9010102@behnel.de>

Hi,

I was made aware of a difference between the ways Cython and CPython raise
exceptions. In CPython, the exception is always instantiated immediately,
even if you raise a plain type, i.e.

    raise TypeError

will actually raise a TypeError() instance. In Cython, the exception is
only stored, not instantiated. The difference can show when the
instantiation fails for some reason or has other side effects. Depending on
*when* and *if* the exception is actually instantiated, these effects may
show at unexpected times during execution, or may be not occur at all in
some corner cases (e.g. when the exception is shadowed by another one in
Py2 or caught by an except block in Cython that contains only C code).

I think this should change. We may consider leaving it as it is for known
exception types that are created without arguments, but otherwise, and
especially for user provided exceptions, we should take the safer route of
always instantiating them, like CPython does. That implies a certain
performance hit in some rather rare cases (of which an explicitly raised
StopIteration is the most visible one, if not exempted), but under normal
conditions where an exception is regularly caught at some point, the
overhead is the same and it would have less surprising corner cases.

Stefan

From dalcinl at gmail.com  Wed Mar 21 09:09:34 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 21 Mar 2012 11:09:34 +0300
Subject: [Cython] Exception instantiation
In-Reply-To: <4F698B5F.9010102@behnel.de>
References: <4F698B5F.9010102@behnel.de>
Message-ID: <CAEcYPwC=Ryk=Fq5B3+XW7MFeJTMiMCdvBz_rCmM1HhJDiaQnxA@mail.gmail.com>

On 21 March 2012 11:03, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
> I think this should change. We may consider leaving it as it is for known
> exception types that are created without arguments,

+1

> but otherwise, and
> especially for user provided exceptions, we should take the safer route of
> always instantiating them, like CPython does.

+1


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Wed Mar 21 14:31:11 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 21 Mar 2012 14:31:11 +0100
Subject: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized
 inside OpenMP blocks
In-Reply-To: <CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
References: <CAEcYPwAgaRXNStoUKWiyYxqV43G0=QiOXoQtzEoOKe4ok99Qcw@mail.gmail.com>
	<CANg26EVBXnzNGnbXOMYoKqrZ9LcaMzJRN7ki4zm3C_Cy8tB8vQ@mail.gmail.com>
	<CAEcYPwARSja=uQAz1+ce+ZiVbhNi6FrDwt0EnKFdPUL4UEAqzA@mail.gmail.com>
Message-ID: <CANg26EU55J8-TAFhPwaTu=zXiMDVqykTQikv=2scRVt+hs=aMw@mail.gmail.com>

Could you try the new fix from my branch? If it doesn't do the trick
then I will need to know whether the macros are passing the
preprocessor.

On 15 March 2012 13:21, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 14 March 2012 08:58, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 9 March 2012 14:22, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> Could you give this fix a try?
>> https://github.com/markflorisson88/cython/commit/2bffde15edc66c7416716051959e3b0cf1d6b41b
>>
>
> I'm still getting the linking error. Your fix does not resolve the issue :-(
>
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Wed Mar 21 14:47:16 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 21 Mar 2012 14:47:16 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F68C3B2.7020206@behnel.de>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
Message-ID: <CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>

On 20 March 2012 18:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 20.03.2012 17:40:
>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>> I put up two new pull requests on github:
>>>
>>> Implementation of PEP 380 (yield from):
>>> https://github.com/cython/cython/pull/96
>>>
>>> Rewrite of dict iteration:
>>> https://github.com/cython/cython/pull/95
>>>
>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>> that these are really nice features, could someone (and especially Mark, as
>>> the responsible release manager) take a look at them and give an opinion
>>> regarding their suitability for 0.16?
>>
>> Sorry about the slowdown, my BT internet connection at home got closed
>> off, so working on a release is rather hard, hopefully that will be
>> fixed within the next few days. Anyway, during the sprints Dag worked
>> on the numpy attributes accesses to generate non-deprecated numpy
>> code, I'm not sure how far along that is and if we can get that in as
>> well, or have it wait until 0.16.1.
>
> I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
> not too long from now. There will certainly be things to fix after the
> release anyway.
>
>
>> I worked on enhancing fused types runtime dispatching (e.g. select the
>> right specialization at runtime from the argument types), especially
>> on matching buffers. It's also a bit of a rewrite as the complexity
>> grew quite a bit and ndarray and fused types were broken in subtle
>> ways (I would do the pull request now if I hadn't left the code at
>> home :).
>
> I took a tiny look at that when I wrote a test case and was surprised to
> see that it imports "sys" inside of the generated function. Looks like
> there's a bit left to optimise.
>
> I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
> "generic functions" module. It uses type dispatching as well, and he wrote
> about the ways he found to tune that. Might be an interesting read.
>

Thanks, I'll look into that. Maybe the modules could be bound as
locals through default arguments.

>> Finally, there is a serious bug in the reversed(range()) optimization
>> that Mike Graham discovered when working on the range() optimization
>> with a runtime step, which is basically wrong for any step that is not
>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>> actually has to figure out what the actual last value was in the range
>> and use that as the start instead). I personally consider this a
>> blocker, so I'll try contacting Mike and see if he is still
>> (interested in) working on that.
>
> Interesting. Yes, I can well imagine that being wrong. These things are
> exceedingly tricky.
>
> However, it's easy to disable the optimisation for a specific case - likely
> easier than fixing it.
>
> Also, I don't consider this a blocker. It's not like it works in the
> current release, so it's not a regression compared to that.
>

Yes, it's not a regression, but I find it hard to release something
that is so broken, and I feel it kind of represents a lack of care
which may make people believe Cython can't be taken seriously.

>> As for the two pull requests, both are quite large, but the dict
>> iteration rewrite is more like an enhancement whereas the 'yield from'
>> is really a new (somewhat big) feature. I would personally prefer to
>> merge only enhancements and bug fixes, but not new features.
>
> Funny. I would have proposed the opposite, because I consider the "yield
> from" implementation (as a new language feature) safer than the dict
> iteration (which impacts existing code). As I said, I wouldn't mind
> postponing both to 0.16.1, but I think it would be better to get them out
> and have people use them, so that we can also fix the bugs earlier.
>

A good point. My reasoning was that we want less new functionality per
release, and keeping 'yield from' out of 0.16 makes us push out the
next release earlier as we will have pending functionality we want to
get out there. If the dict iteration is tested well enough (and it
looked to me like that), then hopefully code in sage or lxml will
break if there are leftover bugs, or our next beta might catch them
(or some). If you do want it in now, then that's fine with me as well,
you have a point.

>> If you
>> feel confident enough that the dict iteration pull request is ready to
>> be merged, then please do so.
>
> I'll take another look over it and decide. Thanks.
>
>
>> We can do an 0.16.1 release with the
>> 'yield from', the 'lnotab' line number mapping from code objects to
>> Cython source
>
> Note that that's not useful yet - it requires changes to the code that
> *uses* the code objects as well, to a) actually reuse the code objects (iff
> C line numbers are disabled in traces) and b) write the proper relative
> line into the byte code offset field. I just put it up to have a place
> where we can discuss further changes.
>
>
>> and probably the boundscheck and wraparound
>> optimizations I started working on (and other things if and when they
>> come up). We can release 0.16.1 somewhat shortly after releasing 0.16.
>> I think a shorter release cycle with less changes would be beneficial
>> and give us swifter feedback about bugs, missing features, wanted
>> features, etc.
>
> +1
>
>
>> So I propose that when we merged dict iteration, fused types
>> enhancements and the range bug fix, we do another beta release. We (I)
>> also need to look into the test failures on Gentoo.
>
> Whatever you choose.
>
> In case the PyPy developers manage to get one change in that I'd like to
> see on their side (two new C-API functions for exception handling), I'll
> also push some more Cython changes based on it that I have lying around.
> That would hugely improve the compatibility for both.
>
> In general, I started using the "CYTHON_COMPILING_IN_..." #defines wherever
> they made sense, and specifically where we rely on CPython internals and
> also in some places where we use borrowed references, especially long-lived
> ones. PyPy can't handle those very well (and certainly not efficiently) due
> to its garbage collector and its emulated ref-counting. I can well imagine
> that that's similar for IronPython.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Wed Mar 21 14:58:50 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Mar 2012 14:58:50 +0100
Subject: [Cython] Jenkins down
Message-ID: <4F69DE9A.6020100@behnel.de>

Hi,

it seems like the sage.math server is broken again, at least it's lacking
mounts when I try to log in. I'll restart Jenkins as soon as it's back working.

Stefan

From markflorisson88 at gmail.com  Wed Mar 21 15:07:09 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 21 Mar 2012 15:07:09 +0100
Subject: [Cython] Jenkins down
In-Reply-To: <4F69DE9A.6020100@behnel.de>
References: <4F69DE9A.6020100@behnel.de>
Message-ID: <CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>

On 21 March 2012 14:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> it seems like the sage.math server is broken again, at least it's lacking
> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Indeed, it's rather worrying and is actually a bit of a slowdown for
development. Dag and I discussed getting some funding from numfocus to
use Jenkins in a cloud environment (along possibly with other
projects, like sage, lxml, numpy, scipy, depending on interest of
respective projects). At PyCon we saw one such company that provides
these services, which is shiningpanda.com (which is down right now,
which isn't very promising :).

From markflorisson88 at gmail.com  Wed Mar 21 15:12:09 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 21 Mar 2012 15:12:09 +0100
Subject: [Cython] Jenkins down
In-Reply-To: <CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
Message-ID: <CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>

On 21 March 2012 15:07, mark florisson <markflorisson88 at gmail.com> wrote:
> On 21 March 2012 14:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> it seems like the sage.math server is broken again, at least it's lacking
>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Indeed, it's rather worrying and is actually a bit of a slowdown for
> development. Dag and I discussed getting some funding from numfocus to
> use Jenkins in a cloud environment (along possibly with other
> projects, like sage, lxml, numpy, scipy, depending on interest of
> respective projects). At PyCon we saw one such company that provides
> these services, which is shiningpanda.com (which is down right now,
> which isn't very promising :).

Or maybe that was my flaky wifi connection :p. Anyway, there is also
TeamCity from Jetbrains http://www.jetbrains.com/teamcity/, which is
supposed to be free for small teams , and it can be used together with
Amazon's cloud services:
http://www.jetbrains.com/teamcity/features/amazon_ec2.html .

From stefan_ml at behnel.de  Wed Mar 21 16:58:20 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Mar 2012 16:58:20 +0100
Subject: [Cython] Jenkins down
In-Reply-To: <CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
	<CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>
Message-ID: <4F69FA9C.2090107@behnel.de>

mark florisson, 21.03.2012 15:12:
> On 21 March 2012 15:07, mark florisson wrote:
>> On 21 March 2012 14:58, Stefan Behnel wrote:
>>> it seems like the sage.math server is broken again, at least it's lacking
>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>
>> Indeed, it's rather worrying and is actually a bit of a slowdown for
>> development.

Sure. However, we get to use it for free, and we've had a good gain from
that for a long time now. It's just bad timing that this coincides with our
release preparations.


>> Dag and I discussed getting some funding from numfocus to
>> use Jenkins in a cloud environment (along possibly with other
>> projects, like sage, lxml, numpy, scipy, depending on interest of
>> respective projects). At PyCon we saw one such company that provides
>> these services, which is shiningpanda.com

Their free OSS plan is one hour per day, which is way too short for us. Not
sure how much we'd want to pay for their dedicated plan, and no idea what
performance we'd get for the money.

For our current setup, you can count way more than one hour of processing
time for each single push and for each day's run of the CPython triggered
integration tests. And then there's our set of separate developer branches
on top of that, plus the fact that we can run builds in parallel to get
faster feedback.

Seriously - thanks William!


> Anyway, there is also
> TeamCity from Jetbrains http://www.jetbrains.com/teamcity/, which is
> supposed to be free for small teams , and it can be used together with
> Amazon's cloud services:
> http://www.jetbrains.com/teamcity/features/amazon_ec2.html .

I'm quite happy with Jenkins, though.

Stefan

From wstein at gmail.com  Wed Mar 21 17:00:51 2012
From: wstein at gmail.com (William Stein)
Date: Wed, 21 Mar 2012 12:00:51 -0400
Subject: [Cython] Jenkins down
In-Reply-To: <4F69FA9C.2090107@behnel.de>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
	<CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>
	<4F69FA9C.2090107@behnel.de>
Message-ID: <CACLE5GAU-fyE8fMKtviGo_fX=qh9pFdHeXCJUkEF+o+qdR9k8w@mail.gmail.com>

On Wed, Mar 21, 2012 at 11:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 21.03.2012 15:12:
>> On 21 March 2012 15:07, mark florisson wrote:
>>> On 21 March 2012 14:58, Stefan Behnel wrote:
>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>
>>> Indeed, it's rather worrying and is actually a bit of a slowdown for
>>> development.
>
> Sure. However, we get to use it for free, and we've had a good gain from
> that for a long time now. It's just bad timing that this coincides with our
> release preparations.
>
>
>>> Dag and I discussed getting some funding from numfocus to
>>> use Jenkins in a cloud environment (along possibly with other
>>> projects, like sage, lxml, numpy, scipy, depending on interest of
>>> respective projects). At PyCon we saw one such company that provides
>>> these services, which is shiningpanda.com
>
> Their free OSS plan is one hour per day, which is way too short for us. Not
> sure how much we'd want to pay for their dedicated plan, and no idea what
> performance we'd get for the money.
>
> For our current setup, you can count way more than one hour of processing
> time for each single push and for each day's run of the CPython triggered
> integration tests. And then there's our set of separate developer branches
> on top of that, plus the fact that we can run builds in parallel to get
> faster feedback.
>
> Seriously - thanks William!

And thanks to the National Science Foundation for buying the machines.

If anybody reading this can volunteer some "sysadmin time and
expertise" that could help improve the stability of sage.math.   The
latest crash was purely linux/software related.

>
>
>> Anyway, there is also
>> TeamCity from Jetbrains http://www.jetbrains.com/teamcity/, which is
>> supposed to be free for small teams , and it can be used together with
>> Amazon's cloud services:
>> http://www.jetbrains.com/teamcity/features/amazon_ec2.html .
>
> I'm quite happy with Jenkins, though.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From markflorisson88 at gmail.com  Wed Mar 21 18:22:11 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 21 Mar 2012 17:22:11 +0000
Subject: [Cython] Jenkins down
In-Reply-To: <4F69FA9C.2090107@behnel.de>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
	<CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>
	<4F69FA9C.2090107@behnel.de>
Message-ID: <CANg26EXXRh9cJmr+gTKTbyk4w9T6RFnumvPoOXMMhXGfH5-h4A@mail.gmail.com>

On 21 March 2012 15:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 21.03.2012 15:12:
>> On 21 March 2012 15:07, mark florisson wrote:
>>> On 21 March 2012 14:58, Stefan Behnel wrote:
>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>
>>> Indeed, it's rather worrying and is actually a bit of a slowdown for
>>> development.
>
> Sure. However, we get to use it for free, and we've had a good gain from
> that for a long time now. It's just bad timing that this coincides with our
> release preparations.
>
>
>>> Dag and I discussed getting some funding from numfocus to
>>> use Jenkins in a cloud environment (along possibly with other
>>> projects, like sage, lxml, numpy, scipy, depending on interest of
>>> respective projects). At PyCon we saw one such company that provides
>>> these services, which is shiningpanda.com
>
> Their free OSS plan is one hour per day, which is way too short for us. Not
> sure how much we'd want to pay for their dedicated plan, and no idea what
> performance we'd get for the money.
>
> For our current setup, you can count way more than one hour of processing
> time for each single push and for each day's run of the CPython triggered
> integration tests. And then there's our set of separate developer branches
> on top of that, plus the fact that we can run builds in parallel to get
> faster feedback.
>
> Seriously - thanks William!
>

Yeah I also like it quite a bit, and I am grateful for these servers
as well. It's just that they have been down quite often lately. For
$240 a month you get one build queue for an unlimited number of
projects, but indeed, I don't know the performance. Just wanted to
bring it up in case there is any interest. At least with amazon I
think you can run all tests in parallel as it can just fire up more
VMs.

>> Anyway, there is also
>> TeamCity from Jetbrains http://www.jetbrains.com/teamcity/, which is
>> supposed to be free for small teams , and it can be used together with
>> Amazon's cloud services:
>> http://www.jetbrains.com/teamcity/features/amazon_ec2.html .
>
> I'm quite happy with Jenkins, though.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Wed Mar 21 19:56:37 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 21 Mar 2012 11:56:37 -0700
Subject: [Cython] Exception instantiation
In-Reply-To: <4F698B5F.9010102@behnel.de>
References: <4F698B5F.9010102@behnel.de>
Message-ID: <CADiQ+QBJbp9dNthTX1tDnYjEjg5yjoF1hYDoWoyygzuMpzCP=Q@mail.gmail.com>

On Wed, Mar 21, 2012 at 1:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I was made aware of a difference between the ways Cython and CPython raise
> exceptions. In CPython, the exception is always instantiated immediately,
> even if you raise a plain type, i.e.
>
> ? ?raise TypeError
>
> will actually raise a TypeError() instance. In Cython, the exception is
> only stored, not instantiated. The difference can show when the
> instantiation fails for some reason or has other side effects. Depending on
> *when* and *if* the exception is actually instantiated, these effects may
> show at unexpected times during execution, or may be not occur at all in
> some corner cases (e.g. when the exception is shadowed by another one in
> Py2 or caught by an except block in Cython that contains only C code).
>
> I think this should change. We may consider leaving it as it is for known
> exception types that are created without arguments, but otherwise, and
> especially for user provided exceptions, we should take the safer route of
> always instantiating them, like CPython does. That implies a certain
> performance hit in some rather rare cases (of which an explicitly raised
> StopIteration is the most visible one, if not exempted), but under normal
> conditions where an exception is regularly caught at some point, the
> overhead is the same and it would have less surprising corner cases.

+1. One could consider making an exception for the no-argument builtin
(side-effect-free) ones, but this would just be an optimization, and
we're typically already in an expensive, exceptional state at this
point.

- Robert

From robertwb at gmail.com  Wed Mar 21 20:06:48 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 21 Mar 2012 12:06:48 -0700
Subject: [Cython] Jenkins down
In-Reply-To: <4F69FA9C.2090107@behnel.de>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EVNJKD_RK5NOGO_ZQOhPEn7imaQ6mSTXxfYCLQox1CA1g@mail.gmail.com>
	<CANg26EW3z=F9aomA0sv3cKObKgE-sNQtaePXHQiEH6H57wr+LA@mail.gmail.com>
	<4F69FA9C.2090107@behnel.de>
Message-ID: <CADiQ+QAVcDkbUj=t_NVM6NHXmR68r5ROG3kAY09J5JmSx3rLvA@mail.gmail.com>

On Wed, Mar 21, 2012 at 8:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 21.03.2012 15:12:
>> On 21 March 2012 15:07, mark florisson wrote:
>>> On 21 March 2012 14:58, Stefan Behnel wrote:
>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>
>>> Indeed, it's rather worrying and is actually a bit of a slowdown for
>>> development.
>
> Sure. However, we get to use it for free, and we've had a good gain from
> that for a long time now. It's just bad timing that this coincides with our
> release preparations.

I'd say that since I've started using it sage.math has averaged close
to 99% uptime, it's just bad timing at the moment.

>>> Dag and I discussed getting some funding from numfocus to
>>> use Jenkins in a cloud environment (along possibly with other
>>> projects, like sage, lxml, numpy, scipy, depending on interest of
>>> respective projects). At PyCon we saw one such company that provides
>>> these services, which is shiningpanda.com
>
> Their free OSS plan is one hour per day, which is way too short for us. Not
> sure how much we'd want to pay for their dedicated plan, and no idea what
> performance we'd get for the money.
>
> For our current setup, you can count way more than one hour of processing
> time for each single push and for each day's run of the CPython triggered
> integration tests. And then there's our set of separate developer branches
> on top of that, plus the fact that we can run builds in parallel to get
> faster feedback.
>
> Seriously - thanks William!

+1, this has made a huge difference in our development process. A
system powerful enough to rebuild and retest Sage as well as running
Cython's own self-tests is certainly a worthwhile sue of this hardware
from Sage's point of view as well.

Unfortunately, I have neither the skills nor time to do the kinds of
sys-adminy things that sage.math needs.

- Robert

From robertwb at gmail.com  Wed Mar 21 21:24:08 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 21 Mar 2012 13:24:08 -0700
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
Message-ID: <CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>

On Wed, Mar 21, 2012 at 6:47 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 20 March 2012 18:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 20.03.2012 17:40:
>>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>>> I put up two new pull requests on github:
>>>>
>>>> Implementation of PEP 380 (yield from):
>>>> https://github.com/cython/cython/pull/96
>>>>
>>>> Rewrite of dict iteration:
>>>> https://github.com/cython/cython/pull/95
>>>>
>>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>>> that these are really nice features, could someone (and especially Mark, as
>>>> the responsible release manager) take a look at them and give an opinion
>>>> regarding their suitability for 0.16?
>>>
>>> Sorry about the slowdown, my BT internet connection at home got closed
>>> off, so working on a release is rather hard, hopefully that will be
>>> fixed within the next few days.

Thanks for the status update. Is there a separate release branch, or
are you working off the github master? Is there a list of
issues/blockers somewhere?

>>> Anyway, during the sprints Dag worked
>>> on the numpy attributes accesses to generate non-deprecated numpy
>>> code, I'm not sure how far along that is and if we can get that in as
>>> well, or have it wait until 0.16.1.
>>
>> I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
>> not too long from now. There will certainly be things to fix after the
>> release anyway.
>>
>>
>>> I worked on enhancing fused types runtime dispatching (e.g. select the
>>> right specialization at runtime from the argument types), especially
>>> on matching buffers. It's also a bit of a rewrite as the complexity
>>> grew quite a bit and ndarray and fused types were broken in subtle
>>> ways (I would do the pull request now if I hadn't left the code at
>>> home :).
>>
>> I took a tiny look at that when I wrote a test case and was surprised to
>> see that it imports "sys" inside of the generated function. Looks like
>> there's a bit left to optimise.
>>
>> I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
>> "generic functions" module. It uses type dispatching as well, and he wrote
>> about the ways he found to tune that. Might be an interesting read.
>>
>
> Thanks, I'll look into that. Maybe the modules could be bound as
> locals through default arguments.
>
>>> Finally, there is a serious bug in the reversed(range()) optimization
>>> that Mike Graham discovered when working on the range() optimization
>>> with a runtime step, which is basically wrong for any step that is not
>>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>>> actually has to figure out what the actual last value was in the range
>>> and use that as the start instead). I personally consider this a
>>> blocker, so I'll try contacting Mike and see if he is still
>>> (interested in) working on that.
>>
>> Interesting. Yes, I can well imagine that being wrong. These things are
>> exceedingly tricky.
>>
>> However, it's easy to disable the optimisation for a specific case - likely
>> easier than fixing it.
>>
>> Also, I don't consider this a blocker. It's not like it works in the
>> current release, so it's not a regression compared to that.
>>
>
> Yes, it's not a regression, but I find it hard to release something
> that is so broken, and I feel it kind of represents a lack of care
> which may make people believe Cython can't be taken seriously.

I'm usually in the "not a regression" camp as well, but this is pretty
fundamental. Sounds like an easy fix, or at least an easy-to-disable
optimization.

>>> As for the two pull requests, both are quite large, but the dict
>>> iteration rewrite is more like an enhancement whereas the 'yield from'
>>> is really a new (somewhat big) feature. I would personally prefer to
>>> merge only enhancements and bug fixes, but not new features.
>>
>> Funny. I would have proposed the opposite, because I consider the "yield
>> from" implementation (as a new language feature) safer than the dict
>> iteration (which impacts existing code). As I said, I wouldn't mind
>> postponing both to 0.16.1, but I think it would be better to get them out
>> and have people use them, so that we can also fix the bugs earlier.
>>
>
> A good point. My reasoning was that we want less new functionality per
> release, and keeping 'yield from' out of 0.16 makes us push out the
> next release earlier as we will have pending functionality we want to
> get out there. If the dict iteration is tested well enough (and it
> looked to me like that), then hopefully code in sage or lxml will
> break if there are leftover bugs, or our next beta might catch them
> (or some). If you do want it in now, then that's fine with me as well,
> you have a point.

Has this been pushed into the devel branch for sufficient testing
there (e.g. on Sage)? Personally, I'd like to see the release get out
sooner rather than later.

- Robert

From robertwb at gmail.com  Wed Mar 21 21:56:10 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 21 Mar 2012 13:56:10 -0700
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
Message-ID: <CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>

On Sat, Mar 10, 2012 at 10:44 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 8 March 2012 14:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> I noticed that people start rushing for the next season on Python's GSoC
>> mailing lists. Do we have any interested developers here, or general ideas
>> about suitable topics? I would expect that we'll do as in the last years
>> and participate under Python's umbrella.

Also, we'd like to see patches from anyone interesting in being a GSoC
student, as this will be a requirement as in past years.

> I will likely be submitting a proposal for the OpenCL support CEP.

OpenCL would be an interesting experiment, but I think still has
limited utility. Dag and I were talking the other day about the
challenge of generating the best possible code for evaluating array
expressions (think inlined memoryview arithmetic) taking into account
memory layout, blocking, etc. which Fortran does really well which
could be an interesting direction.

> As for other proposals, anyone can come up with something themselves or
> choose a suitable CEP. Some ideas:

Numbering items for clarity:

1.

> - fused cdef classes (probably not as an entire gsoc project)

2.

> - profile guided optimizations (using python's profilers and/or a
> custom profiler that collects data such as types etc, which can be
> used to specialize variables inside loops (or entire functions) with a
> fallback to normal mode in case the type changes)

3.

> - cython library to contain common functionality (although that might
> be a bit boring and rather involved)

4.

> - better type inference, that would be enabled by default and again
> handle thing like reassignments of variables and fallbacks to the
> default object type. With entry caching Cython could build a database
> of types ((extension) classes, functions, variables) used in the
> modules and functions that are compiled (also def functions), and
> infer the types used and specialize on those. Maybe a switch should be
> added to cython to handle circular dependencies, or maybe with the
> distutils preprocessing it can run all the type inference first and
> keep track of unresolved entries, and try to fill those in after
> building the database. For bonus points the user can be allowed to
> write plugins to aid the process.

5.

> - llvm based JIT :), i.e. have Cython instrument the generated code to
> record information and use that to create specializations at runtime
> (probably far out for a gsoc)


What I would most like to see is the common component in 2 and 4, i.e.
the ability to generate optimized code for imperfectly-inferred types,
with transparent fallback to generic code if conditions are not met at
runtime (during as well as at entering the optimized code path). Too
many times we have to decide between being safe for all cases or fast
for the common case.

That being said, there are few people I'd trust with such an ambitious
project, and you're on that short list :).

- Robert

From d.s.seljebotn at astro.uio.no  Wed Mar 21 22:11:19 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 21 Mar 2012 14:11:19 -0700
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
	<CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>
Message-ID: <4F6A43F7.3080807@astro.uio.no>

On 03/21/2012 01:56 PM, Robert Bradshaw wrote:
> On Sat, Mar 10, 2012 at 10:44 PM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> On 8 March 2012 14:27, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>>> Hi,
>>>
>>> I noticed that people start rushing for the next season on Python's GSoC
>>> mailing lists. Do we have any interested developers here, or general ideas
>>> about suitable topics? I would expect that we'll do as in the last years
>>> and participate under Python's umbrella.
>
> Also, we'd like to see patches from anyone interesting in being a GSoC
> student, as this will be a requirement as in past years.
>
>> I will likely be submitting a proposal for the OpenCL support CEP.
>
> OpenCL would be an interesting experiment, but I think still has
> limited utility. Dag and I were talking the other day about the
> challenge of generating the best possible code for evaluating array
> expressions (think inlined memoryview arithmetic) taking into account
> memory layout, blocking, etc. which Fortran does really well which
> could be an interesting direction.

Yes, a lot of water has run in the river since March 8 here. Anyone 
interested in reading up on current ideas on what Mark is thinking for 
his GSoC proposal should read up on the numpy-discussion thread "Looking 
for people interested in helping with Python compiler to LLVM".

Dag

>
>> As for other proposals, anyone can come up with something themselves or
>> choose a suitable CEP. Some ideas:
>
> Numbering items for clarity:
>
> 1.
>
>> - fused cdef classes (probably not as an entire gsoc project)
>
> 2.
>
>> - profile guided optimizations (using python's profilers and/or a
>> custom profiler that collects data such as types etc, which can be
>> used to specialize variables inside loops (or entire functions) with a
>> fallback to normal mode in case the type changes)
>
> 3.
>
>> - cython library to contain common functionality (although that might
>> be a bit boring and rather involved)
>
> 4.
>
>> - better type inference, that would be enabled by default and again
>> handle thing like reassignments of variables and fallbacks to the
>> default object type. With entry caching Cython could build a database
>> of types ((extension) classes, functions, variables) used in the
>> modules and functions that are compiled (also def functions), and
>> infer the types used and specialize on those. Maybe a switch should be
>> added to cython to handle circular dependencies, or maybe with the
>> distutils preprocessing it can run all the type inference first and
>> keep track of unresolved entries, and try to fill those in after
>> building the database. For bonus points the user can be allowed to
>> write plugins to aid the process.
>
> 5.
>
>> - llvm based JIT :), i.e. have Cython instrument the generated code to
>> record information and use that to create specializations at runtime
>> (probably far out for a gsoc)
>
>
> What I would most like to see is the common component in 2 and 4, i.e.
> the ability to generate optimized code for imperfectly-inferred types,
> with transparent fallback to generic code if conditions are not met at
> runtime (during as well as at entering the optimized code path). Too
> many times we have to decide between being safe for all cases or fast
> for the common case.
>
> That being said, there are few people I'd trust with such an ambitious
> project, and you're on that short list :).
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From robertwb at gmail.com  Wed Mar 21 22:39:14 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 21 Mar 2012 14:39:14 -0700
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <4F6A43F7.3080807@astro.uio.no>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
	<CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>
	<4F6A43F7.3080807@astro.uio.no>
Message-ID: <CADiQ+QCcLxm2-d27bJrYDBE0C3ouGAsaQACgz=_3zcuz1_=PnA@mail.gmail.com>

On Wed, Mar 21, 2012 at 2:11 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 03/21/2012 01:56 PM, Robert Bradshaw wrote:
>>
>> On Sat, Mar 10, 2012 at 10:44 PM, mark florisson
>> <markflorisson88 at gmail.com> ?wrote:
>>>
>>> On 8 March 2012 14:27, Stefan Behnel<stefan_ml at behnel.de> ?wrote:
>>>>
>>>> Hi,
>>>>
>>>> I noticed that people start rushing for the next season on Python's GSoC
>>>> mailing lists. Do we have any interested developers here, or general
>>>> ideas
>>>> about suitable topics? I would expect that we'll do as in the last years
>>>> and participate under Python's umbrella.
>>
>>
>> Also, we'd like to see patches from anyone interesting in being a GSoC
>> student, as this will be a requirement as in past years.
>>
>>> I will likely be submitting a proposal for the OpenCL support CEP.
>>
>>
>> OpenCL would be an interesting experiment, but I think still has
>> limited utility. Dag and I were talking the other day about the
>> challenge of generating the best possible code for evaluating array
>> expressions (think inlined memoryview arithmetic) taking into account
>> memory layout, blocking, etc. which Fortran does really well which
>> could be an interesting direction.
>
>
> Yes, a lot of water has run in the river since March 8 here. Anyone
> interested in reading up on current ideas on what Mark is thinking for his
> GSoC proposal should read up on the numpy-discussion thread "Looking for
> people interested in helping with Python compiler to LLVM".

Thanks for the pointer. Clearly I'm not subscribed to all the relevant
lists (and, admittedly, just finally starting to catch up on the ones
I am subscribed to).

- Robert

From stefan_ml at behnel.de  Thu Mar 22 08:18:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 08:18:07 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
	<CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
Message-ID: <4F6AD22F.6040602@behnel.de>

Robert Bradshaw, 21.03.2012 21:24:
> On Wed, Mar 21, 2012 at 6:47 AM, mark florisson wrote:
>> On 20 March 2012 18:51, Stefan Behnel wrote:
>>> mark florisson, 20.03.2012 17:40:
>>>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>>>> I put up two new pull requests on github:
>>>>>
>>>>> Implementation of PEP 380 (yield from):
>>>>> https://github.com/cython/cython/pull/96
>>>>>
>>>>> Rewrite of dict iteration:
>>>>> https://github.com/cython/cython/pull/95
>>>>>
>>>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>>>> that these are really nice features, could someone (and especially Mark, as
>>>>> the responsible release manager) take a look at them and give an opinion
>>>>> regarding their suitability for 0.16?
>>>>
>>>> Sorry about the slowdown, my BT internet connection at home got closed
>>>> off, so working on a release is rather hard, hopefully that will be
>>>> fixed within the next few days.
> 
> Thanks for the status update. Is there a separate release branch, or
> are you working off the github master?

Not currently. I also agree that we should try to get out the release soon.
What about this: I merge my pending pull requests, we test everything in
Sage (it's been normally tested in my dev branch already), and if that
doesn't break anything, we release another beta? If it does break anything,
it's either easy enough to fix, or we can create a release branch at that
point with the status before the merges.

sage.math seems to be back up, although the /levi drive isn't. If there are
no objections, I'll make sure Jenkins gets back up and then do the above.


> Is there a list of issues/blockers somewhere?

Don't know.


>>>> Finally, there is a serious bug in the reversed(range()) optimization
>>>> that Mike Graham discovered when working on the range() optimization
>>>> with a runtime step, which is basically wrong for any step that is not
>>>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>>>> actually has to figure out what the actual last value was in the range
>>>> and use that as the start instead). I personally consider this a
>>>> blocker, so I'll try contacting Mike and see if he is still
>>>> (interested in) working on that.
>>>
>>> Interesting. Yes, I can well imagine that being wrong. These things are
>>> exceedingly tricky.
>>>
>>> However, it's easy to disable the optimisation for a specific case - likely
>>> easier than fixing it.
>>>
>>> Also, I don't consider this a blocker. It's not like it works in the
>>> current release, so it's not a regression compared to that.
>>
>> Yes, it's not a regression, but I find it hard to release something
>> that is so broken, and I feel it kind of represents a lack of care
>> which may make people believe Cython can't be taken seriously.
> 
> I'm usually in the "not a regression" camp as well, but this is pretty
> fundamental. Sounds like an easy fix, or at least an easy-to-disable
> optimization.

I'll take a look - maybe not before the weekend though.

Stefan

From stefan_ml at behnel.de  Thu Mar 22 08:34:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 08:34:51 +0100
Subject: [Cython] [cython-users] GSoC 2012
In-Reply-To: <CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>
References: <4F58B3DA.5070602@behnel.de>
	<CANg26EWFUuMMOLzY095Ny7ChETUAGzJvZMty-J8DHZK+GatStg@mail.gmail.com>
	<CADiQ+QDCYnz4DruDrnoZitFsLLX0Z1BroNogtQUPAD7MmUbeSA@mail.gmail.com>
Message-ID: <4F6AD61B.6050705@behnel.de>

Robert Bradshaw, 21.03.2012 21:56:
> On Sat, Mar 10, 2012 at 10:44 PM, mark florisson wrote:
> 2.
>> - profile guided optimizations (using python's profilers and/or a
>> custom profiler that collects data such as types etc, which can be
>> used to specialize variables inside loops (or entire functions) with a
>> fallback to normal mode in case the type changes)
> 4.
>> - better type inference, that would be enabled by default and again
>> handle thing like reassignments of variables and fallbacks to the
>> default object type. With entry caching Cython could build a database
>> of types ((extension) classes, functions, variables) used in the
>> modules and functions that are compiled (also def functions), and
>> infer the types used and specialize on those. Maybe a switch should be
>> added to cython to handle circular dependencies, or maybe with the
>> distutils preprocessing it can run all the type inference first and
>> keep track of unresolved entries, and try to fill those in after
>> building the database. For bonus points the user can be allowed to
>> write plugins to aid the process.
> 
> 
> What I would most like to see is the common component in 2 and 4, i.e.
> the ability to generate optimized code for imperfectly-inferred types,
> with transparent fallback to generic code if conditions are not met at
> runtime (during as well as at entering the optimized code path).

Absolutely. The dict iteration change is pointing exactly in that
direction, but there are so many other places where the same thing applies.

Basically, it has separate C implementations for different cases all folded
into an inlined helper function with flag parameters, and then sets some of
the flags to 1/0 constants at compile time and determines others at
runtime. The C compiler can then just drop any inaccessible code and use
the inlined remainings of the function to infer a good way of streamlining
the rest of the surrounding code.

This also turned out to be a perfect way to enable or disable certain
optimisations and/or implementation details for different backends (PyPy
and CPython currently). Much better than actually generating different C
code for them.

Stefan

From markflorisson88 at gmail.com  Thu Mar 22 13:40:02 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 22 Mar 2012 12:40:02 +0000
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F6AD22F.6040602@behnel.de>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
	<CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
	<4F6AD22F.6040602@behnel.de>
Message-ID: <CANg26EU-RSasy1nTnkO-QRGw00hHTOZHn4r4O51pAGKPAYNQPg@mail.gmail.com>

On 22 March 2012 07:18, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 21.03.2012 21:24:
>> On Wed, Mar 21, 2012 at 6:47 AM, mark florisson wrote:
>>> On 20 March 2012 18:51, Stefan Behnel wrote:
>>>> mark florisson, 20.03.2012 17:40:
>>>>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>>>>> I put up two new pull requests on github:
>>>>>>
>>>>>> Implementation of PEP 380 (yield from):
>>>>>> https://github.com/cython/cython/pull/96
>>>>>>
>>>>>> Rewrite of dict iteration:
>>>>>> https://github.com/cython/cython/pull/95
>>>>>>
>>>>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>>>>> that these are really nice features, could someone (and especially Mark, as
>>>>>> the responsible release manager) take a look at them and give an opinion
>>>>>> regarding their suitability for 0.16?
>>>>>
>>>>> Sorry about the slowdown, my BT internet connection at home got closed
>>>>> off, so working on a release is rather hard, hopefully that will be
>>>>> fixed within the next few days.
>>
>> Thanks for the status update. Is there a separate release branch, or
>> are you working off the github master?
>
> Not currently. I also agree that we should try to get out the release soon.
> What about this: I merge my pending pull requests, we test everything in
> Sage (it's been normally tested in my dev branch already), and if that
> doesn't break anything, we release another beta? If it does break anything,
> it's either easy enough to fix, or we can create a release branch at that
> point with the status before the merges.
>
> sage.math seems to be back up, although the /levi drive isn't. If there are
> no objections, I'll make sure Jenkins gets back up and then do the above.
>
>
>> Is there a list of issues/blockers somewhere?
>
> Don't know.
>

Mostly the pull requests and this reversed bug
(http://trac.cython.org/cython_trac/ticket/763), which I marked as a
blocker. As much as I would like to push stuff out early, I really
find it more important to push something that is as stable and useful
as possible, even if it delays a release by a couple of weeks.

I'll make a release branch, a clone of the current master, but I want
several other things in there as well:
    1) Dag's pull request for the portable attribute access of
np.ndarray (I think this is important as np.ndarray is commonly used
and in future numpy versions the attributes might be removed
altogether)
    2) Runtime dispatch for buffers for fused types. This would make
fused types much more useable and useful for many people. It's nearly
finished, I just need to add a few more tests and test on Jenkins

Basically until Jenkins is up and everything is tested there, we
shouldn't push anything into the release branch.

>>>>> Finally, there is a serious bug in the reversed(range()) optimization
>>>>> that Mike Graham discovered when working on the range() optimization
>>>>> with a runtime step, which is basically wrong for any step that is not
>>>>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>>>>> actually has to figure out what the actual last value was in the range
>>>>> and use that as the start instead). I personally consider this a
>>>>> blocker, so I'll try contacting Mike and see if he is still
>>>>> (interested in) working on that.
>>>>
>>>> Interesting. Yes, I can well imagine that being wrong. These things are
>>>> exceedingly tricky.
>>>>
>>>> However, it's easy to disable the optimisation for a specific case - likely
>>>> easier than fixing it.
>>>>
>>>> Also, I don't consider this a blocker. It's not like it works in the
>>>> current release, so it's not a regression compared to that.
>>>
>>> Yes, it's not a regression, but I find it hard to release something
>>> that is so broken, and I feel it kind of represents a lack of care
>>> which may make people believe Cython can't be taken seriously.
>>
>> I'm usually in the "not a regression" camp as well, but this is pretty
>> fundamental. Sounds like an easy fix, or at least an easy-to-disable
>> optimization.
>
> I'll take a look - maybe not before the weekend though.

It would be easy to disable. Personally I'd like to give the people at
PyCon who where interested in sprinting with us the chance to finish
what they were working on, in this case Mike Graham, who indicated
that he means to finish his work on the bug (and add proper tests for
them). I think we shouldn't pull work out of people's hands, but
instead encourage them to work on Cython and possibly contribute in
the future.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Thu Mar 22 13:46:15 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 22 Mar 2012 12:46:15 +0000
Subject: [Cython] Jenkins down
In-Reply-To: <4F69DE9A.6020100@behnel.de>
References: <4F69DE9A.6020100@behnel.de>
Message-ID: <CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>

On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> it seems like the sage.math server is broken again, at least it's lacking
> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Logging in through ssh seems to work, should I run any of the scripts
from your home directory Stefan? e.g. restart_dead_hudson.sh or
starthudson.sh?

From wstein at gmail.com  Thu Mar 22 14:11:53 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 09:11:53 -0400
Subject: [Cython] sage.math problems?
In-Reply-To: <CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
Message-ID: <CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>

On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby <david.kirkby at onetel.net> wrote:
> On 22 March 2012 12:33, William Stein <wstein at gmail.com> wrote:
>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>> Physical access is thus probably required. ?I've emailed some people who are in
>> Seattle, and we'll see. ?(I'm thousands of miles away now, and will
>> only get further away
>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>
> I believe any decent IT professional will tell you that you should not
> be using USB drives on servers.

Well I'm definitely currently not using USB drives on this server. :-(

If possible, can the stuff that was being done on /levi (jenkins for
Cython) be moved to /scratch, which is a reasonably fast NFS
filesystem.

 -- William

From markflorisson88 at gmail.com  Thu Mar 22 15:21:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 22 Mar 2012 14:21:43 +0000
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
	<CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
Message-ID: <CANg26EVyO34rJQ7PM_DXvZ_DyTo61Gztd_z3MEFOjAecaEmfgQ@mail.gmail.com>

On 21 March 2012 20:24, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Wed, Mar 21, 2012 at 6:47 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 20 March 2012 18:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 20.03.2012 17:40:
>>>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>>>> I put up two new pull requests on github:
>>>>>
>>>>> Implementation of PEP 380 (yield from):
>>>>> https://github.com/cython/cython/pull/96
>>>>>
>>>>> Rewrite of dict iteration:
>>>>> https://github.com/cython/cython/pull/95
>>>>>
>>>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>>>> that these are really nice features, could someone (and especially Mark, as
>>>>> the responsible release manager) take a look at them and give an opinion
>>>>> regarding their suitability for 0.16?
>>>>
>>>> Sorry about the slowdown, my BT internet connection at home got closed
>>>> off, so working on a release is rather hard, hopefully that will be
>>>> fixed within the next few days.
>
> Thanks for the status update. Is there a separate release branch, or
> are you working off the github master? Is there a list of
> issues/blockers somewhere?

I created a release branch called release0.16 (there was a previous
release branch for 0.15.1 which I could have pruned as well).

>>>> Anyway, during the sprints Dag worked
>>>> on the numpy attributes accesses to generate non-deprecated numpy
>>>> code, I'm not sure how far along that is and if we can get that in as
>>>> well, or have it wait until 0.16.1.
>>>
>>> I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
>>> not too long from now. There will certainly be things to fix after the
>>> release anyway.
>>>
>>>
>>>> I worked on enhancing fused types runtime dispatching (e.g. select the
>>>> right specialization at runtime from the argument types), especially
>>>> on matching buffers. It's also a bit of a rewrite as the complexity
>>>> grew quite a bit and ndarray and fused types were broken in subtle
>>>> ways (I would do the pull request now if I hadn't left the code at
>>>> home :).
>>>
>>> I took a tiny look at that when I wrote a test case and was surprised to
>>> see that it imports "sys" inside of the generated function. Looks like
>>> there's a bit left to optimise.
>>>
>>> I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
>>> "generic functions" module. It uses type dispatching as well, and he wrote
>>> about the ways he found to tune that. Might be an interesting read.
>>>
>>
>> Thanks, I'll look into that. Maybe the modules could be bound as
>> locals through default arguments.
>>
>>>> Finally, there is a serious bug in the reversed(range()) optimization
>>>> that Mike Graham discovered when working on the range() optimization
>>>> with a runtime step, which is basically wrong for any step that is not
>>>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>>>> actually has to figure out what the actual last value was in the range
>>>> and use that as the start instead). I personally consider this a
>>>> blocker, so I'll try contacting Mike and see if he is still
>>>> (interested in) working on that.
>>>
>>> Interesting. Yes, I can well imagine that being wrong. These things are
>>> exceedingly tricky.
>>>
>>> However, it's easy to disable the optimisation for a specific case - likely
>>> easier than fixing it.
>>>
>>> Also, I don't consider this a blocker. It's not like it works in the
>>> current release, so it's not a regression compared to that.
>>>
>>
>> Yes, it's not a regression, but I find it hard to release something
>> that is so broken, and I feel it kind of represents a lack of care
>> which may make people believe Cython can't be taken seriously.
>
> I'm usually in the "not a regression" camp as well, but this is pretty
> fundamental. Sounds like an easy fix, or at least an easy-to-disable
> optimization.
>
>>>> As for the two pull requests, both are quite large, but the dict
>>>> iteration rewrite is more like an enhancement whereas the 'yield from'
>>>> is really a new (somewhat big) feature. I would personally prefer to
>>>> merge only enhancements and bug fixes, but not new features.
>>>
>>> Funny. I would have proposed the opposite, because I consider the "yield
>>> from" implementation (as a new language feature) safer than the dict
>>> iteration (which impacts existing code). As I said, I wouldn't mind
>>> postponing both to 0.16.1, but I think it would be better to get them out
>>> and have people use them, so that we can also fix the bugs earlier.
>>>
>>
>> A good point. My reasoning was that we want less new functionality per
>> release, and keeping 'yield from' out of 0.16 makes us push out the
>> next release earlier as we will have pending functionality we want to
>> get out there. If the dict iteration is tested well enough (and it
>> looked to me like that), then hopefully code in sage or lxml will
>> break if there are leftover bugs, or our next beta might catch them
>> (or some). If you do want it in now, then that's fine with me as well,
>> you have a point.
>
> Has this been pushed into the devel branch for sufficient testing
> there (e.g. on Sage)? Personally, I'd like to see the release get out
> sooner rather than later.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Thu Mar 22 16:59:44 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 08:59:44 -0700
Subject: [Cython] Jenkins down
In-Reply-To: <CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
Message-ID: <CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>

Thanks for the offer. One problem I see is that /levi/scratch (a local
disk) has been dismounted (or reconfigured). To save ourselves random
permissions issues, it's probably best to run this script/job as
scoder (I've got sudo).

I'll see what I can do to get it running, assuming the sage.math
issues have been resolved.

- Robert

On Thu, Mar 22, 2012 at 5:46 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> it seems like the sage.math server is broken again, at least it's lacking
>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Logging in through ssh seems to work, should I run any of the scripts
> from your home directory Stefan? e.g. restart_dead_hudson.sh or
> starthudson.sh?
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Thu Mar 22 17:03:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 17:03:54 +0100
Subject: [Cython] sage.math problems?
In-Reply-To: <CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
Message-ID: <4F6B4D6A.9080103@behnel.de>

William Stein, 22.03.2012 14:11:
> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby wrote:
>> On 22 March 2012 12:33, William Stein wrote:
>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>> Physical access is thus probably required.  I've emailed some people who are in
>>> Seattle, and we'll see.  (I'm thousands of miles away now, and will
>>> only get further away
>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>
>> I believe any decent IT professional will tell you that you should not
>> be using USB drives on servers.
> 
> Well I'm definitely currently not using USB drives on this server. :-(
> 
> If possible, can the stuff that was being done on /levi (jenkins for
> Cython) be moved to /scratch, which is a reasonably fast NFS
> filesystem.

The whole Jenkins installation was on /levi (maybe not our best idea
ever...). I have backups of the complete configuration, but we'd still
loose the installed plugins and the build history. We'd literally start
from scratch and it would take some time to set it all up again. If it
turns out to be a matter of replugging the USB drive (or something
similarly simple), I would prefer copying the original installation over
(including the build history), rather than rebuilding it.

Also, the sage build directory was on levi and I don't know how that was
set up - Robert would know.

Stefan

From robertwb at gmail.com  Thu Mar 22 17:06:01 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 09:06:01 -0700
Subject: [Cython] sage.math problems?
In-Reply-To: <CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
Message-ID: <CADiQ+QCg3D0XykNF1LbJV8jL0Brcg=ZGKnr7UXZK8v0yOD_z1w@mail.gmail.com>

On Thu, Mar 22, 2012 at 6:11 AM, William Stein <wstein at gmail.com> wrote:
> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby <david.kirkby at onetel.net> wrote:
>> On 22 March 2012 12:33, William Stein <wstein at gmail.com> wrote:
>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>> Physical access is thus probably required. ?I've emailed some people who are in
>>> Seattle, and we'll see. ?(I'm thousands of miles away now, and will
>>> only get further away
>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>
>> I believe any decent IT professional will tell you that you should not
>> be using USB drives on servers.
>
> Well I'm definitely currently not using USB drives on this server. :-(
>
> If possible, can the stuff that was being done on /levi (jenkins for
> Cython) be moved to /scratch, which is a reasonably fast NFS
> filesystem.

The Cython tests do a lot of disk IO (checking out the Cython
repository, writing .c files, writing .o and .so files, starting
Python, ..). This could be an nsf mount, but is there somewhere
better? I think we'd like to store our configuration information in a
more persistant location.

- Robert

From robertwb at gmail.com  Thu Mar 22 17:07:25 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 09:07:25 -0700
Subject: [Cython] sage.math problems?
In-Reply-To: <4F6B4D6A.9080103@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de>
Message-ID: <CADiQ+QAv0uDjwTVbAYOYf7WQGkpXUXKP=BPTv5E+D2kqb=MOWw@mail.gmail.com>

On Thu, Mar 22, 2012 at 9:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> William Stein, 22.03.2012 14:11:
>> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby wrote:
>>> On 22 March 2012 12:33, William Stein wrote:
>>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>>> Physical access is thus probably required. ?I've emailed some people who are in
>>>> Seattle, and we'll see. ?(I'm thousands of miles away now, and will
>>>> only get further away
>>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>>
>>> I believe any decent IT professional will tell you that you should not
>>> be using USB drives on servers.
>>
>> Well I'm definitely currently not using USB drives on this server. :-(
>>
>> If possible, can the stuff that was being done on /levi (jenkins for
>> Cython) be moved to /scratch, which is a reasonably fast NFS
>> filesystem.
>
> The whole Jenkins installation was on /levi (maybe not our best idea
> ever...). I have backups of the complete configuration, but we'd still
> loose the installed plugins and the build history. We'd literally start
> from scratch and it would take some time to set it all up again. If it
> turns out to be a matter of replugging the USB drive (or something
> similarly simple), I would prefer copying the original installation over
> (including the build history), rather than rebuilding it.
>
> Also, the sage build directory was on levi and I don't know how that was
> set up - Robert would know.

It was just a vanilla copy of Sage + some patches. (I don't know if
the patches were saved elsewhere :(...)

- Robert

From vitja.makarov at gmail.com  Thu Mar 22 17:12:29 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 22 Mar 2012 20:12:29 +0400
Subject: [Cython] sage.math problems?
In-Reply-To: <4F6B4D6A.9080103@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de>
Message-ID: <CAKGHGPQcTnYB8XytRHq8BN3vvQE61yQqz0RVtCN6zxPAJ=9tUg@mail.gmail.com>

2012/3/22 Stefan Behnel <stefan_ml at behnel.de>:
> William Stein, 22.03.2012 14:11:
>> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby wrote:
>>> On 22 March 2012 12:33, William Stein wrote:
>>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>>> Physical access is thus probably required. ?I've emailed some people who are in
>>>> Seattle, and we'll see. ?(I'm thousands of miles away now, and will
>>>> only get further away
>>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>>
>>> I believe any decent IT professional will tell you that you should not
>>> be using USB drives on servers.
>>
>> Well I'm definitely currently not using USB drives on this server. :-(
>>
>> If possible, can the stuff that was being done on /levi (jenkins for
>> Cython) be moved to /scratch, which is a reasonably fast NFS
>> filesystem.
>
> The whole Jenkins installation was on /levi (maybe not our best idea
> ever...). I have backups of the complete configuration, but we'd still
> loose the installed plugins and the build history. We'd literally start
> from scratch and it would take some time to set it all up again. If it
> turns out to be a matter of replugging the USB drive (or something
> similarly simple), I would prefer copying the original installation over
> (including the build history), rather than rebuilding it.
>

Do you have sudo on that machine? Does the disk show up when you run lsusb?
If it doesn't reboot may help. Anyway, can you send me dmesg output?


> Also, the sage build directory was on levi and I don't know how that was
> set up - Robert would know.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
vitja.

From stefan_ml at behnel.de  Thu Mar 22 17:43:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 17:43:51 +0100
Subject: [Cython] sage.math problems?
In-Reply-To: <CAKGHGPQcTnYB8XytRHq8BN3vvQE61yQqz0RVtCN6zxPAJ=9tUg@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de>
	<CAKGHGPQcTnYB8XytRHq8BN3vvQE61yQqz0RVtCN6zxPAJ=9tUg@mail.gmail.com>
Message-ID: <4F6B56C7.90303@behnel.de>

Vitja Makarov, 22.03.2012 17:12:
> Does the disk show up when you run lsusb?

No.

> If it doesn't reboot may help. Anyway, can you send me dmesg output?

It's not in dmesg either. Maybe it just switched itself off after the power
failure?

Stefan

From d.s.seljebotn at astro.uio.no  Thu Mar 22 17:58:29 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 09:58:29 -0700
Subject: [Cython] sage.math problems?
In-Reply-To: <4F6B4D6A.9080103@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de>
Message-ID: <4F6B5A35.8000401@astro.uio.no>

On 03/22/2012 09:03 AM, Stefan Behnel wrote:
> William Stein, 22.03.2012 14:11:
>> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby wrote:
>>> On 22 March 2012 12:33, William Stein wrote:
>>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>>> Physical access is thus probably required.  I've emailed some people who are in
>>>> Seattle, and we'll see.  (I'm thousands of miles away now, and will
>>>> only get further away
>>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>>
>>> I believe any decent IT professional will tell you that you should not
>>> be using USB drives on servers.
>>
>> Well I'm definitely currently not using USB drives on this server. :-(
>>
>> If possible, can the stuff that was being done on /levi (jenkins for
>> Cython) be moved to /scratch, which is a reasonably fast NFS
>> filesystem.
>
> The whole Jenkins installation was on /levi (maybe not our best idea
> ever...). I have backups of the complete configuration, but we'd still
> loose the installed plugins and the build history. We'd literally start
> from scratch and it would take some time to set it all up again. If it
> turns out to be a matter of replugging the USB drive (or something
> similarly simple), I would prefer copying the original installation over
> (including the build history), rather than rebuilding it.
>

I hope it doesn't come to this, but if it does, I think we should really 
look hard at ShiningPanda instead.

Honestly, my feeling is that if we can't rally up $240/month in funding 
among Cython users then we might as well give up. (And specifically, we 
could ask NumFOCUS first, and possibly share the queue with other open 
source projects on their lists, and possibly ask ShiningPanda for an OSS 
project rebate).

I'm not primarily concerned with uptime, but with the time spent in 
maintaining the systems. The more that can be moved into the cloud the 
better for open source projects, if it means less maintenance, which is 
usually the case; see our move to GitHub.

ShiningPanda also offers features like testing on Windows.

Dag

From stefan_ml at behnel.de  Thu Mar 22 18:52:13 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 18:52:13 +0100
Subject: [Cython] funding (Re:  sage.math problems?)
In-Reply-To: <4F6B5A35.8000401@astro.uio.no>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
Message-ID: <4F6B66CD.7060409@behnel.de>

Dag Sverre Seljebotn, 22.03.2012 17:58:
> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>> I would prefer copying the original installation over
>> (including the build history), rather than rebuilding it.
> 
> I hope it doesn't come to this, but if it does, I think we should really
> look hard at ShiningPanda instead.

We could set up an OSS test account to see what we'd get for our money,
i.e. how much of our build/test cycle we can put into one hour in their
environment.


> Honestly, my feeling is that if we can't rally up $240/month in funding
> among Cython users then we might as well give up.

As long as we have sage.math, I can think of better things to do with 240$
per month. Didn't we want to organise another workshop at some point?

Regarding funding in general, maybe we should just start putting up one or
two of those sexy funding bars on our web site, like the PyPy devs do for
their funded projects. Assuming that goes well, it would also allow us to
put money on dedicated projects by paying basically ourselves for doing
tasks that we won't normally spend our precious spare time on (e.g. because
they appear too large for a weekend), but that we and our users deem
necessary for some reason.

Basically, any "real" CEP that we consider doable and that we'd have a
developer for could get a funding account where users could "vote" for it
by donating money.

(and that's where the legal issues start ...)


> (And specifically, we
> could ask NumFOCUS first, and possibly share the queue with other open
> source projects on their lists, and possibly ask ShiningPanda for an OSS
> project rebate).

Sure.


> I'm not primarily concerned with uptime, but with the time spent in
> maintaining the systems. The more that can be moved into the cloud the
> better for open source projects, if it means less maintenance, which is
> usually the case; see our move to GitHub.

I don't know how much time the maintenance takes on UW side, but they're
using the machines for many other things, so I guess they'd have to invest
the time anyway. So they won't win much by us moving out.

Speaking for myself, I don't consider the time wasted that I invested into
the Jenkins setup so far, and I'm also not sure there'd be all that much to
gain by no longer administrating the server installation itself by
ourselves. The bulk of the work is about configuring jobs and writing
build/test/whatever scripts, which still applies to a cloud installation
(with, I assume, the added disadvantage of no longer being able to ssh
directly into the machine).


> ShiningPanda also offers features like testing on Windows.

That *is* a feature, but it also takes up additional (paid) time. We
wouldn't have to run continuous tests on it, though, just trigger tests
manually when we want them.

Stefan

From wstein at gmail.com  Thu Mar 22 19:04:52 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 14:04:52 -0400
Subject: [Cython] sage.math problems?
In-Reply-To: <4F6B4D6A.9080103@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de>
Message-ID: <CACLE5GByRFkAq04P=3m8-NGyVxb2=QoKU+ctQi+KpdK1385sKg@mail.gmail.com>

On Thu, Mar 22, 2012 at 12:03 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> William Stein, 22.03.2012 14:11:
>> On Thu, Mar 22, 2012 at 9:04 AM, David Kirkby wrote:
>>> On 22 March 2012 12:33, William Stein wrote:
>>>> The usb device (for /levi) never appeared after I remotely rebooted sage.math.
>>>> Physical access is thus probably required. ?I've emailed some people who are in
>>>> Seattle, and we'll see. ?(I'm thousands of miles away now, and will
>>>> only get further away
>>>> -- to Europe -- until Apr 2, when I'll be back in Seattle.)
>>>
>>> I believe any decent IT professional will tell you that you should not
>>> be using USB drives on servers.
>>
>> Well I'm definitely currently not using USB drives on this server. :-(
>>
>> If possible, can the stuff that was being done on /levi (jenkins for
>> Cython) be moved to /scratch, which is a reasonably fast NFS
>> filesystem.
>
> The whole Jenkins installation was on /levi (maybe not our best idea
> ever...). I have backups of the complete configuration, but we'd still
> loose the installed plugins and the build history. We'd literally start
> from scratch and it would take some time to set it all up again. If it
> turns out to be a matter of replugging the USB drive (or something
> similarly simple), I would prefer copying the original installation over
> (including the build history), rather than rebuilding it.

OK.  I'm sure it is just a matter of replugging the USB drive.

>
> Also, the sage build directory was on levi and I don't know how that was
> set up - Robert would know.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From stefan_ml at behnel.de  Thu Mar 22 19:10:10 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 19:10:10 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F68C3B2.7020206@behnel.de>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
Message-ID: <4F6B6B02.4040205@behnel.de>

Stefan Behnel, 20.03.2012 18:51:
> mark florisson, 20.03.2012 17:40:
>> As for the two pull requests, both are quite large, but the dict
>> iteration rewrite is more like an enhancement whereas the 'yield from'
>> is really a new (somewhat big) feature. I would personally prefer to
>> merge only enhancements and bug fixes, but not new features.
> 
> Funny. I would have proposed the opposite, because I consider the "yield
> from" implementation (as a new language feature) safer than the dict
> iteration (which impacts existing code). As I said, I wouldn't mind
> postponing both to 0.16.1, but I think it would be better to get them out
> and have people use them, so that we can also fix the bugs earlier.

Coming back to this, I think the main question is: what does our versioning
scheme mean? Personally, I would expect a 0.16 release to come with new
features compared to 0.15.x, and a 0.16.1 release to fix the bugs that
these features (and other changes) have introduced, without changing too
much of what makes up the 0.16 release series.

I know that we've rarely followed that distinction (we sometimes didn't
even have point releases), so it doesn't really matter what we add in
0.16.1, I guess. I just thought I'd bring up the question what would make a
0.17 release different from a 0.16.1 release. So far, it appears to have
been more of an issue of how much time has passed since the last release,
which usually goes hand in hand with a substantial set of changes.

Stefan

From wstein at gmail.com  Thu Mar 22 19:11:47 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 14:11:47 -0400
Subject: [Cython] Jenkins down
In-Reply-To: <CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
	<CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
Message-ID: <CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>

On Thu, Mar 22, 2012 at 11:59 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
> Thanks for the offer. One problem I see is that /levi/scratch (a local
> disk) has been dismounted (or reconfigured). To save ourselves random
> permissions issues, it's probably best to run this script/job as
> scoder (I've got sudo).
>
> I'll see what I can do to get it running, assuming the sage.math
> issues have been resolved.

David Roe is going to unplug/replug the disk.

 -- William

>
> - Robert
>
> On Thu, Mar 22, 2012 at 5:46 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Hi,
>>>
>>> it seems like the sage.math server is broken again, at least it's lacking
>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>
>>> Stefan
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Logging in through ssh seems to work, should I run any of the scripts
>> from your home directory Stefan? e.g. restart_dead_hudson.sh or
>> starthudson.sh?
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From d.s.seljebotn at astro.uio.no  Thu Mar 22 19:21:08 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 11:21:08 -0700
Subject: [Cython] funding (Re:  sage.math problems?)
In-Reply-To: <4F6B66CD.7060409@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
Message-ID: <4F6B6D94.8070300@astro.uio.no>

On 03/22/2012 10:52 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 22.03.2012 17:58:
>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>> I would prefer copying the original installation over
>>> (including the build history), rather than rebuilding it.
>>
>> I hope it doesn't come to this, but if it does, I think we should really
>> look hard at ShiningPanda instead.
>
> We could set up an OSS test account to see what we'd get for our money,
> i.e. how much of our build/test cycle we can put into one hour in their
> environment.
>
>
>> Honestly, my feeling is that if we can't rally up $240/month in funding
>> among Cython users then we might as well give up.
>
> As long as we have sage.math, I can think of better things to do with 240$
> per month. Didn't we want to organise another workshop at some point?

I honestly don't think there's a lack of money for things like this once 
we go around asking; the most expensive part of any workshop is the time 
of the participants.

Workshops in particular may be fundable through William's grant. The 
main problem is when people are available. Robert will not be available 
for some time, and neither will I (though I'm much less important these 
days), but if the rest would like to meet up I'm sure it is fundable one 
way or the other.

> Regarding funding in general, maybe we should just start putting up one or
> two of those sexy funding bars on our web site, like the PyPy devs do for
> their funded projects. Assuming that goes well, it would also allow us to
> put money on dedicated projects by paying basically ourselves for doing
> tasks that we won't normally spend our precious spare time on (e.g. because
> they appear too large for a weekend), but that we and our users deem
> necessary for some reason.

Well, outright funding for projects is of a totally different order of 
magnitude. The $240 doesn't pay for more than between 1 and 4 hours of 
"grown-up" developer time (at least in Western Europe and US); for 
serious feature-funding you're starting to need thousands or 
tens-of-thousands of dollars rather than hundreds.

Smaller constant-amount bounties (like GSoC) for fun stuff one would 
have motivation to do otherwise is a different matter. My impression is 
that a) core devs have no more time for Cython than they spend already, 
b) Cython development is a bit too difficult to enter for random 
"bounty-hunters".

Those are just a couple of minor concerns -- I'm not saying we shouldn't 
do it!

Slightly related: I believe the best thing we can do to attract more 
developers is to seriously clean up the codebase. My new year's 
resolution is that if I get some days for working on Cython this year (I 
hope to), I'll spend it only on cleaning up the codebase, not on (even 
simple) features.

> Basically, any "real" CEP that we consider doable and that we'd have a
> developer for could get a funding account where users could "vote" for it
> by donating money.
>
> (and that's where the legal issues start ...)

Well, I seem to remember from a talk that NumFOCUS will have a full-time 
(or part-time?) position to deal with such administration. And they'll 
be set up as a non-profit (so tax-exempt for US-based donations). So I 
think that's a better route than PayPal.

Should we solicit donations on our webpage with a link to NumFOCUS? (I 
can ask NumFOCUS whether they're cool with that.)

> Speaking for myself, I don't consider the time wasted that I invested into
> the Jenkins setup so far, and I'm also not sure there'd be all that much to
> gain by no longer administrating the server installation itself by
> ourselves. The bulk of the work is about configuring jobs and writing
> build/test/whatever scripts, which still applies to a cloud installation
> (with, I assume, the added disadvantage of no longer being able to ssh
> directly into the machine).

There is SSH access at least to the environment where the tests are run, 
according to their web pages.

>
>
>> ShiningPanda also offers features like testing on Windows.
>
> That *is* a feature, but it also takes up additional (paid) time. We
> wouldn't have to run continuous tests on it, though, just trigger tests
> manually when we want them.

And it's in beta so far. But it's a feature sage.math will never get.

Dag

From d.s.seljebotn at astro.uio.no  Thu Mar 22 19:25:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 11:25:47 -0700
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CANg26EU-RSasy1nTnkO-QRGw00hHTOZHn4r4O51pAGKPAYNQPg@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
	<CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
	<4F6AD22F.6040602@behnel.de>
	<CANg26EU-RSasy1nTnkO-QRGw00hHTOZHn4r4O51pAGKPAYNQPg@mail.gmail.com>
Message-ID: <4F6B6EAB.2020606@astro.uio.no>

On 03/22/2012 05:40 AM, mark florisson wrote:
> On 22 March 2012 07:18, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Robert Bradshaw, 21.03.2012 21:24:
>>> On Wed, Mar 21, 2012 at 6:47 AM, mark florisson wrote:
>>>> On 20 March 2012 18:51, Stefan Behnel wrote:
>>>>> mark florisson, 20.03.2012 17:40:
>>>>>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>>>>>> I put up two new pull requests on github:
>>>>>>>
>>>>>>> Implementation of PEP 380 (yield from):
>>>>>>> https://github.com/cython/cython/pull/96
>>>>>>>
>>>>>>> Rewrite of dict iteration:
>>>>>>> https://github.com/cython/cython/pull/95
>>>>>>>
>>>>>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>>>>>> that these are really nice features, could someone (and especially Mark, as
>>>>>>> the responsible release manager) take a look at them and give an opinion
>>>>>>> regarding their suitability for 0.16?
>>>>>>
>>>>>> Sorry about the slowdown, my BT internet connection at home got closed
>>>>>> off, so working on a release is rather hard, hopefully that will be
>>>>>> fixed within the next few days.
>>>
>>> Thanks for the status update. Is there a separate release branch, or
>>> are you working off the github master?
>>
>> Not currently. I also agree that we should try to get out the release soon.
>> What about this: I merge my pending pull requests, we test everything in
>> Sage (it's been normally tested in my dev branch already), and if that
>> doesn't break anything, we release another beta? If it does break anything,
>> it's either easy enough to fix, or we can create a release branch at that
>> point with the status before the merges.
>>
>> sage.math seems to be back up, although the /levi drive isn't. If there are
>> no objections, I'll make sure Jenkins gets back up and then do the above.
>>
>>
>>> Is there a list of issues/blockers somewhere?
>>
>> Don't know.
>>
>
> Mostly the pull requests and this reversed bug
> (http://trac.cython.org/cython_trac/ticket/763), which I marked as a
> blocker. As much as I would like to push stuff out early, I really
> find it more important to push something that is as stable and useful
> as possible, even if it delays a release by a couple of weeks.

IMO, the ideal would be to release 0.16 now and 0.16.1 in a couple of 
weeks, but I respect that this depends on how much more additional work 
the release manager feels that involves.

I agree that we should wait for the patch that was started on for that 
bug, bringing in new developers is the most important thing we can do 
and satisfying users who don't live on trunk is less important than that.

Dag

From wstein at gmail.com  Thu Mar 22 19:31:36 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 14:31:36 -0400
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B6D94.8070300@astro.uio.no>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
Message-ID: <CACLE5GD=npduVHB4LiEZ3tkq313eXD7QiXxSqCFKq-h9v1t_NA@mail.gmail.com>

On Thu, Mar 22, 2012 at 2:21 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 03/22/2012 10:52 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 22.03.2012 17:58:
>>>
>>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>>>
>>>> I would prefer copying the original installation over
>>>> (including the build history), rather than rebuilding it.
>>>
>>>
>>> I hope it doesn't come to this, but if it does, I think we should really
>>> look hard at ShiningPanda instead.
>>
>>
>> We could set up an OSS test account to see what we'd get for our money,
>> i.e. how much of our build/test cycle we can put into one hour in their
>> environment.
>>
>>
>>> Honestly, my feeling is that if we can't rally up $240/month in funding
>>> among Cython users then we might as well give up.
>>
>>
>> As long as we have sage.math, I can think of better things to do with 240$
>> per month. Didn't we want to organise another workshop at some point?
>
>
> I honestly don't think there's a lack of money for things like this once we
> go around asking; the most expensive part of any workshop is the time of the
> participants.
>
> Workshops in particular may be fundable through William's grant. The main
> problem is when people are available. Robert will not be available for some
> time, and neither will I (though I'm much less important these days), but if
> the rest would like to meet up I'm sure it is fundable one way or the other.

True.

>>
>>> ShiningPanda also offers features like testing on Windows.
>>
>>
>> That *is* a feature, but it also takes up additional (paid) time. We
>> wouldn't have to run continuous tests on it, though, just trigger tests
>> manually when we want them.
>
>
> And it's in beta so far. But it's a feature sage.math will never get.

Well, I could let you start a Windows Virtual Machine, and you could
connect via VNC...

>
> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From wstein at gmail.com  Thu Mar 22 19:38:02 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 14:38:02 -0400
Subject: [Cython] Jenkins down
In-Reply-To: <CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
	<CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
	<CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>
Message-ID: <CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>

Hi,

/levi is back on sage.math.

If you use it, I encourage you to continue doing so.  However, please
setup a crontab'd rsync backup of
anything important on there to your /home directory.

 -- William

On Thu, Mar 22, 2012 at 2:11 PM, William Stein <wstein at gmail.com> wrote:
> On Thu, Mar 22, 2012 at 11:59 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
>> Thanks for the offer. One problem I see is that /levi/scratch (a local
>> disk) has been dismounted (or reconfigured). To save ourselves random
>> permissions issues, it's probably best to run this script/job as
>> scoder (I've got sudo).
>>
>> I'll see what I can do to get it running, assuming the sage.math
>> issues have been resolved.
>
> David Roe is going to unplug/replug the disk.
>
> ?-- William
>
>>
>> - Robert
>>
>> On Thu, Mar 22, 2012 at 5:46 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Hi,
>>>>
>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>>
>>>> Stefan
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Logging in through ssh seems to work, should I run any of the scripts
>>> from your home directory Stefan? e.g. restart_dead_hudson.sh or
>>> starthudson.sh?
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> --
> William Stein
> Professor of Mathematics
> University of Washington
> http://wstein.org



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From robertwb at gmail.com  Thu Mar 22 19:39:59 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 11:39:59 -0700
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B66CD.7060409@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
Message-ID: <CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>

On Thu, Mar 22, 2012 at 10:52 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 22.03.2012 17:58:
>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>> I would prefer copying the original installation over
>>> (including the build history), rather than rebuilding it.
>>
>> I hope it doesn't come to this, but if it does, I think we should really
>> look hard at ShiningPanda instead.
>
> We could set up an OSS test account to see what we'd get for our money,
> i.e. how much of our build/test cycle we can put into one hour in their
> environment.
>
>
>> Honestly, my feeling is that if we can't rally up $240/month in funding
>> among Cython users then we might as well give up.
>
> As long as we have sage.math, I can think of better things to do with 240$
> per month. Didn't we want to organise another workshop at some point?

+1, this would be a much more effective use of funding, though also
easier to get funding for.

> Regarding funding in general, maybe we should just start putting up one or
> two of those sexy funding bars on our web site, like the PyPy devs do for
> their funded projects. Assuming that goes well, it would also allow us to
> put money on dedicated projects by paying basically ourselves for doing
> tasks that we won't normally spend our precious spare time on (e.g. because
> they appear too large for a weekend), but that we and our users deem
> necessary for some reason.

While that's a good idea in theory, I'm not sure how many additional
hours would be freed up just because we could pay ourselves for it.
Perhaps it would act primarily as an additional incentive to align our
efforts with user request (though there are certainly already
non-monetary incentives). A budget of a couple hundred a month is
likely an order of magnitude too small to get as serious developer to
attack "larger than can be done in a weekend" projects. Also, the
monetization of Cython development changes the spirit of things a bit,
and while I am a big fan of people being able to make money, or even a
living, off of open source development, it complicates things a lot
from a legal, financial, and political perspective.

The current model of organization X is willing to pay developer Y for
feature Z directly seems to work well enough for the moment. E.g. with
GSoC, the bottleneck is finding good enough students and time to
mentor them, not slots (=funding). Opening up funding to non-students
could help a bit, but IMHO wouldn't change the balance that much (the
gainfully employed cost a lot more and have less spare time).

> Basically, any "real" CEP that we consider doable and that we'd have a
> developer for could get a funding account where users could "vote" for it
> by donating money.
>
> (and that's where the legal issues start ...)
>
>
>> (And specifically, we
>> could ask NumFOCUS first, and possibly share the queue with other open
>> source projects on their lists, and possibly ask ShiningPanda for an OSS
>> project rebate).
>
> Sure.
>
>
>> I'm not primarily concerned with uptime, but with the time spent in
>> maintaining the systems. The more that can be moved into the cloud the
>> better for open source projects, if it means less maintenance, which is
>> usually the case; see our move to GitHub.
>
> I don't know how much time the maintenance takes on UW side, but they're
> using the machines for many other things, so I guess they'd have to invest
> the time anyway. So they won't win much by us moving out.

Yes. We add to the load a bit, but not much. And I don't see us saving
administration time (the most lacking resource here) by moving to
another service (if anything, it'd likely be more work).

> Speaking for myself, I don't consider the time wasted that I invested into
> the Jenkins setup so far, and I'm also not sure there'd be all that much to
> gain by no longer administrating the server installation itself by
> ourselves. The bulk of the work is about configuring jobs and writing
> build/test/whatever scripts, which still applies to a cloud installation
> (with, I assume, the added disadvantage of no longer being able to ssh
> directly into the machine).
>
>
>> ShiningPanda also offers features like testing on Windows.
>
> That *is* a feature, but it also takes up additional (paid) time. We
> wouldn't have to run continuous tests on it, though, just trigger tests
> manually when we want them.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Thu Mar 22 19:45:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 19:45:43 +0100
Subject: [Cython] Jenkins down
In-Reply-To: <CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
	<CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
	<CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>
	<CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>
Message-ID: <4F6B7357.6000606@behnel.de>

William Stein, 22.03.2012 19:38:
> /levi is back on sage.math.
> 
> If you use it, I encourage you to continue doing so.  However, please
> setup a crontab'd rsync backup of
> anything important on there to your /home directory.

Thanks!

I'm on it.

Stefan


From robertwb at gmail.com  Thu Mar 22 19:45:45 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 11:45:45 -0700
Subject: [Cython] Jenkins down
In-Reply-To: <CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
	<CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
	<CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>
	<CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>
Message-ID: <CADiQ+QAL4Rq2SO6Bra6CdaysxG0jeBY5oSWbFShP9mew5GTAWw@mail.gmail.com>

Thanks! I've restarted hudson/jenkins.
https://sage.math.washington.edu:8091/hudson/

Lets try to put all the configuration/persistant data into someone's
home directory, and use /levi/scratch for the workspaces only.

William, is /levi/scratch the best place for file-IO intensive work? I
think it should be slightly more persistent than /tmp (though we
should look into whether that could be good enough).

- Robert


On Thu, Mar 22, 2012 at 11:38 AM, William Stein <wstein at gmail.com> wrote:
> Hi,
>
> /levi is back on sage.math.
>
> If you use it, I encourage you to continue doing so. ?However, please
> setup a crontab'd rsync backup of
> anything important on there to your /home directory.
>
> ?-- William
>
> On Thu, Mar 22, 2012 at 2:11 PM, William Stein <wstein at gmail.com> wrote:
>> On Thu, Mar 22, 2012 at 11:59 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>> Thanks for the offer. One problem I see is that /levi/scratch (a local
>>> disk) has been dismounted (or reconfigured). To save ourselves random
>>> permissions issues, it's probably best to run this script/job as
>>> scoder (I've got sudo).
>>>
>>> I'll see what I can do to get it running, assuming the sage.math
>>> issues have been resolved.
>>
>> David Roe is going to unplug/replug the disk.
>>
>> ?-- William
>>
>>>
>>> - Robert
>>>
>>> On Thu, Mar 22, 2012 at 5:46 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>> Hi,
>>>>>
>>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>>>
>>>>> Stefan
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>> Logging in through ssh seems to work, should I run any of the scripts
>>>> from your home directory Stefan? e.g. restart_dead_hudson.sh or
>>>> starthudson.sh?
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>>
>> --
>> William Stein
>> Professor of Mathematics
>> University of Washington
>> http://wstein.org
>
>
>
> --
> William Stein
> Professor of Mathematics
> University of Washington
> http://wstein.org
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Thu Mar 22 19:50:12 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 22 Mar 2012 18:50:12 +0000
Subject: [Cython] dynamically compile and import pure python modules
Message-ID: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>

Hey,

For the fused type runtime dispatch I found it very convenient to use
the with statement, but that is not supported in Python 2.4. However,
the compiler could dynamically compile compiler code with the compiler
itself and import it (pyximport), if it is not needed to compile
Cython itself. I gave it a try and it seems to work like a charm (but
probably needs more testing :):
https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2

It seems like a bit of a hack, so I thought I'd bring it up here and
ask for opinions, i.e. ask if anyone thinks it is a terrible idea.

Mark

From wstein at gmail.com  Thu Mar 22 20:02:44 2012
From: wstein at gmail.com (William Stein)
Date: Thu, 22 Mar 2012 15:02:44 -0400
Subject: [Cython] Jenkins down
In-Reply-To: <CADiQ+QAL4Rq2SO6Bra6CdaysxG0jeBY5oSWbFShP9mew5GTAWw@mail.gmail.com>
References: <4F69DE9A.6020100@behnel.de>
	<CANg26EUedLNVqSuzdKa-6CAPEuO38CTn49FdQd_Jup71JR6i0g@mail.gmail.com>
	<CADiQ+QALULLpAjBbOPPsGzp0LaXJWtivhda4jThAiUGQUxYuTw@mail.gmail.com>
	<CACLE5GBSqAhDMkFR_gg1AEGR7BsXYdb1N-OdDjN0hDTnrXAt-w@mail.gmail.com>
	<CACLE5GBGnZ_Hy3abtBHDuqPWG8s3gJVg4k+dW7c4ZHeP78Okaw@mail.gmail.com>
	<CADiQ+QAL4Rq2SO6Bra6CdaysxG0jeBY5oSWbFShP9mew5GTAWw@mail.gmail.com>
Message-ID: <CACLE5GCVNavw5MgHsyxeCA4dd=zgTQwnzUvQdSM62N-ryrbySQ@mail.gmail.com>

On Thu, Mar 22, 2012 at 2:45 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> Thanks! I've restarted hudson/jenkins.
> https://sage.math.washington.edu:8091/hudson/
>
> Lets try to put all the configuration/persistant data into someone's
> home directory, and use /levi/scratch for the workspaces only.
>
> William, is /levi/scratch the best place for file-IO intensive work? I
> think it should be slightly more persistent than /tmp (though we
> should look into whether that could be good enough).

Yes.  Use /tmp if you're not using too much disk space -- it's a RAM disk!

>
> - Robert
>
>
> On Thu, Mar 22, 2012 at 11:38 AM, William Stein <wstein at gmail.com> wrote:
>> Hi,
>>
>> /levi is back on sage.math.
>>
>> If you use it, I encourage you to continue doing so. ?However, please
>> setup a crontab'd rsync backup of
>> anything important on there to your /home directory.
>>
>> ?-- William
>>
>> On Thu, Mar 22, 2012 at 2:11 PM, William Stein <wstein at gmail.com> wrote:
>>> On Thu, Mar 22, 2012 at 11:59 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>> Thanks for the offer. One problem I see is that /levi/scratch (a local
>>>> disk) has been dismounted (or reconfigured). To save ourselves random
>>>> permissions issues, it's probably best to run this script/job as
>>>> scoder (I've got sudo).
>>>>
>>>> I'll see what I can do to get it running, assuming the sage.math
>>>> issues have been resolved.
>>>
>>> David Roe is going to unplug/replug the disk.
>>>
>>> ?-- William
>>>
>>>>
>>>> - Robert
>>>>
>>>> On Thu, Mar 22, 2012 at 5:46 AM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 21 March 2012 13:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> it seems like the sage.math server is broken again, at least it's lacking
>>>>>> mounts when I try to log in. I'll restart Jenkins as soon as it's back working.
>>>>>>
>>>>>> Stefan
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>> Logging in through ssh seems to work, should I run any of the scripts
>>>>> from your home directory Stefan? e.g. restart_dead_hudson.sh or
>>>>> starthudson.sh?
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>>
>>> --
>>> William Stein
>>> Professor of Mathematics
>>> University of Washington
>>> http://wstein.org
>>
>>
>>
>> --
>> William Stein
>> Professor of Mathematics
>> University of Washington
>> http://wstein.org
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

From stefan_ml at behnel.de  Thu Mar 22 20:03:23 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 20:03:23 +0100
Subject: [Cython] funding (Re:  sage.math problems?)
In-Reply-To: <4F6B6D94.8070300@astro.uio.no>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
Message-ID: <4F6B777B.1090709@behnel.de>

Dag Sverre Seljebotn, 22.03.2012 19:21:
> On 03/22/2012 10:52 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 22.03.2012 17:58:
>>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>>> I would prefer copying the original installation over
>>>> (including the build history), rather than rebuilding it.
>>>
>>> I hope it doesn't come to this, but if it does, I think we should really
>>> look hard at ShiningPanda instead.
>>
>> We could set up an OSS test account to see what we'd get for our money,
>> i.e. how much of our build/test cycle we can put into one hour in their
>> environment.
>>
>>> Honestly, my feeling is that if we can't rally up $240/month in funding
>>> among Cython users then we might as well give up.
>>
>> As long as we have sage.math, I can think of better things to do with 240$
>> per month. Didn't we want to organise another workshop at some point?
> 
> I honestly don't think there's a lack of money for things like this once we
> go around asking; the most expensive part of any workshop is the time of
> the participants.
> 
> Workshops in particular may be fundable through William's grant. The main
> problem is when people are available. Robert will not be available for some
> time, and neither will I (though I'm much less important these days), but
> if the rest would like to meet up I'm sure it is fundable one way or the
> other.

Yep, sounds like a time rather than money issue.


>> Regarding funding in general, maybe we should just start putting up one or
>> two of those sexy funding bars on our web site, like the PyPy devs do for
>> their funded projects. Assuming that goes well, it would also allow us to
>> put money on dedicated projects by paying basically ourselves for doing
>> tasks that we won't normally spend our precious spare time on (e.g. because
>> they appear too large for a weekend), but that we and our users deem
>> necessary for some reason.
> 
> Well, outright funding for projects is of a totally different order of
> magnitude. The $240 doesn't pay for more than between 1 and 4 hours of
> "grown-up" developer time (at least in Western Europe and US); for serious
> feature-funding you're starting to need thousands or tens-of-thousands of
> dollars rather than hundreds.

PyPy manages to get those amounts. Just look at their home page, the
projects are all in the tens of thousands of dollars. I don't think that's
impossible for us.


> Smaller constant-amount bounties (like GSoC) for fun stuff one would have
> motivation to do otherwise is a different matter. My impression is that a)
> core devs have no more time for Cython than they spend already, b) Cython
> development is a bit too difficult to enter for random "bounty-hunters".

Regarding b), PyPy is far worse.

Regarding a), I'm not so sure. At least I would consider it a way to focus
my work. Currently, I'm rather reluctant to starting anything that looks
like more than a couple of evenings or a weekend.


> Slightly related: I believe the best thing we can do to attract more
> developers is to seriously clean up the codebase. My new year's resolution
> is that if I get some days for working on Cython this year (I hope to),
> I'll spend it only on cleaning up the codebase, not on (even simple) features.

Yay!


>> Basically, any "real" CEP that we consider doable and that we'd have a
>> developer for could get a funding account where users could "vote" for it
>> by donating money.
>>
>> (and that's where the legal issues start ...)
> 
> Well, I seem to remember from a talk that NumFOCUS will have a full-time
> (or part-time?) position to deal with such administration. And they'll be
> set up as a non-profit (so tax-exempt for US-based donations). So I think
> that's a better route than PayPal.

Sounds like it. Is their tax-exempt status restricted to the US? That would
be unfortunate for many donators.


> Should we solicit donations on our webpage with a link to NumFOCUS? (I can
> ask NumFOCUS whether they're cool with that.)

Interesting. If they have the administration set up (BTW, is that paid or
do they want something for that?), they should be able to handle project
specific money as well.


>> Speaking for myself, I don't consider the time wasted that I invested into
>> the Jenkins setup so far, and I'm also not sure there'd be all that much to
>> gain by no longer administrating the server installation itself by
>> ourselves. The bulk of the work is about configuring jobs and writing
>> build/test/whatever scripts, which still applies to a cloud installation
>> (with, I assume, the added disadvantage of no longer being able to ssh
>> directly into the machine).
> 
> There is SSH access at least to the environment where the tests are run,
> according to their web pages.

Ok, that's cool.


>>> ShiningPanda also offers features like testing on Windows.
>>
>> That *is* a feature, but it also takes up additional (paid) time. We
>> wouldn't have to run continuous tests on it, though, just trigger tests
>> manually when we want them.
> 
> And it's in beta so far. But it's a feature sage.math will never get.

Sure. However, most code we write isn't platform specific, and having a CI
server run Windows tests doesn't always help in debugging them if you don't
have a Windows box to reproduce the problem. Their SSH access may help on
that one (assuming that you get a usable environment there, including
MinGW/gdb or Microsoft's compiler environment), but I should note that
setting up the Jenkins jobs for Windows basically means rewriting them
completely (and remotely). What do you even get when you ssh into their
Windows box? Not a DOS prompt, I hope?

Stefan

From d.s.seljebotn at astro.uio.no  Thu Mar 22 20:12:04 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 12:12:04 -0700
Subject: [Cython] dynamically compile and import pure python modules
In-Reply-To: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>
References: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>
Message-ID: <4F6B7984.3000400@astro.uio.no>

On 03/22/2012 11:50 AM, mark florisson wrote:
> Hey,
>
> For the fused type runtime dispatch I found it very convenient to use
> the with statement, but that is not supported in Python 2.4. However,
> the compiler could dynamically compile compiler code with the compiler
> itself and import it (pyximport), if it is not needed to compile
> Cython itself. I gave it a try and it seems to work like a charm (but
> probably needs more testing :):
> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2
>
> It seems like a bit of a hack, so I thought I'd bring it up here and
> ask for opinions, i.e. ask if anyone thinks it is a terrible idea.

How about dropping Python 2.4 support for running Cython (though 
generated C files still run)?

Dag

From d.s.seljebotn at astro.uio.no  Thu Mar 22 20:13:29 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 12:13:29 -0700
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
Message-ID: <4F6B79D9.7020307@astro.uio.no>

On 03/22/2012 11:39 AM, Robert Bradshaw wrote:
> On Thu, Mar 22, 2012 at 10:52 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Dag Sverre Seljebotn, 22.03.2012 17:58:
>>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>>> I would prefer copying the original installation over
>>>> (including the build history), rather than rebuilding it.
>>>
>>> I hope it doesn't come to this, but if it does, I think we should really
>>> look hard at ShiningPanda instead.
>>
>> We could set up an OSS test account to see what we'd get for our money,
>> i.e. how much of our build/test cycle we can put into one hour in their
>> environment.
>>
>>
>>> Honestly, my feeling is that if we can't rally up $240/month in funding
>>> among Cython users then we might as well give up.
>>
>> As long as we have sage.math, I can think of better things to do with 240$
>> per month. Didn't we want to organise another workshop at some point?
>
> +1, this would be a much more effective use of funding, though also
> easier to get funding for.
>
>> Regarding funding in general, maybe we should just start putting up one or
>> two of those sexy funding bars on our web site, like the PyPy devs do for
>> their funded projects. Assuming that goes well, it would also allow us to
>> put money on dedicated projects by paying basically ourselves for doing
>> tasks that we won't normally spend our precious spare time on (e.g. because
>> they appear too large for a weekend), but that we and our users deem
>> necessary for some reason.
>
> While that's a good idea in theory, I'm not sure how many additional
> hours would be freed up just because we could pay ourselves for it.
> Perhaps it would act primarily as an additional incentive to align our
> efforts with user request (though there are certainly already
> non-monetary incentives). A budget of a couple hundred a month is
> likely an order of magnitude too small to get as serious developer to
> attack "larger than can be done in a weekend" projects. Also, the
> monetization of Cython development changes the spirit of things a bit,
> and while I am a big fan of people being able to make money, or even a
> living, off of open source development, it complicates things a lot
> from a legal, financial, and political perspective.
>
> The current model of organization X is willing to pay developer Y for
> feature Z directly seems to work well enough for the moment. E.g. with
> GSoC, the bottleneck is finding good enough students and time to
> mentor them, not slots (=funding). Opening up funding to non-students
> could help a bit, but IMHO wouldn't change the balance that much (the
> gainfully employed cost a lot more and have less spare time).
>
>> Basically, any "real" CEP that we consider doable and that we'd have a
>> developer for could get a funding account where users could "vote" for it
>> by donating money.
>>
>> (and that's where the legal issues start ...)
>>
>>
>>> (And specifically, we
>>> could ask NumFOCUS first, and possibly share the queue with other open
>>> source projects on their lists, and possibly ask ShiningPanda for an OSS
>>> project rebate).
>>
>> Sure.
>>
>>
>>> I'm not primarily concerned with uptime, but with the time spent in
>>> maintaining the systems. The more that can be moved into the cloud the
>>> better for open source projects, if it means less maintenance, which is
>>> usually the case; see our move to GitHub.
>>
>> I don't know how much time the maintenance takes on UW side, but they're
>> using the machines for many other things, so I guess they'd have to invest
>> the time anyway. So they won't win much by us moving out.
>
> Yes. We add to the load a bit, but not much. And I don't see us saving
> administration time (the most lacking resource here) by moving to
> another service (if anything, it'd likely be more work).

OK. My comments were all predicated on the assumption saving 
administration time; and I totally trust yours and Stefan's judgement here.

Dag

From d.s.seljebotn at astro.uio.no  Thu Mar 22 20:14:55 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Mar 2012 12:14:55 -0700
Subject: [Cython] funding (Re:  sage.math problems?)
In-Reply-To: <4F6B777B.1090709@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
Message-ID: <4F6B7A2F.6050000@astro.uio.no>

On 03/22/2012 12:03 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 22.03.2012 19:21:
>> Slightly related: I believe the best thing we can do to attract more
>> developers is to seriously clean up the codebase. My new year's resolution
>> is that if I get some days for working on Cython this year (I hope to),
>> I'll spend it only on cleaning up the codebase, not on (even simple) features.
>
> Yay!

Granted, that's only because Mark now takes care of adding all the 
special numerical computing features to the language ;-)

Dag

From stefan_ml at behnel.de  Thu Mar 22 20:16:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 20:16:07 +0100
Subject: [Cython] dynamically compile and import pure python modules
In-Reply-To: <4F6B7984.3000400@astro.uio.no>
References: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>
	<4F6B7984.3000400@astro.uio.no>
Message-ID: <4F6B7A77.5050608@behnel.de>

Dag Sverre Seljebotn, 22.03.2012 20:12:
> On 03/22/2012 11:50 AM, mark florisson wrote:
>> For the fused type runtime dispatch I found it very convenient to use
>> the with statement, but that is not supported in Python 2.4. However,
>> the compiler could dynamically compile compiler code with the compiler
>> itself and import it (pyximport), if it is not needed to compile
>> Cython itself. I gave it a try and it seems to work like a charm (but
>> probably needs more testing :):
>> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2
>>
>>
>> It seems like a bit of a hack, so I thought I'd bring it up here and
>> ask for opinions, i.e. ask if anyone thinks it is a terrible idea.
> 
> How about dropping Python 2.4 support for running Cython (though generated
> C files still run)?

For one, it would make it trickier to run the test suite.

Stefan

From robertwb at gmail.com  Thu Mar 22 20:26:26 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 12:26:26 -0700
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B777B.1090709@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
Message-ID: <CADiQ+QC6ijbte5hJTLDN+Z=upJN5vYjVt8z=gMfCAg3PibwSNA@mail.gmail.com>

On Thu, Mar 22, 2012 at 12:03 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 22.03.2012 19:21:
>> On 03/22/2012 10:52 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 22.03.2012 17:58:
>>>> On 03/22/2012 09:03 AM, Stefan Behnel wrote:
>>>>> I would prefer copying the original installation over
>>>>> (including the build history), rather than rebuilding it.
>>>>
>>>> I hope it doesn't come to this, but if it does, I think we should really
>>>> look hard at ShiningPanda instead.
>>>
>>> We could set up an OSS test account to see what we'd get for our money,
>>> i.e. how much of our build/test cycle we can put into one hour in their
>>> environment.
>>>
>>>> Honestly, my feeling is that if we can't rally up $240/month in funding
>>>> among Cython users then we might as well give up.
>>>
>>> As long as we have sage.math, I can think of better things to do with 240$
>>> per month. Didn't we want to organise another workshop at some point?
>>
>> I honestly don't think there's a lack of money for things like this once we
>> go around asking; the most expensive part of any workshop is the time of
>> the participants.
>>
>> Workshops in particular may be fundable through William's grant. The main
>> problem is when people are available. Robert will not be available for some
>> time, and neither will I (though I'm much less important these days), but
>> if the rest would like to meet up I'm sure it is fundable one way or the
>> other.
>
> Yep, sounds like a time rather than money issue.
>
>
>>> Regarding funding in general, maybe we should just start putting up one or
>>> two of those sexy funding bars on our web site, like the PyPy devs do for
>>> their funded projects. Assuming that goes well, it would also allow us to
>>> put money on dedicated projects by paying basically ourselves for doing
>>> tasks that we won't normally spend our precious spare time on (e.g. because
>>> they appear too large for a weekend), but that we and our users deem
>>> necessary for some reason.
>>
>> Well, outright funding for projects is of a totally different order of
>> magnitude. The $240 doesn't pay for more than between 1 and 4 hours of
>> "grown-up" developer time (at least in Western Europe and US); for serious
>> feature-funding you're starting to need thousands or tens-of-thousands of
>> dollars rather than hundreds.
>
> PyPy manages to get those amounts. Just look at their home page, the
> projects are all in the tens of thousands of dollars. I don't think that's
> impossible for us.

Cool, that's promising. Do you think we would have the developer
resources to follow through if we did something like this?

>> Smaller constant-amount bounties (like GSoC) for fun stuff one would have
>> motivation to do otherwise is a different matter. My impression is that a)
>> core devs have no more time for Cython than they spend already, b) Cython
>> development is a bit too difficult to enter for random "bounty-hunters".
>
> Regarding b), PyPy is far worse.

Possibly.

> Regarding a), I'm not so sure. At least I would consider it a way to focus
> my work. Currently, I'm rather reluctant to starting anything that looks
> like more than a couple of evenings or a weekend.

Though the crazyiness in my personal life over the last 6+ months is
winding down, I doubt I'd have enough spare time to be, e.g., the
equivalent of 1/4-full time, or that funding would significantly
change the situation. But that's not an argument against not letting
you (and the rest of the project) benefit.

>> Slightly related: I believe the best thing we can do to attract more
>> developers is to seriously clean up the codebase. My new year's resolution
>> is that if I get some days for working on Cython this year (I hope to),
>> I'll spend it only on cleaning up the codebase, not on (even simple) features.
>
> Yay!

+100

>>> Basically, any "real" CEP that we consider doable and that we'd have a
>>> developer for could get a funding account where users could "vote" for it
>>> by donating money.
>>>
>>> (and that's where the legal issues start ...)
>>
>> Well, I seem to remember from a talk that NumFOCUS will have a full-time
>> (or part-time?) position to deal with such administration. And they'll be
>> set up as a non-profit (so tax-exempt for US-based donations). So I think
>> that's a better route than PayPal.
>
> Sounds like it. Is their tax-exempt status restricted to the US? That would
> be unfortunate for many donators.

Tax-exempt status is a criteria set by each individual government, not
by the entity. I'm sure if it helped money flow in they would consider
filing the necessary paperwork.

>> Should we solicit donations on our webpage with a link to NumFOCUS? (I can
>> ask NumFOCUS whether they're cool with that.)
>
> Interesting. If they have the administration set up (BTW, is that paid or
> do they want something for that?), they should be able to handle project
> specific money as well.
>
>
>>> Speaking for myself, I don't consider the time wasted that I invested into
>>> the Jenkins setup so far, and I'm also not sure there'd be all that much to
>>> gain by no longer administrating the server installation itself by
>>> ourselves. The bulk of the work is about configuring jobs and writing
>>> build/test/whatever scripts, which still applies to a cloud installation
>>> (with, I assume, the added disadvantage of no longer being able to ssh
>>> directly into the machine).
>>
>> There is SSH access at least to the environment where the tests are run,
>> according to their web pages.
>
> Ok, that's cool.
>
>
>>>> ShiningPanda also offers features like testing on Windows.
>>>
>>> That *is* a feature, but it also takes up additional (paid) time. We
>>> wouldn't have to run continuous tests on it, though, just trigger tests
>>> manually when we want them.
>>
>> And it's in beta so far. But it's a feature sage.math will never get.
>
> Sure. However, most code we write isn't platform specific, and having a CI
> server run Windows tests doesn't always help in debugging them if you don't
> have a Windows box to reproduce the problem. Their SSH access may help on
> that one (assuming that you get a usable environment there, including
> MinGW/gdb or Microsoft's compiler environment), but I should note that
> setting up the Jenkins jobs for Windows basically means rewriting them
> completely (and remotely). What do you even get when you ssh into their
> Windows box? Not a DOS prompt, I hope?

Even if we couldn't debug remotely, just having a continuous build
setup would be a huge boon for being able to locate the commits that
cause trouble rather than trying to debug a release candidate much
later.

- Robert

From jdh2358 at gmail.com  Thu Mar 22 20:39:57 2012
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 22 Mar 2012 14:39:57 -0500
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B777B.1090709@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
Message-ID: <CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>

On Thu, Mar 22, 2012 at 2:03 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

>
> > Well, I seem to remember from a talk that NumFOCUS will have a full-time
> > (or part-time?) position to deal with such administration. And they'll be
> > set up as a non-profit (so tax-exempt for US-based donations). So I think
> > that's a better route than PayPal.
>
> Sounds like it. Is their tax-exempt status restricted to the US? That would
> be unfortunate for many donators.
>

NUMFOCUS is in the process of trying to get 501c status, but does not yet
have it.  This will take some time.  This means individual donations will
not be tax exempt in the US until we have that status.  If businesses are
donating, they can deduct it as a business expense, but individuals are out
of luck for now.


>
> > Should we solicit donations on our webpage with a link to NumFOCUS? (I
> can
> > ask NumFOCUS whether they're cool with that.)
>
> Interesting. If they have the administration set up (BTW, is that paid or
> do they want something for that?), they should be able to handle project
> specific money as well.


NUMFOCUS does have a full time administrator, and will be happy to process
and distribute targeted donations for Cython.  This is at a "no cost to
you" basis, ie the foundation will not take overhead but will cover these
costs out of NUMFOCUS donations.  At least that is my understanding, but I
can confirm at the next director's meeting.

JDH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120322/f9127f47/attachment.html>

From stefan_ml at behnel.de  Thu Mar 22 20:49:17 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 20:49:17 +0100
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
	<CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>
Message-ID: <4F6B823D.4040007@behnel.de>

John Hunter, 22.03.2012 20:39:
> On Thu, Mar 22, 2012 at 2:03 PM, Stefan Behnel wrote:
>>> Well, I seem to remember from a talk that NumFOCUS will have a full-time
>>> (or part-time?) position to deal with such administration. And they'll be
>>> set up as a non-profit (so tax-exempt for US-based donations). So I think
>>> that's a better route than PayPal.
>>
>> Sounds like it. Is their tax-exempt status restricted to the US? That would
>> be unfortunate for many donators.
> 
> NUMFOCUS is in the process of trying to get 501c status, but does not yet
> have it.  This will take some time.  This means individual donations will
> not be tax exempt in the US until we have that status.  If businesses are
> donating, they can deduct it as a business expense, but individuals are out
> of luck for now.

Ok, just for clarification: the foundation is US based and not currently
targeting any other countries, right?

Personally, I think most serious donations would be from companies anyway,
which would then not be restricted to the US. And for individuals, we could
still put up a PayPal button (if we have anything to let them pay for).


>>> Should we solicit donations on our webpage with a link to NumFOCUS? (I
>>> can ask NumFOCUS whether they're cool with that.)
>>
>> Interesting. If they have the administration set up (BTW, is that paid or
>> do they want something for that?), they should be able to handle project
>> specific money as well.
> 
> NUMFOCUS does have a full time administrator, and will be happy to process
> and distribute targeted donations for Cython.  This is at a "no cost to
> you" basis, ie the foundation will not take overhead but will cover these
> costs out of NUMFOCUS donations.  At least that is my understanding, but I
> can confirm at the next director's meeting.

Cool. Then I'm all for taking that route.

Stefan

From robertwb at gmail.com  Thu Mar 22 20:54:56 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 12:54:56 -0700
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B823D.4040007@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
	<CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>
	<4F6B823D.4040007@behnel.de>
Message-ID: <CADiQ+QAqzbHhDK1C+fiug=-Gvvm0w_ECU5NQEeZct9esn_jrTg@mail.gmail.com>

On Thu, Mar 22, 2012 at 12:49 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> John Hunter, 22.03.2012 20:39:
>> On Thu, Mar 22, 2012 at 2:03 PM, Stefan Behnel wrote:
>>>> Well, I seem to remember from a talk that NumFOCUS will have a full-time
>>>> (or part-time?) position to deal with such administration. And they'll be
>>>> set up as a non-profit (so tax-exempt for US-based donations). So I think
>>>> that's a better route than PayPal.
>>>
>>> Sounds like it. Is their tax-exempt status restricted to the US? That would
>>> be unfortunate for many donators.
>>
>> NUMFOCUS is in the process of trying to get 501c status, but does not yet
>> have it. ?This will take some time. ?This means individual donations will
>> not be tax exempt in the US until we have that status. ?If businesses are
>> donating, they can deduct it as a business expense, but individuals are out
>> of luck for now.
>
> Ok, just for clarification: the foundation is US based and not currently
> targeting any other countries, right?
>
> Personally, I think most serious donations would be from companies anyway,
> which would then not be restricted to the US. And for individuals, we could
> still put up a PayPal button (if we have anything to let them pay for).
>
>
>>>> Should we solicit donations on our webpage with a link to NumFOCUS? (I
>>>> can ask NumFOCUS whether they're cool with that.)
>>>
>>> Interesting. If they have the administration set up (BTW, is that paid or
>>> do they want something for that?), they should be able to handle project
>>> specific money as well.
>>
>> NUMFOCUS does have a full time administrator, and will be happy to process
>> and distribute targeted donations for Cython. ?This is at a "no cost to
>> you" basis, ie the foundation will not take overhead but will cover these
>> costs out of NUMFOCUS donations. ?At least that is my understanding, but I
>> can confirm at the next director's meeting.
>
> Cool. Then I'm all for taking that route.

That would be great!

- Robert

From jdh2358 at gmail.com  Thu Mar 22 21:03:12 2012
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 22 Mar 2012 15:03:12 -0500
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B823D.4040007@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
	<CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>
	<4F6B823D.4040007@behnel.de>
Message-ID: <CAGD8yY8Q1JX55sqPKezjCgtPWUDoFyzp1qcznvvKzN8ffpEACg@mail.gmail.com>

On Thu, Mar 22, 2012 at 2:49 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

>
> Ok, just for clarification: the foundation is US based and not currently
> targeting any other countries, right?
>

That's my understanding.  Getting US non-profit status is a lot of work to
start with.  I'm not familiar with what it will take for other countries
but I will bring it up at the next meeting.

JDH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120322/c778ca1c/attachment.html>

From stefan_ml at behnel.de  Thu Mar 22 21:07:38 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 21:07:38 +0100
Subject: [Cython] dynamically compile and import pure python modules
In-Reply-To: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>
References: <CANg26EXkmFNOeZudNc1MK=9j1uuXUqyOwa2WAW3-7n_Ws1MLvg@mail.gmail.com>
Message-ID: <4F6B868A.1000802@behnel.de>

mark florisson, 22.03.2012 19:50:
> For the fused type runtime dispatch I found it very convenient to use
> the with statement, but that is not supported in Python 2.4. However,
> the compiler could dynamically compile compiler code with the compiler
> itself and import it (pyximport), if it is not needed to compile
> Cython itself. I gave it a try and it seems to work like a charm (but
> probably needs more testing :):
> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2

The advantages are limited, so I'm leaning towards seeing the drawbacks, of
which there are at least some. For one, *running* Cython (as opposed to
installing it) becomes more complex and involves a (much) higher first time
overhead. We'd also start putting shared libraries into user directories
without asking them first. Might be a problem on shared installations with
many users.

Note also that Cython no longer compiles itself when installing in PyPy (at
all), but that would be easy to special case here (and PyPy obviously has
features like the "with" statement).

Next, I think it would tempt us to split source files into separate modules
just because that way we can use a specific feature in one of them because
it'll get compiled (and the other half is needed for bootstrapping). That
would be bad design. OTOH, it might be worth taking a general look at the
code base to see what's really required for bootstrapping, or maybe for
compiling pure Python code in general. Factoring that out, and thus
separating the Python compiler from the Cython specific language features
might (might!) be an actual improvement. (Then again, there's .pxd
overriding and the "cython" module, which add Cython features back in, and
those two make Cython much more attactive...)

I also started to dislike pyximport because it's way outdated, fragile,
complicated and its features are overly hacked together (and I'm not
entirely innocent in that regard). I would love to see a rewrite that also
supports compiling packages properly. Not a GSoC by itself, but it's
certainly a worthy project. What about a project that aims for separating
out a Python compiler and rewriting pyximport as a jitty frontend for it?
Maybe not the greatest use case ever, but a fun project, I'd say.

Stefan

From robertwb at gmail.com  Thu Mar 22 21:44:11 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 13:44:11 -0700
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F6B6B02.4040205@behnel.de>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de> <4F6B6B02.4040205@behnel.de>
Message-ID: <CADiQ+QDTP56oyewZC=cUMzwjB4-96mA-_PPt0WaTLXA25_CkRA@mail.gmail.com>

On Thu, Mar 22, 2012 at 11:10 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 20.03.2012 18:51:
>> mark florisson, 20.03.2012 17:40:
>>> As for the two pull requests, both are quite large, but the dict
>>> iteration rewrite is more like an enhancement whereas the 'yield from'
>>> is really a new (somewhat big) feature. I would personally prefer to
>>> merge only enhancements and bug fixes, but not new features.
>>
>> Funny. I would have proposed the opposite, because I consider the "yield
>> from" implementation (as a new language feature) safer than the dict
>> iteration (which impacts existing code). As I said, I wouldn't mind
>> postponing both to 0.16.1, but I think it would be better to get them out
>> and have people use them, so that we can also fix the bugs earlier.
>
> Coming back to this, I think the main question is: what does our versioning
> scheme mean? Personally, I would expect a 0.16 release to come with new
> features compared to 0.15.x, and a 0.16.1 release to fix the bugs that
> these features (and other changes) have introduced, without changing too
> much of what makes up the 0.16 release series.

Yes, this is loosely what our versioning scheme means.

> I know that we've rarely followed that distinction (we sometimes didn't
> even have point releases), so it doesn't really matter what we add in
> 0.16.1, I guess. I just thought I'd bring up the question what would make a
> 0.17 release different from a 0.16.1 release. So far, it appears to have
> been more of an issue of how much time has passed since the last release,
> which usually goes hand in hand with a substantial set of changes.

If we have something big, then the next release will be 0.17,
otherwise it'll be 0.16.1. There is a correlation with time, but
that's not the deciding factor. (If we released right after memory
views went in last fall, despite little time passing from 0.15.1, that
would have still merited 0.16.)

The 1.0 goal is still full Python compatibility, as decided at the
workshop. The work on CyFunctions has gotten us much closer (in fact
the list at http://trac.cython.org/cython_trac/query?milestone=1.0
seems very out of date.)

- Robert

From robertwb at gmail.com  Thu Mar 22 21:52:47 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 22 Mar 2012 13:52:47 -0700
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <4F6B6EAB.2020606@astro.uio.no>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
	<4F68C3B2.7020206@behnel.de>
	<CANg26EU-xZVtf6DAr2bgmneyxAW-K4W4gzhrjnPOujemZuMGDw@mail.gmail.com>
	<CADiQ+QBTBazLv0+FsoWGPYSktFnnf2A9QQYjWD3eNWAyB1=HgA@mail.gmail.com>
	<4F6AD22F.6040602@behnel.de>
	<CANg26EU-RSasy1nTnkO-QRGw00hHTOZHn4r4O51pAGKPAYNQPg@mail.gmail.com>
	<4F6B6EAB.2020606@astro.uio.no>
Message-ID: <CADiQ+QBLfy-RR8HUjZfHQKsqck_QChoNX8u7KwXV+KA+qesKPA@mail.gmail.com>

One more note about release numbering, ideally no 0.x.y release should
be in any way backwards incompatible.

On Thu, Mar 22, 2012 at 11:25 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
>
> IMO, the ideal would be to release 0.16 now and 0.16.1 in a couple of weeks,
> but I respect that this depends on how much more additional work the release
> manager feels that involves.

Shorter release cycles should mean less work. I'd be all for a 0.16.1
shortly following a 0.16 (and a 0.16.2 not long after that). IMHO
we've gone way to long (partially my fault) since 0.15.1.

> I agree that we should wait for the patch that was started on for that bug,
> bringing in new developers is the most important thing we can do and
> satisfying users who don't live on trunk is less important than that.

In your experience, how many people live on trunk? I think most people
only upgrade to stable releases, and this is certainly what lives in
the standard distributions/package repositories.

Well, we don't want to release with the bug, and if that's the only
thing, than I wouldn't want to wait too long on it before a release
(though disabling the optimization and getting a proper fix in later
is fine). I agree that encouraging and enabling new developers is very
important (and in this particular case got the wrong impression that
Mike wasn't looking at it).

- Robert

From fperez.net at gmail.com  Thu Mar 22 22:19:56 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 22 Mar 2012 14:19:56 -0700
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CAGD8yY8Q1JX55sqPKezjCgtPWUDoFyzp1qcznvvKzN8ffpEACg@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de> <4F6B6D94.8070300@astro.uio.no>
	<4F6B777B.1090709@behnel.de>
	<CAGD8yY8OevMFR0kspbqu98grKeJZTTQOpKpRG9X0YnnHGx_PyQ@mail.gmail.com>
	<4F6B823D.4040007@behnel.de>
	<CAGD8yY8Q1JX55sqPKezjCgtPWUDoFyzp1qcznvvKzN8ffpEACg@mail.gmail.com>
Message-ID: <CAHAreOpvAURG5_6E-xQSYhvbvv6fpe6GqhZ2r7qzL3MHw_32Vg@mail.gmail.com>

On Thu, Mar 22, 2012 at 1:03 PM, John Hunter <jdh2358 at gmail.com> wrote:
> That's my understanding. ?Getting US non-profit status is a lot of work to
> start with. ?I'm not familiar with what it will take for other countries but
> I will bring it up at the next meeting.

At PyCon, we had some brief conversations with folks involved with the
European side of Python activities, and Travis also had some
conversations with the PSF (which is a US-based c3 non-profit).  I
hope that over the next few weeks we'll be able to clarify some all of
this and start offering a solution for the very problem being
discussed here.

I'm doubly motivated to make this happen both as part of NumFocus and
b/c I want to do the exact same thing regarding IPython funding.  In
IPython we've recently incurred expenses that are well beyond my
'pocket change' comfort zone, and we need to recover those costs.

So stay tuned, it won't be days before we know, but hopefully not more
than a few weeks and we'll have a slightly more coherent answer to
report.

Cheers,

f

From stefan_ml at behnel.de  Thu Mar 22 22:53:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Mar 2012 22:53:54 +0100
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
Message-ID: <4F6B9F72.70802@behnel.de>

Robert Bradshaw, 22.03.2012 19:39:
> On Thu, Mar 22, 2012 at 10:52 AM, Stefan Behnel wrote:
>> Regarding funding in general, maybe we should just start putting up one or
>> two of those sexy funding bars on our web site, like the PyPy devs do for
>> their funded projects. Assuming that goes well, it would also allow us to
>> put money on dedicated projects by paying basically ourselves for doing
>> tasks that we won't normally spend our precious spare time on (e.g. because
>> they appear too large for a weekend), but that we and our users deem
>> necessary for some reason.
> 
> While that's a good idea in theory, I'm not sure how many additional
> hours would be freed up just because we could pay ourselves for it.

And if more than one person frees hours for a given project, how would we
distribute the money? And how do we know we can still trust each other when
it comes to counting the hours? ;)


> Perhaps it would act primarily as an additional incentive to align our
> efforts with user request (though there are certainly already
> non-monetary incentives).

There sure are, and I'm sure that won't change. We should see it as an
addition to what we invest voluntarily. No-one's going to pay for code
cleanup and refactoring, for example, or for tweaks and "having fun at the
weekend" code and "I hate that being slow" optimisations.

We are not necessarily talking about large projects here that represent
person months of value. If I were to decide if I'd start implementing a
feature that looks like taking me, say, 10 days, and I'm not seriously
self-motivated in doing it, I won't even start because I know that I'll
have enough other things to do in the meantime that weigh in equally for
me. But, when I know I'll be paid for doing it, I'll certainly consider
shifting my priorities. And even if it takes three months to finish it in
my spare time, it would still be done in the end, which is much better than
just staying an open tracker entry forever.


> the
> monetization of Cython development changes the spirit of things a bit,
> and while I am a big fan of people being able to make money, or even a
> living, off of open source development

I think if that works depends a lot on what you do exactly, who the users
are and also what you do in order to sell it (and yourself). It doesn't
work for every project and certainly not for everyone.


> it complicates things a lot
> from a legal, financial, and political perspective.

Yes, I'm seeing that, too. But in any case, before it comes to asking for
donations for a given feature/project/fix/whatever, one of the first
questions will be: who can do it? And when? I think that will kill a lot of
political hassle early enough (although hopefully not the project in
question ;).


> The current model of organization X is willing to pay developer Y for
> feature Z directly seems to work well enough for the moment.

That would still work. However, a donation based model would allow us to
lower the barrier. Paying a whole feature may be too much for a single
(smaller) company, and they would have to know exactly what they want in
order to ask us to do it for them. If, instead, we put up a list of
projects we consider worth doing and they can make a donation of, say, 5%
or 10% of the actual sum and let others pay for the same feature as well,
they can just use it to show their appreciation for the general gain we
give them, without desperately needing a given feature themselves. It would
also allow users to contribute money for "nice to have" features, which is
otherwise less likely to happen.


> E.g. with
> GSoC, the bottleneck is finding good enough students and time to
> mentor them, not slots (=funding).

The mentors are not getting paid in a GSoC. So we invest our time by
guiding the student, and that's regardless of the usability of the outcome.
Even if there is an outcome, it's not unheard of that the mere overhead of
cleaning up and integrating the contribution comes close to reimplementing
it. It doesn't always work out as well as with Dag and Mark.

I'm not saying GSoCs are bad - we've certainly had a boost of overall
development power through them. But they are just one way to fund the
development, and not always the best one.


> Opening up funding to non-students
> could help a bit, but IMHO wouldn't change the balance that much (the
> gainfully employed cost a lot more and have less spare time).

It's certainly not the right way to attract new developers. But it's a way
to advance the development.

Stefan

From markflorisson88 at gmail.com  Fri Mar 23 14:26:12 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 23 Mar 2012 13:26:12 +0000
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6B9F72.70802@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
Message-ID: <CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>

On 22 March 2012 21:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 22.03.2012 19:39:
>> On Thu, Mar 22, 2012 at 10:52 AM, Stefan Behnel wrote:
>>> Regarding funding in general, maybe we should just start putting up one or
>>> two of those sexy funding bars on our web site, like the PyPy devs do for
>>> their funded projects. Assuming that goes well, it would also allow us to
>>> put money on dedicated projects by paying basically ourselves for doing
>>> tasks that we won't normally spend our precious spare time on (e.g. because
>>> they appear too large for a weekend), but that we and our users deem
>>> necessary for some reason.
>>
>> While that's a good idea in theory, I'm not sure how many additional
>> hours would be freed up just because we could pay ourselves for it.
>
> And if more than one person frees hours for a given project, how would we
> distribute the money? And how do we know we can still trust each other when
> it comes to counting the hours? ;)
>
>
>> Perhaps it would act primarily as an additional incentive to align our
>> efforts with user request (though there are certainly already
>> non-monetary incentives).
>
> There sure are, and I'm sure that won't change. We should see it as an
> addition to what we invest voluntarily. No-one's going to pay for code
> cleanup and refactoring, for example, or for tweaks and "having fun at the
> weekend" code and "I hate that being slow" optimisations.
>
> We are not necessarily talking about large projects here that represent
> person months of value. If I were to decide if I'd start implementing a
> feature that looks like taking me, say, 10 days, and I'm not seriously
> self-motivated in doing it, I won't even start because I know that I'll
> have enough other things to do in the meantime that weigh in equally for
> me. But, when I know I'll be paid for doing it, I'll certainly consider
> shifting my priorities. And even if it takes three months to finish it in
> my spare time, it would still be done in the end, which is much better than
> just staying an open tracker entry forever.
>
>
>> the
>> monetization of Cython development changes the spirit of things a bit,
>> and while I am a big fan of people being able to make money, or even a
>> living, off of open source development
>
> I think if that works depends a lot on what you do exactly, who the users
> are and also what you do in order to sell it (and yourself). It doesn't
> work for every project and certainly not for everyone.
>
>
>> it complicates things a lot
>> from a legal, financial, and political perspective.
>
> Yes, I'm seeing that, too. But in any case, before it comes to asking for
> donations for a given feature/project/fix/whatever, one of the first
> questions will be: who can do it? And when? I think that will kill a lot of
> political hassle early enough (although hopefully not the project in
> question ;).
>
>
>> The current model of organization X is willing to pay developer Y for
>> feature Z directly seems to work well enough for the moment.
>
> That would still work. However, a donation based model would allow us to
> lower the barrier. Paying a whole feature may be too much for a single
> (smaller) company, and they would have to know exactly what they want in
> order to ask us to do it for them. If, instead, we put up a list of
> projects we consider worth doing and they can make a donation of, say, 5%
> or 10% of the actual sum and let others pay for the same feature as well,
> they can just use it to show their appreciation for the general gain we
> give them, without desperately needing a given feature themselves. It would
> also allow users to contribute money for "nice to have" features, which is
> otherwise less likely to happen.
>
>
>> E.g. with
>> GSoC, the bottleneck is finding good enough students and time to
>> mentor them, not slots (=funding).
>
> The mentors are not getting paid in a GSoC. So we invest our time by
> guiding the student, and that's regardless of the usability of the outcome.
> Even if there is an outcome, it's not unheard of that the mere overhead of
> cleaning up and integrating the contribution comes close to reimplementing
> it. It doesn't always work out as well as with Dag and Mark.
>
> I'm not saying GSoCs are bad - we've certainly had a boost of overall
> development power through them. But they are just one way to fund the
> development, and not always the best one.
>
>
>> Opening up funding to non-students
>> could help a bit, but IMHO wouldn't change the balance that much (the
>> gainfully employed cost a lot more and have less spare time).
>
> It's certainly not the right way to attract new developers. But it's a way
> to advance the development.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

This may be OT for this thread, but sas numpy removed at some point
from Jenkins? I'm seeing this for all python versions since Februari
25:

Following tests excluded because of missing dependencies on your system:
   run.memoryviewattrs
   run.numpy_ValueError_T172
   run.numpy_bufacc_T155
   run.numpy_cimport
   run.numpy_memoryview
   run.numpy_parallel
   run.numpy_test
ALL DONE

From markflorisson88 at gmail.com  Fri Mar 23 15:09:57 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 23 Mar 2012 14:09:57 +0000
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
	<CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
Message-ID: <CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>

On 23 March 2012 13:26, mark florisson <markflorisson88 at gmail.com> wrote:
> On 22 March 2012 21:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Robert Bradshaw, 22.03.2012 19:39:
>>> On Thu, Mar 22, 2012 at 10:52 AM, Stefan Behnel wrote:
>>>> Regarding funding in general, maybe we should just start putting up one or
>>>> two of those sexy funding bars on our web site, like the PyPy devs do for
>>>> their funded projects. Assuming that goes well, it would also allow us to
>>>> put money on dedicated projects by paying basically ourselves for doing
>>>> tasks that we won't normally spend our precious spare time on (e.g. because
>>>> they appear too large for a weekend), but that we and our users deem
>>>> necessary for some reason.
>>>
>>> While that's a good idea in theory, I'm not sure how many additional
>>> hours would be freed up just because we could pay ourselves for it.
>>
>> And if more than one person frees hours for a given project, how would we
>> distribute the money? And how do we know we can still trust each other when
>> it comes to counting the hours? ;)
>>
>>
>>> Perhaps it would act primarily as an additional incentive to align our
>>> efforts with user request (though there are certainly already
>>> non-monetary incentives).
>>
>> There sure are, and I'm sure that won't change. We should see it as an
>> addition to what we invest voluntarily. No-one's going to pay for code
>> cleanup and refactoring, for example, or for tweaks and "having fun at the
>> weekend" code and "I hate that being slow" optimisations.
>>
>> We are not necessarily talking about large projects here that represent
>> person months of value. If I were to decide if I'd start implementing a
>> feature that looks like taking me, say, 10 days, and I'm not seriously
>> self-motivated in doing it, I won't even start because I know that I'll
>> have enough other things to do in the meantime that weigh in equally for
>> me. But, when I know I'll be paid for doing it, I'll certainly consider
>> shifting my priorities. And even if it takes three months to finish it in
>> my spare time, it would still be done in the end, which is much better than
>> just staying an open tracker entry forever.
>>
>>
>>> the
>>> monetization of Cython development changes the spirit of things a bit,
>>> and while I am a big fan of people being able to make money, or even a
>>> living, off of open source development
>>
>> I think if that works depends a lot on what you do exactly, who the users
>> are and also what you do in order to sell it (and yourself). It doesn't
>> work for every project and certainly not for everyone.
>>
>>
>>> it complicates things a lot
>>> from a legal, financial, and political perspective.
>>
>> Yes, I'm seeing that, too. But in any case, before it comes to asking for
>> donations for a given feature/project/fix/whatever, one of the first
>> questions will be: who can do it? And when? I think that will kill a lot of
>> political hassle early enough (although hopefully not the project in
>> question ;).
>>
>>
>>> The current model of organization X is willing to pay developer Y for
>>> feature Z directly seems to work well enough for the moment.
>>
>> That would still work. However, a donation based model would allow us to
>> lower the barrier. Paying a whole feature may be too much for a single
>> (smaller) company, and they would have to know exactly what they want in
>> order to ask us to do it for them. If, instead, we put up a list of
>> projects we consider worth doing and they can make a donation of, say, 5%
>> or 10% of the actual sum and let others pay for the same feature as well,
>> they can just use it to show their appreciation for the general gain we
>> give them, without desperately needing a given feature themselves. It would
>> also allow users to contribute money for "nice to have" features, which is
>> otherwise less likely to happen.
>>
>>
>>> E.g. with
>>> GSoC, the bottleneck is finding good enough students and time to
>>> mentor them, not slots (=funding).
>>
>> The mentors are not getting paid in a GSoC. So we invest our time by
>> guiding the student, and that's regardless of the usability of the outcome.
>> Even if there is an outcome, it's not unheard of that the mere overhead of
>> cleaning up and integrating the contribution comes close to reimplementing
>> it. It doesn't always work out as well as with Dag and Mark.
>>
>> I'm not saying GSoCs are bad - we've certainly had a boost of overall
>> development power through them. But they are just one way to fund the
>> development, and not always the best one.
>>
>>
>>> Opening up funding to non-students
>>> could help a bit, but IMHO wouldn't change the balance that much (the
>>> gainfully employed cost a lot more and have less spare time).
>>
>> It's certainly not the right way to attract new developers. But it's a way
>> to advance the development.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> This may be OT for this thread, but sas numpy removed at some point
> from Jenkins? I'm seeing this for all python versions since Februari
> 25:
>
> Following tests excluded because of missing dependencies on your system:
> ? run.memoryviewattrs
> ? run.numpy_ValueError_T172
> ? run.numpy_bufacc_T155
> ? run.numpy_cimport
> ? run.numpy_memoryview
> ? run.numpy_parallel
> ? run.numpy_test
> ALL DONE

I also changed the cython-sdist to pull from the release0.16 branch,
to make sure things work in the release.

From stefan_ml at behnel.de  Fri Mar 23 15:24:37 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Mar 2012 15:24:37 +0100
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
	<CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
	<CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>
Message-ID: <4F6C87A5.8010300@behnel.de>

mark florisson, 23.03.2012 15:09:
> I also changed the cython-sdist to pull from the release0.16 branch,
> to make sure things work in the release.

There are dedicated cython-release jobs for that. Not sure in what state
they are (haven't been used for a while), but they were supposed to work on
the "release" branch. If you had named the branch plain "release", they
would have done the right thing.

Cutting away cython-devel-sdist means that we loose the testing for the
master (which I am using for the pull request merges right now) and also
loose the coherent build history.

Stefan

From stefan_ml at behnel.de  Fri Mar 23 15:26:21 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Mar 2012 15:26:21 +0100
Subject: [Cython] NumPy dependency in Jenkins builds
Message-ID: <4F6C880D.6010208@behnel.de>

mark florisson, 23.03.2012 14:26:
> This may be OT for this thread

... in which case it's quite common to start a new one ...

> but sas numpy removed at some point from Jenkins? I'm seeing this for
> all python versions since Februari 25:
> 
> Following tests excluded because of missing dependencies on your
> system:
>    run.memoryviewattrs
>    run.numpy_ValueError_T172
>    run.numpy_bufacc_T155
>    run.numpy_cimport
>    run.numpy_memoryview
>    run.numpy_parallel
>    run.numpy_test
> ALL DONE

May be my fault. I think when I unified the build jobs, I might have
disabled it because we didn't have a NumPy version for Py3 at the time.

I'll look into it.

Stefan


From markflorisson88 at gmail.com  Fri Mar 23 15:29:24 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 23 Mar 2012 14:29:24 +0000
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6C87A5.8010300@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
	<CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
	<CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>
	<4F6C87A5.8010300@behnel.de>
Message-ID: <CANg26EWS9iZFJrUgN+5eyZ6iWGEqnNsHH-JZdMEf7SfUbH_70Q@mail.gmail.com>

On 23 March 2012 14:24, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 23.03.2012 15:09:
>> I also changed the cython-sdist to pull from the release0.16 branch,
>> to make sure things work in the release.
>
> There are dedicated cython-release jobs for that. Not sure in what state
> they are (haven't been used for a while), but they were supposed to work on
> the "release" branch. If you had named the branch plain "release", they
> would have done the right thing.

Ok I'll rename the release branch then.

> Cutting away cython-devel-sdist means that we loose the testing for the
> master (which I am using for the pull request merges right now) and also
> loose the coherent build history.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Mar 23 20:40:40 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Mar 2012 20:40:40 +0100
Subject: [Cython] NumPy test failures in Py2.4
Message-ID: <4F6CD1B8.1060802@behnel.de>

Hi,

after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
failure in Py2.4 (2.7 is ok):

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py24-ext/224/testReport/junit/doctest/DocTestCase/Doctest__numpy_test/

"""
File ".../run/c/numpy_test.so", line 207, in numpy_test
Failed example:
    print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a',
'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
Exception raised:
    Traceback (most recent call last):
      File "numpy_test.pyx", line 451, in
numpy_test.test_partially_packed_align (numpy_test.c:7967)
    ValueError: Buffer dtype mismatch; next field is at offset 13 but 16
expected
----------------------------------------------------------------------
File ".../run/c/numpy_test.so", line 211, in numpy_test
Failed example:
    print(test_partially_packed_align_2(np.zeros((1,),
dtype=np.dtype([('a', 'b'), ('b', 'i'), ('c', 'b'), ('sub', np.dtype('b,i',
align=True))]))))
Exception raised:
      File "numpy_test.pyx", line 459, in
numpy_test.test_partially_packed_align_2 (numpy_test.c:8186)
    ValueError: Buffer dtype mismatch; next field is at offset 7 but 10
expected
"""

Does anyone know what to make of this?

Stefan

From stefan_ml at behnel.de  Fri Mar 23 20:48:25 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Mar 2012 20:48:25 +0100
Subject: [Cython] NumPy dependency in Jenkins builds
In-Reply-To: <4F6C880D.6010208@behnel.de>
References: <4F6C880D.6010208@behnel.de>
Message-ID: <4F6CD389.6000808@behnel.de>

Stefan Behnel, 23.03.2012 15:26:
> mark florisson, 23.03.2012 14:26:
>> This may be OT for this thread
> 
> ... in which case it's quite common to start a new one ...
> 
>> but sas numpy removed at some point from Jenkins? I'm seeing this for
>> all python versions since Februari 25:
>>
>> Following tests excluded because of missing dependencies on your
>> system:
>>    run.memoryviewattrs
>>    run.numpy_ValueError_T172
>>    run.numpy_bufacc_T155
>>    run.numpy_cimport
>>    run.numpy_memoryview
>>    run.numpy_parallel
>>    run.numpy_test
>> ALL DONE
> 
> May be my fault. I think when I unified the build jobs, I might have
> disabled it because we didn't have a NumPy version for Py3 at the time.
> 
> I'll look into it.

I've re-enabled them for all CPython builds except for the latest py3k
branch. NumPy 1.6.1 doesn't compile there due to the new Unicode buffer
layout (PEP 383).

Stefan

From stefan_ml at behnel.de  Fri Mar 23 20:53:56 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Mar 2012 20:53:56 +0100
Subject: [Cython] Sphinx docs cleanup for the release - anyone?
Message-ID: <4F6CD4D4.2010207@behnel.de>

Hi,

I'm seeing several warnings and errors in the Sphinx docs build:

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-docs/477/console

Most of them look trivial, but some may not be. Does anyone have the time
to go through them?

Stefan

From stefan_ml at behnel.de  Sat Mar 24 17:06:45 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 24 Mar 2012 17:06:45 +0100
Subject: [Cython] Release branch usage
Message-ID: <4F6DF115.4080301@behnel.de>

mark florisson, 23.03.2012 15:29:
> On 23 March 2012 14:24, Stefan Behnel wrote:
>> mark florisson, 23.03.2012 15:09:
>>> I also changed the cython-sdist to pull from the release0.16
>>> branch, to make sure things work in the release.
>> 
>> There are dedicated cython-release jobs for that. Not sure in what
>> state they are (haven't been used for a while), but they were supposed
>> to work on the "release" branch. If you had named the branch plain
>> "release", they would have done the right thing.
> 
> Ok I'll rename the release branch then.

Thanks. Basically, I think the way the "release" branch should be used is
just to collect the release relevant changes up to the release, and once
that's done (and there's a tag for the release), it should simply be merged
back into the master branch to die there.

That way, we can recreate it for each new release, including emergency fix
releases, which would go on top of the release tag.

Stefan


From stefan_ml at behnel.de  Sat Mar 24 17:17:19 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 24 Mar 2012 17:17:19 +0100
Subject: [Cython] some pull requests for 0.16?
In-Reply-To: <CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
References: <4F65BFD2.8010706@behnel.de>
	<CANg26EXULr0is_z4RqWUmFZ6zMapMiuCrUNdEmf=6R8=-kDuag@mail.gmail.com>
Message-ID: <4F6DF38F.2020500@behnel.de>

mark florisson, 20.03.2012 17:40:
> Finally, there is a serious bug in the reversed(range()) optimization
> that Mike Graham discovered when working on the range() optimization
> with a runtime step, which is basically wrong for any step that is not
> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
> actually has to figure out what the actual last value was in the range
> and use that as the start instead). I personally consider this a
> blocker, so I'll try contacting Mike and see if he is still
> (interested in) working on that.

I've disabled the optimisation in the release branch.

https://github.com/cython/cython/commit/f766a7fc2a4b7fa67b2db8592034980bff9c5783

Given that this didn't work before, disabling it isn't a regression, so
I've closed the ticket (#763) and added a new one as request for
optimisation (#765).

Stefan

From markflorisson88 at gmail.com  Sat Mar 24 23:24:53 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 24 Mar 2012 22:24:53 +0000
Subject: [Cython] NumPy test failures in Py2.4
In-Reply-To: <4F6CD1B8.1060802@behnel.de>
References: <4F6CD1B8.1060802@behnel.de>
Message-ID: <CANg26EU3XqVVbA82uYVLknDg_X+5xgSD-26GMveJiSQxMLTVoA@mail.gmail.com>

On 23 March 2012 19:40, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
> failure in Py2.4 (2.7 is ok):
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py24-ext/224/testReport/junit/doctest/DocTestCase/Doctest__numpy_test/
>
> """
> File ".../run/c/numpy_test.so", line 207, in numpy_test
> Failed example:
> ? ?print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a',
> 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
> Exception raised:
> ? ?Traceback (most recent call last):
> ? ? ?File "numpy_test.pyx", line 451, in
> numpy_test.test_partially_packed_align (numpy_test.c:7967)
> ? ?ValueError: Buffer dtype mismatch; next field is at offset 13 but 16
> expected
> ----------------------------------------------------------------------
> File ".../run/c/numpy_test.so", line 211, in numpy_test
> Failed example:
> ? ?print(test_partially_packed_align_2(np.zeros((1,),
> dtype=np.dtype([('a', 'b'), ('b', 'i'), ('c', 'b'), ('sub', np.dtype('b,i',
> align=True))]))))
> Exception raised:
> ? ? ?File "numpy_test.pyx", line 459, in
> numpy_test.test_partially_packed_align_2 (numpy_test.c:8186)
> ? ?ValueError: Buffer dtype mismatch; next field is at offset 7 but 10
> expected
> """
>
> Does anyone know what to make of this?

That's probably a regression from my changes to the buffer format
parser. It should be aligning on the second integer in the substruct,
for which the entire struct is aligned and the first character padded.
I'll look into it, thanks.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sat Mar 24 23:25:44 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 24 Mar 2012 22:25:44 +0000
Subject: [Cython] NumPy dependency in Jenkins builds
In-Reply-To: <4F6CD389.6000808@behnel.de>
References: <4F6C880D.6010208@behnel.de>
	<4F6CD389.6000808@behnel.de>
Message-ID: <CANg26EX2bC3btin9zDFm7ND5yi31hQtGf0m1g+pm5A=LgQQQrQ@mail.gmail.com>

On 23 March 2012 19:48, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 23.03.2012 15:26:
>> mark florisson, 23.03.2012 14:26:
>>> This may be OT for this thread
>>
>> ... in which case it's quite common to start a new one ...
>>
>>> but sas numpy removed at some point from Jenkins? I'm seeing this for
>>> all python versions since Februari 25:
>>>
>>> Following tests excluded because of missing dependencies on your
>>> system:
>>> ? ?run.memoryviewattrs
>>> ? ?run.numpy_ValueError_T172
>>> ? ?run.numpy_bufacc_T155
>>> ? ?run.numpy_cimport
>>> ? ?run.numpy_memoryview
>>> ? ?run.numpy_parallel
>>> ? ?run.numpy_test
>>> ALL DONE
>>
>> May be my fault. I think when I unified the build jobs, I might have
>> disabled it because we didn't have a NumPy version for Py3 at the time.
>>
>> I'll look into it.
>
> I've re-enabled them for all CPython builds except for the latest py3k
> branch. NumPy 1.6.1 doesn't compile there due to the new Unicode buffer
> layout (PEP 383).
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Thanks, that's really useful.

From stefan_ml at behnel.de  Sun Mar 25 09:14:09 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 25 Mar 2012 09:14:09 +0200
Subject: [Cython] [cython-users] pointer problem
In-Reply-To: <CACvQ4hQ7t0U245LNimFRqLC=pYDMJH-61MB4uRNW3eq7JJqd+g@mail.gmail.com>
References: <CACvQ4hSFSp+tntNXmxk66L1wsAFJuE4+k+Rw54Wz4ySCzJf+WA@mail.gmail.com>
	<CAMbjb11ynSbiteCRpmuNR=HgDfppZ=TFC0q5vSDAH+rPp2vZhQ@mail.gmail.com>
	<4F671638.7040304@yahoo.no>
	<CACvQ4hQ8pMGVVoFiBZThUj=Y5c75_Lp+W0ts9NP6tNyn_N4Tig@mail.gmail.com>
	<4F675189.4030200@behnel.de>
	<CACvQ4hQ7t0U245LNimFRqLC=pYDMJH-61MB4uRNW3eq7JJqd+g@mail.gmail.com>
Message-ID: <4F6EC5C1.6000803@behnel.de>

Liguo Kong, 25.03.2012 04:04:
> I am not sure of the proper way to send a patch (never did it before), so I
> am sending patch file to you

Thanks. It's actually better to send short patches to the cython-devel
mailing list (so that the core developers can take a look - note that it's
subscribers-only), but the *best* way to do it is to set up a pull request
on github. You can even edit the file online for that (there's a button for
that when you look at the file), and once that's done, we can discuss the
change online as well. Another advantage of this is that you get your name
written into our source history. :)

There are a couple of problems with your change, though, so I'm resending
it to cython-devel for discussion here. For one, you are using Cython
specific syntax and .pyx files, so this does not belong into the "pure
mode" documentation. Also, there is a typo in your example code - a cimport
does not use the ".pxd" extension in the module name (Cython will add that
automatically when looking for the file).

Most importantly, however, It's not clear to me what you are trying to tell
the reader. Is it how to pass NumPy data into a C function? In that case,
it would belong into the NumPy specific part of the documentation (and your
C function would likely want to know something about the size of the array
your are passing). If the intention is to generally describe how to deal
with C functions that take pointers, it would rather belong into an earlier
part of the tutorial, and there's the "using C libraries" tutorial which
presents these things already.

Could you clarify that a bit?

Stefan


Your original patch for reference:

--- pure.rst	2012-03-13 21:08:19.000000000 -0700
+++ pure.patched.rst	2012-03-24 19:18:25.000000000 -0700
@@ -142,6 +142,30 @@
 arrays as ``cython.int[10]``. A limited attempt is made to emulate these more
 complex types, but only so much can be done from the Python language.

+Since use of pointers in C is ubiquitous, here we give a quick example of how
+to call C functions whose arguments contain pointers. Suppose you have a
+file in C 'C_func_file.c', which contains a function C_func with
+the following header::
+  void C_func(double * CPointer)
+
+where CPointer points a one-dimensional array of size N, whose data you want
+to access through a numpy array.
+
+Now assume that you write a corresponding .pxd file C_func_file.pxd to
+make the function cimport-able. Now you can access it in a :file: `.pyx`
+file :file: `NumpyCPointerExample.pyx`::
+
+    from C_func_file.pxd cimport C_func
+    import cython
+    cimport numpy as np
+
+    def f():
+        np.ndarray[np.double_t, ndim=1, mode="c"] numar = np.zeros((N))
+        C_func(<cython.double *> numar.data)
+        # followed by some manipulations of the numarray array
+        return
+
+
 Decorators
 --------------------------------


From markflorisson88 at gmail.com  Mon Mar 26 14:13:44 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 26 Mar 2012 13:13:44 +0100
Subject: [Cython] NumPy test failures in Py2.4
In-Reply-To: <CANg26EU3XqVVbA82uYVLknDg_X+5xgSD-26GMveJiSQxMLTVoA@mail.gmail.com>
References: <4F6CD1B8.1060802@behnel.de>
	<CANg26EU3XqVVbA82uYVLknDg_X+5xgSD-26GMveJiSQxMLTVoA@mail.gmail.com>
Message-ID: <CANg26EU6njefdZj4i1nRhwkJ60mZTTiJpMaho6K-19_80_ZcRA@mail.gmail.com>

On 24 March 2012 22:24, mark florisson <markflorisson88 at gmail.com> wrote:
> On 23 March 2012 19:40, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
>> failure in Py2.4 (2.7 is ok):
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py24-ext/224/testReport/junit/doctest/DocTestCase/Doctest__numpy_test/
>>
>> """
>> File ".../run/c/numpy_test.so", line 207, in numpy_test
>> Failed example:
>> ? ?print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a',
>> 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
>> Exception raised:
>> ? ?Traceback (most recent call last):
>> ? ? ?File "numpy_test.pyx", line 451, in
>> numpy_test.test_partially_packed_align (numpy_test.c:7967)
>> ? ?ValueError: Buffer dtype mismatch; next field is at offset 13 but 16
>> expected
>> ----------------------------------------------------------------------
>> File ".../run/c/numpy_test.so", line 211, in numpy_test
>> Failed example:
>> ? ?print(test_partially_packed_align_2(np.zeros((1,),
>> dtype=np.dtype([('a', 'b'), ('b', 'i'), ('c', 'b'), ('sub', np.dtype('b,i',
>> align=True))]))))
>> Exception raised:
>> ? ? ?File "numpy_test.pyx", line 459, in
>> numpy_test.test_partially_packed_align_2 (numpy_test.c:8186)
>> ? ?ValueError: Buffer dtype mismatch; next field is at offset 7 but 10
>> expected
>> """
>>
>> Does anyone know what to make of this?
>
> That's probably a regression from my changes to the buffer format
> parser. It should be aligning on the second integer in the substruct,
> for which the entire struct is aligned and the first character padded.
> I'll look into it, thanks.

No, there is no regression, the buffer format parser has always been
wrong in this regard. The tests have just always passed because the
buffer was obtained through numpy.pxd's __getbuffer__, which flattens
the structs. Since numpy supports the buffer interface now, it tries
to parse the actual format string but fails.

Basically the problem is that with aligned structs inside padded
structs the format parser needs to start aligning in isolation from
the actual offset. E.g. it still needs to isolate an int on a 4-byte
boundary, but in relation to the start of the aligned struct, which
may in itself not be aligned. This is not a regression so I'm leaving
it as is, as the fix is complicated and I'm rather preoccupied.
Basically I think the buffer format parser should be rewritten to use
a single stack (and so does Dag), which would make things a lot less
complicated, but that's not a trivial task.

>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Mon Mar 26 15:12:32 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 26 Mar 2012 15:12:32 +0200
Subject: [Cython] NumPy test failures in Py2.4
In-Reply-To: <CANg26EU6njefdZj4i1nRhwkJ60mZTTiJpMaho6K-19_80_ZcRA@mail.gmail.com>
References: <4F6CD1B8.1060802@behnel.de>
	<CANg26EU3XqVVbA82uYVLknDg_X+5xgSD-26GMveJiSQxMLTVoA@mail.gmail.com>
	<CANg26EU6njefdZj4i1nRhwkJ60mZTTiJpMaho6K-19_80_ZcRA@mail.gmail.com>
Message-ID: <4F706B40.9080402@behnel.de>

mark florisson, 26.03.2012 14:13:
>> On 23 March 2012 19:40, Stefan Behnel wrote:
>>> after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
>>> failure in Py2.4 (2.7 is ok):
>>> [...]
> Basically the problem is that with aligned structs inside padded
> structs the format parser needs to start aligning in isolation from
> the actual offset. E.g. it still needs to isolate an int on a 4-byte
> boundary, but in relation to the start of the aligned struct, which
> may in itself not be aligned. This is not a regression so I'm leaving
> it as is, as the fix is complicated and I'm rather preoccupied.
> Basically I think the buffer format parser should be rewritten to use
> a single stack (and so does Dag), which would make things a lot less
> complicated, but that's not a trivial task.

Ok, so how do we deal with this for the time being? Disable the test in
CPython versions where it fails? (that would be 2.4 and 2.5, I guess?)

BTW, maybe we should start using a decorator for the tests that compile
correctly on a given Python version but fail when being run. That would
allow us to handle cases like this more easily, i.e. to keep tests in one
test file even though some of them are being ignored in a specific environment.

Stefan

From markflorisson88 at gmail.com  Mon Mar 26 16:27:19 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 26 Mar 2012 15:27:19 +0100
Subject: [Cython] NumPy test failures in Py2.4
In-Reply-To: <4F706B40.9080402@behnel.de>
References: <4F6CD1B8.1060802@behnel.de>
	<CANg26EU3XqVVbA82uYVLknDg_X+5xgSD-26GMveJiSQxMLTVoA@mail.gmail.com>
	<CANg26EU6njefdZj4i1nRhwkJ60mZTTiJpMaho6K-19_80_ZcRA@mail.gmail.com>
	<4F706B40.9080402@behnel.de>
Message-ID: <CANg26EXg-JY__qsR0k1WbpiMq_CbBSaH1xJ4TZ5C87c-6TQbUg@mail.gmail.com>

On 26 March 2012 14:12, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 26.03.2012 14:13:
>>> On 23 March 2012 19:40, Stefan Behnel wrote:
>>>> after re-enabling the NumPy tests, I'm getting two NumPy (1.6.1) test
>>>> failure in Py2.4 (2.7 is ok):
>>>> [...]
>> Basically the problem is that with aligned structs inside padded
>> structs the format parser needs to start aligning in isolation from
>> the actual offset. E.g. it still needs to isolate an int on a 4-byte
>> boundary, but in relation to the start of the aligned struct, which
>> may in itself not be aligned. This is not a regression so I'm leaving
>> it as is, as the fix is complicated and I'm rather preoccupied.
>> Basically I think the buffer format parser should be rewritten to use
>> a single stack (and so does Dag), which would make things a lot less
>> complicated, but that's not a trivial task.
>
> Ok, so how do we deal with this for the time being? Disable the test in
> CPython versions where it fails? (that would be 2.4 and 2.5, I guess?)

Yes I disabled the tests. I also added some code to clear exceptions
in case the releasebuffer of getbuffer capsules weren't found. Hudson
is now blue, so I'd like to test Dag's pull request for the numpy API
changes and merge that and then do another beta release. Improved
fused type dispatching can then wait until 0.16.1.

> BTW, maybe we should start using a decorator for the tests that compile
> correctly on a given Python version but fail when being run. That would
> allow us to handle cases like this more easily, i.e. to keep tests in one
> test file even though some of them are being ignored in a specific environment.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Mon Mar 26 19:42:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 26 Mar 2012 18:42:43 +0100
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <4F6C87A5.8010300@behnel.de>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
	<CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
	<CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>
	<4F6C87A5.8010300@behnel.de>
Message-ID: <CANg26EUw=Uf12FDt1EDM=LoKWE4-f60md5WBDsF4pOb9cZ8-ZA@mail.gmail.com>

On 23 March 2012 14:24, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 23.03.2012 15:09:
>> I also changed the cython-sdist to pull from the release0.16 branch,
>> to make sure things work in the release.
>
> There are dedicated cython-release jobs for that. Not sure in what state
> they are (haven't been used for a while), but they were supposed to work on
> the "release" branch. If you had named the branch plain "release", they
> would have done the right thing.

Apparently something is wrong with the py31 and py32 release jobs:
/levi/scratch/hudson/tmp/hudson738187841750656805.sh: line 17:
/levi/scratch/robertwb/hudson/hudson/jobs/cython-release-tests/workspace/BACKEND/c/PYVERSION/py31-ext/python/bin/python:
No such file or directory (see
https://sage.math.washington.edu:8091/hudson/view/release/job/cython-release-tests/BACKEND=c,PYVERSION=py31-ext/5/console).
The originating problem is: ln: accessing
`/levi/scratch/robertwb/hudson/hudson/jobs/py31-ext-hg/lastStable/archive/python31-bin.tar.gz':
No such file or directory

> Cutting away cython-devel-sdist means that we loose the testing for the
> master (which I am using for the pull request merges right now) and also
> loose the coherent build history.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Mar 27 09:00:57 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Mar 2012 09:00:57 +0200
Subject: [Cython] funding (Re: sage.math problems?)
In-Reply-To: <CANg26EUw=Uf12FDt1EDM=LoKWE4-f60md5WBDsF4pOb9cZ8-ZA@mail.gmail.com>
References: <20244989.5174.1332306434892.JavaMail.geo-discussion-forums@pbje9>
	<4F69F25F.6090202@behnel.de> <4F6AD7ED.7000904@behnel.de>
	<CACLE5GCq+6rwYhXgYwVZXEKoYhCEOg3aAT4cLqv+sS9FCo5CZg@mail.gmail.com>
	<CANX10hA-9s1D3WKeuHGYwWKC3jerjWqaGrWG_soRODtQRanrPw@mail.gmail.com>
	<CACLE5GABoo=w=2RgaGjSaDm-pUkgHB-tXCy8FfSZssZwQKtRDw@mail.gmail.com>
	<4F6B4D6A.9080103@behnel.de> <4F6B5A35.8000401@astro.uio.no>
	<4F6B66CD.7060409@behnel.de>
	<CADiQ+QBUF-3Y2N2DdPCFR+wts4HO33AkGWtTPV2d8FsV2X11Rg@mail.gmail.com>
	<4F6B9F72.70802@behnel.de>
	<CANg26EUsXJx=+w9xJp_1VqSiJsEUpKgTLYCtgcOfiBYzpnfXig@mail.gmail.com>
	<CANg26EXj0G-n_GcuzKabCfZLkNor8oKq6wT74ESprDWZ8zeLwg@mail.gmail.com>
	<4F6C87A5.8010300@behnel.de>
	<CANg26EUw=Uf12FDt1EDM=LoKWE4-f60md5WBDsF4pOb9cZ8-ZA@mail.gmail.com>
Message-ID: <4F7165A9.8050003@behnel.de>

mark florisson, 26.03.2012 19:42:
> Apparently something is wrong with the py31 and py32 release jobs:
> /levi/scratch/hudson/tmp/hudson738187841750656805.sh: line 17:
> /levi/scratch/robertwb/hudson/hudson/jobs/cython-release-tests/workspace/BACKEND/c/PYVERSION/py31-ext/python/bin/python:
> No such file or directory (see
> https://sage.math.washington.edu:8091/hudson/view/release/job/cython-release-tests/BACKEND=c,PYVERSION=py31-ext/5/console).
> The originating problem is: ln: accessing
> `/levi/scratch/robertwb/hudson/hudson/jobs/py31-ext-hg/lastStable/archive/python31-bin.tar.gz':
> No such file or directory

Ah, sorry. A typo in the final archiving step. Should work now.

Stefan

From vitja.makarov at gmail.com  Tue Mar 27 12:57:13 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 27 Mar 2012 14:57:13 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
Message-ID: <CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>

2012/2/27 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
> Great, thanks!
>
>
> 2012/2/25 mark florisson <markflorisson88 at gmail.com>
>>
>> 2012/2/24 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
>> > Hi,
>> >
>> > could you please also look at incorporating the following patch before
>> > releasing 0.16? (if it has not already been merged)
>> >
>> > https://github.com/cython/cython/pull/67
>> >
>> > It has been more or less validated, but a test case is needed.
>> >
>> > This patch makes using C++ templates much more convenient with Cython.
>> >
>> > Currently I have to use hacks like the following which looks ugly and
>> > make
>> > the code less readable:
>> > ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData<float>"
>> >
>> > Also thank you for all the work done on Cython, I have been using it
>> > (and
>> > Pyrex before) intensively? for more than 6 years now, and it makes
>> > integrating Python and C/C++ really convenient.
>> >
>> > Thanks in advance
>> >
>> > S?bastien
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > http://mail.python.org/mailman/listinfo/cython-devel
>> >
>>
>> Ok I merged it and added a test. I also fixed a lot of tests to run
>> under MSVC on windows. I'm thinking to merge
>> https://github.com/cython/cython/pull/77, see if everything still
>> passes on Jenkins, and then pushing out a beta release for 0.16. I
>> created some release notes, please feel free to add to the page
>> (especially to the feature and improvements lists), they might be
>> incomplete: http://wiki.cython.org/ReleaseNotes-0.16
>>
>> Are there any other last-minute bug fixes pending?


Recently I've found this bug on the tracker:

http://trac.cython.org/cython_trac/ticket/766

I think I can fix it before release or wait unitl 0.16.1 since itsn't
a regression.


-- 
vitja.

From markflorisson88 at gmail.com  Tue Mar 27 13:20:00 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 27 Mar 2012 12:20:00 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
Message-ID: <CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>

On 27 March 2012 11:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/27 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
>> Great, thanks!
>>
>>
>> 2012/2/25 mark florisson <markflorisson88 at gmail.com>
>>>
>>> 2012/2/24 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
>>> > Hi,
>>> >
>>> > could you please also look at incorporating the following patch before
>>> > releasing 0.16? (if it has not already been merged)
>>> >
>>> > https://github.com/cython/cython/pull/67
>>> >
>>> > It has been more or less validated, but a test case is needed.
>>> >
>>> > This patch makes using C++ templates much more convenient with Cython.
>>> >
>>> > Currently I have to use hacks like the following which looks ugly and
>>> > make
>>> > the code less readable:
>>> > ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData<float>"
>>> >
>>> > Also thank you for all the work done on Cython, I have been using it
>>> > (and
>>> > Pyrex before) intensively? for more than 6 years now, and it makes
>>> > integrating Python and C/C++ really convenient.
>>> >
>>> > Thanks in advance
>>> >
>>> > S?bastien
>>> >
>>> > _______________________________________________
>>> > cython-devel mailing list
>>> > cython-devel at python.org
>>> > http://mail.python.org/mailman/listinfo/cython-devel
>>> >
>>>
>>> Ok I merged it and added a test. I also fixed a lot of tests to run
>>> under MSVC on windows. I'm thinking to merge
>>> https://github.com/cython/cython/pull/77, see if everything still
>>> passes on Jenkins, and then pushing out a beta release for 0.16. I
>>> created some release notes, please feel free to add to the page
>>> (especially to the feature and improvements lists), they might be
>>> incomplete: http://wiki.cython.org/ReleaseNotes-0.16
>>>
>>> Are there any other last-minute bug fixes pending?
>
>
> Recently I've found this bug on the tracker:
>
> http://trac.cython.org/cython_trac/ticket/766
>
> I think I can fix it before release or wait unitl 0.16.1 since itsn't
> a regression.
>

Sure, that's fine, I think there will be some time before the next
beta. I tested the release in my own branch and jenkins was blue, but
the release build seems to disagree. The py32 C++ build shows some
refcount error:

numpy_memoryview.cpp:16922: warning: ?void
__pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject*, Py_buffer*)?
defined but not used
python: Modules/gcmodule.c:327: visit_decref: Assertion
`gc->gc.gc_refs != 0' failed.

and some of the other python 3 tests are also failing.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Mar 27 13:58:51 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Mar 2012 13:58:51 +0200
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
Message-ID: <4F71AB7B.7090007@behnel.de>

mark florisson, 27.03.2012 13:20:
> I tested the release in my own branch and jenkins was blue, but
> the release build seems to disagree.

The release (and master) branch is tested against the "-ext" builds of
CPython, which have some external packages installed, including NumPy. You
have to change the config of your "tests" job to use those instead of the
plain CPython builds. The build job can stay as it is.

As for the inner workings, there's a normal "pyXY-hg" job to build CPython
and a corresponding "pyXY-ext-hg" job that takes the build and installs a
list of packages into it, then creates a new install archive. You can then
reference either of the two archives in your build/test jobs by asking for
a "pyXY" or "pyXY-ext" Python.


> The py32 C++ build shows some refcount error:
> 
> numpy_memoryview.cpp:16922: warning: ?void
> __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject*, Py_buffer*)?
> defined but not used

These are legitimate warnings that are worth fixing (at some point), I
think. They seem to originate from the buffer implementation in numpy.pxd.
Those sort-of-external special methods shouldn't lead to the generation of
a Python wrapper function.


> python: Modules/gcmodule.c:327: visit_decref: Assertion
> `gc->gc.gc_refs != 0' failed.

It's surprising that that only occurs in one out of four Py3 test
configurations. Maybe there's something indeterministic in those tests?


> and some of the other python 3 tests are also failing.

Yep, some of them look really funny (complaining about getting exactly the
expected output), others are the typical Py3 problems (e.g. printing bytes).

Note that the reason the py3k tests are not impacted is that it does not
have NumPy. So the tests would equally fail in all Py3 versions.

Stefan

From stefan_ml at behnel.de  Tue Mar 27 16:20:32 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Mar 2012 16:20:32 +0200
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
Message-ID: <4F71CCB0.8070700@behnel.de>

Hi,

the NumPy related tests use a file "numpy_common.pxi" that contains this
useless code:

"""
cdef extern from *:
   bint FALSE "0"
   void import_array()
   void import_umath()

if FALSE:
    import_array()
    import_umath()
"""

Does this serve any purpose? It currently leads to build failures of the
tests in C++ mode because the "import_umath()" call seems to inline code
with a bare "return" statement, which is not allowed in the module init
function (which returns a reference to the module).

Is there any reason why this can't just die?

Stefan

From dalcinl at gmail.com  Tue Mar 27 21:34:13 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 27 Mar 2012 22:34:13 +0300
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <4F71CCB0.8070700@behnel.de>
References: <4F71CCB0.8070700@behnel.de>
Message-ID: <CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>

On 27 March 2012 17:20, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> the NumPy related tests use a file "numpy_common.pxi" that contains this
> useless code:
>
> """
> cdef extern from *:
> ? bint FALSE "0"
> ? void import_array()
> ? void import_umath()
>
> if FALSE:
> ? ?import_array()
> ? ?import_umath()
> """
>
> Does this serve any purpose? It currently leads to build failures of the
> tests in C++ mode because the "import_umath()" call seems to inline code
> with a bare "return" statement, which is not allowed in the module init
> function (which returns a reference to the module).
>
> Is there any reason why this can't just die?
>

Oh! long long ago Cython testsuite was able to run without a single C
compiler warning... That numpy common file was a hack to silent
"defined but not used" warnings about the import array/umath functions
defined in  NumPy headers.

I think you can safely remove that file, it serves no useful purpose IMHO...

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at gmail.com  Tue Mar 27 22:13:11 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 27 Mar 2012 13:13:11 -0700
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <4F71CCB0.8070700@behnel.de>
References: <4F71CCB0.8070700@behnel.de>
Message-ID: <CADiQ+QDmVHptoq_6P7Q1EYi_Z8CnCnPa=hvqC8t1gigMz25kZg@mail.gmail.com>

On Tue, Mar 27, 2012 at 7:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> the NumPy related tests use a file "numpy_common.pxi" that contains this
> useless code:
>
> """
> cdef extern from *:
> ? bint FALSE "0"
> ? void import_array()
> ? void import_umath()
>
> if FALSE:
> ? ?import_array()
> ? ?import_umath()
> """
>
> Does this serve any purpose? It currently leads to build failures of the
> tests in C++ mode because the "import_umath()" call seems to inline code
> with a bare "return" statement, which is not allowed in the module init
> function (which returns a reference to the module).
>
> Is there any reason why this can't just die?

IIRC, this was to avoid warnings about unused functions when importing
the numpy headers without manually invoking these functions.

Perhaps taking their address rather than "calling" them would be
better (though this may involve creating yet more unused
variables...).

- Robert

From redbrain at gcc.gnu.org  Tue Mar 27 23:17:15 2012
From: redbrain at gcc.gnu.org (Philip Herron)
Date: Tue, 27 Mar 2012 22:17:15 +0100
Subject: [Cython] Gsoc project
Message-ID: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>

Hey

I got linked to your idea
http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
by David Malcolm on his plugin mailing list.

I am looking to apply to Gsoc once again this year i have done gsoc
2010 and 2011 on GCC implementing my own GCC front-end for python
which is still in very early stages since its a huge task. But i am
tempted to apply to this project to implement a more self contained
project to give back to the community more promptly while that hacking
on my own front-end on my own timer. And i think it would benefit me
to get to understand in more detail different aspects of python which
is what i need and would gain very much experience from.

I was wondering if you could give me some more details on how this
could all work i am not 100% familiar with cython but i think i
understand it to a good extend from playing with it for most of my
evening. I just want to make sure i understand the basic use case of
this fully, When a user could have something like:

-header foo.h

extern int add (int, int);

-source foo.c

#include "foo.h"

int add (int x, int y)
{
  return x+y;
}

We use the plugin to go over the decls created and create a pxd file like:

cdef int add (int a, int b):
    return a + b

Although this is a really basic example i just want to make sure i
understand whats going on. Maybe some more of you have input? I guess
this would be best suited as a proposal for Python rather than GCC?

--Phil

From d.s.seljebotn at astro.uio.no  Wed Mar 28 05:05:25 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 27 Mar 2012 20:05:25 -0700
Subject: [Cython] Gsoc project
In-Reply-To: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
Message-ID: <4F727FF5.5010909@astro.uio.no>

On 03/27/2012 02:17 PM, Philip Herron wrote:
> Hey
>
> I got linked to your idea
> http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
> by David Malcolm on his plugin mailing list.
>
> I am looking to apply to Gsoc once again this year i have done gsoc
> 2010 and 2011 on GCC implementing my own GCC front-end for python
> which is still in very early stages since its a huge task. But i am
> tempted to apply to this project to implement a more self contained
> project to give back to the community more promptly while that hacking
> on my own front-end on my own timer. And i think it would benefit me
> to get to understand in more detail different aspects of python which
> is what i need and would gain very much experience from.

Excellent! After talking to lots of people at PyCon about Cython, it is 
obvious that auto-generation of pxd files is *the* most missed feature 
in Cython today. If you do this, lots of Cython users will be very grateful.

>
> I was wondering if you could give me some more details on how this
> could all work i am not 100% familiar with cython but i think i
> understand it to a good extend from playing with it for most of my
> evening. I just want to make sure i understand the basic use case of
> this fully, When a user could have something like:
>
> -header foo.h
>
> extern int add (int, int);
>
> -source foo.c
>
> #include "foo.h"
>
> int add (int x, int y)
> {
>    return x+y;
> }
>
> We use the plugin to go over the decls created and create a pxd file like:
>
> cdef int add (int a, int b):
>      return a + b
>
> Although this is a really basic example i just want to make sure i
> understand whats going on. Maybe some more of you have input? I guess
> this would be best suited as a proposal for Python rather than GCC?

This isn't quite what should be done. Cython generates C code that 
includes C header files; what the pxd files are needed for is to provide 
declarations for Cython about what is available on the C side (during 
the Cython->C translation/compilation).

So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd 
like this:

cdef extern from "foo.h":
     int add(int, int)

Let us know if you have any question; you may want to look at examples 
for using Cython to wrap C code, such as

https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd

and the rest of the pyzmq code.

Moving over to the idea of making this a GSoC:

First, we have a policy of requiring patches from prospective students 
in addition to their application. Often, this has been to fix a bug or 
two in Cython. However, given that pxd generation can be done without 
much digging into Cython itself, I think that something like a crude 
prototype of the pxd generator (supporting only a subset of C) would be 
a better fit (other devs, what do you think?)

The project should contain at least:

  - The wrapper generator itself
  - Tests for it (including the task of figuring out how to test this, 
possibly both unit tests and integration tests)
  - A strategy for testing it for all relevant versions of gcc; one 
should probably set up Jenkins jobs for it

Even then, I feel that this is rather small for a full GSoC, even when 
supporting the subset of C++ supported by Cython, I would estimate a 
month or so (and GSoC is two months). So it should be extended in one 
direction or another. Some ideas:

  - Very often one is not interested in the full header file. One really 
wants "the API", not a translation of the C header. This probably 
requires a) some heuristics, and b) the possibility for, as easily as 
possible, write some selectors/configuration for what should be included 
and not. Making that end-user-friendly is perhaps a challenge, I'm not sure.

One idea here is to make possible an interplay where you look at the pyx 
file what needs to be wrapped. I.e. you first try to use a function in 
the pyx file as if it had already been declared, then run the pxd 
generator feeding in the pyx files (and .h files), and out comes the 
required pxd file bridging the two (containing only the used subset).

  - Support using clang to parse C code in addition

  - There's a problem in that an often-used Cython approach is:

  1) Generate C file from pyx and pxd files
  2) Ship to other computers
  3) Compile C file

However, this is fragile when combined with auto-generated pxd files, 
because the resulting pxd may be different depending on whether -DFOO is 
given to gcc or not.

The above 3 steps are possible because Cython often does not care about 
the exact type of something, just basic type and signedness. So if you do

cdef extern from "foo.h":
     ctypedef int sometype_t

then sometype_t can actually be a short or a char, and Cython doesn't 
care. (Similarly, not all fields of a struct needs to be exposed, only 
the ones that form part of the API.)

However, I'm not sure if the quality of an auto-generated pxd file is 
good enough for this approach.

So either a) the wrapper generator and Cython must be plugged into the 
typical setup.py build, or b) one figures out something clever (or, 
likely, more than one clever thing) which allows to continue using the 
above workflow.

Either a) and b), or both, could be part of the project. a) essentially 
requires looking at Cython.Distutils. For b), it *may* involve hooking 
into gcc *before* the preprocessor is run and take into account #ifdef 
etc, if that is even possible, and new features in Cython for specifying 
in a pxd file that "there's an #ifdef here", and see if that can somehow 
result in intelligently generated C code.

PS. I should stress that a pxd generator is *very* useful -- because it 
would do 90% of the job, and even if humans need to do the last 10% it 
is still a major timesaver.

  - More straightforward than the above: Parse Fortran through the 
gfortran GCC frontend. The Fwrap program 
(https://github.com/fwrap/fwrap) has been dormant in terms of 
development past couple of years, but is still the most promising way of 
bringing Fortran and Cython together.

Part of Fwrap's problem is the existing parser. Changing to using the 
gfortran as the parser would be spectacular, and probably revive the 
project. It has a solid test suite, so one would basically replace the 
parser component of Fwrap, make sure the test suite passes, and that 
would be it.

(Of course, few people outside the scientific community cares anything 
about Fortran.)

Those are some ideas. Remember: This is *your* project, so make sure you 
focus on features you'd find fun to play with and implement. And do NOT 
take all of the above, that's way too much :-), just find one or two 
extra features that help make the GSoC application really appealing.

Dag

From d.s.seljebotn at astro.uio.no  Wed Mar 28 05:08:41 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 27 Mar 2012 20:08:41 -0700
Subject: [Cython] Gsoc project
In-Reply-To: <4F727FF5.5010909@astro.uio.no>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
Message-ID: <4F7280B9.2060801@astro.uio.no>

On 03/27/2012 08:05 PM, Dag Sverre Seljebotn wrote:
> On 03/27/2012 02:17 PM, Philip Herron wrote:
>> Hey
>>
>> I got linked to your idea
>> http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
>>
>> by David Malcolm on his plugin mailing list.
>>
>> I am looking to apply to Gsoc once again this year i have done gsoc
>> 2010 and 2011 on GCC implementing my own GCC front-end for python
>> which is still in very early stages since its a huge task. But i am
>> tempted to apply to this project to implement a more self contained
>> project to give back to the community more promptly while that hacking
>> on my own front-end on my own timer. And i think it would benefit me
>> to get to understand in more detail different aspects of python which
>> is what i need and would gain very much experience from.
>
> Excellent! After talking to lots of people at PyCon about Cython, it is
> obvious that auto-generation of pxd files is *the* most missed feature
> in Cython today. If you do this, lots of Cython users will be very
> grateful.
>
>>
>> I was wondering if you could give me some more details on how this
>> could all work i am not 100% familiar with cython but i think i
>> understand it to a good extend from playing with it for most of my
>> evening. I just want to make sure i understand the basic use case of
>> this fully, When a user could have something like:
>>
>> -header foo.h
>>
>> extern int add (int, int);
>>
>> -source foo.c
>>
>> #include "foo.h"
>>
>> int add (int x, int y)
>> {
>> return x+y;
>> }
>>
>> We use the plugin to go over the decls created and create a pxd file
>> like:
>>
>> cdef int add (int a, int b):
>> return a + b
>>
>> Although this is a really basic example i just want to make sure i
>> understand whats going on. Maybe some more of you have input? I guess
>> this would be best suited as a proposal for Python rather than GCC?
>
> This isn't quite what should be done. Cython generates C code that
> includes C header files; what the pxd files are needed for is to provide
> declarations for Cython about what is available on the C side (during
> the Cython->C translation/compilation).
>
> So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd
> like this:
>
> cdef extern from "foo.h":
> int add(int, int)
>
> Let us know if you have any question; you may want to look at examples
> for using Cython to wrap C code, such as
>
> https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd
>
> and the rest of the pyzmq code.
>
> Moving over to the idea of making this a GSoC:
>
> First, we have a policy of requiring patches from prospective students
> in addition to their application. Often, this has been to fix a bug or
> two in Cython. However, given that pxd generation can be done without
> much digging into Cython itself, I think that something like a crude
> prototype of the pxd generator (supporting only a subset of C) would be
> a better fit (other devs, what do you think?)
>
> The project should contain at least:
>
> - The wrapper generator itself
> - Tests for it (including the task of figuring out how to test this,
> possibly both unit tests and integration tests)
> - A strategy for testing it for all relevant versions of gcc; one should
> probably set up Jenkins jobs for it
>
> Even then, I feel that this is rather small for a full GSoC, even when
> supporting the subset of C++ supported by Cython, I would estimate a
> month or so (and GSoC is two months). So it should be extended in one
> direction or another. Some ideas:

I should stress that even if you only include the above in the proposal, 
it would definitely still get consideration. It may well be better to go 
slowly but creating something rock solid, than having lots of bells and 
whistles.

It is also possible to label the above the core features, and whatever 
you decide on in addition as "optional bonus goals" in your proposal.

Dag

>
> - Very often one is not interested in the full header file. One really
> wants "the API", not a translation of the C header. This probably
> requires a) some heuristics, and b) the possibility for, as easily as
> possible, write some selectors/configuration for what should be included
> and not. Making that end-user-friendly is perhaps a challenge, I'm not
> sure.
>
> One idea here is to make possible an interplay where you look at the pyx
> file what needs to be wrapped. I.e. you first try to use a function in
> the pyx file as if it had already been declared, then run the pxd
> generator feeding in the pyx files (and .h files), and out comes the
> required pxd file bridging the two (containing only the used subset).
>
> - Support using clang to parse C code in addition
>
> - There's a problem in that an often-used Cython approach is:
>
> 1) Generate C file from pyx and pxd files
> 2) Ship to other computers
> 3) Compile C file
>
> However, this is fragile when combined with auto-generated pxd files,
> because the resulting pxd may be different depending on whether -DFOO is
> given to gcc or not.
>
> The above 3 steps are possible because Cython often does not care about
> the exact type of something, just basic type and signedness. So if you do
>
> cdef extern from "foo.h":
> ctypedef int sometype_t
>
> then sometype_t can actually be a short or a char, and Cython doesn't
> care. (Similarly, not all fields of a struct needs to be exposed, only
> the ones that form part of the API.)
>
> However, I'm not sure if the quality of an auto-generated pxd file is
> good enough for this approach.
>
> So either a) the wrapper generator and Cython must be plugged into the
> typical setup.py build, or b) one figures out something clever (or,
> likely, more than one clever thing) which allows to continue using the
> above workflow.
>
> Either a) and b), or both, could be part of the project. a) essentially
> requires looking at Cython.Distutils. For b), it *may* involve hooking
> into gcc *before* the preprocessor is run and take into account #ifdef
> etc, if that is even possible, and new features in Cython for specifying
> in a pxd file that "there's an #ifdef here", and see if that can somehow
> result in intelligently generated C code.
>
> PS. I should stress that a pxd generator is *very* useful -- because it
> would do 90% of the job, and even if humans need to do the last 10% it
> is still a major timesaver.
>
> - More straightforward than the above: Parse Fortran through the
> gfortran GCC frontend. The Fwrap program
> (https://github.com/fwrap/fwrap) has been dormant in terms of
> development past couple of years, but is still the most promising way of
> bringing Fortran and Cython together.
>
> Part of Fwrap's problem is the existing parser. Changing to using the
> gfortran as the parser would be spectacular, and probably revive the
> project. It has a solid test suite, so one would basically replace the
> parser component of Fwrap, make sure the test suite passes, and that
> would be it.
>
> (Of course, few people outside the scientific community cares anything
> about Fortran.)
>
> Those are some ideas. Remember: This is *your* project, so make sure you
> focus on features you'd find fun to play with and implement. And do NOT
> take all of the above, that's way too much :-), just find one or two
> extra features that help make the GSoC application really appealing.
>
> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From stefan_ml at behnel.de  Wed Mar 28 08:49:21 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Mar 2012 08:49:21 +0200
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
References: <4F71CCB0.8070700@behnel.de>
	<CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
Message-ID: <4F72B471.1030804@behnel.de>

Lisandro Dalcin, 27.03.2012 21:34:
> On 27 March 2012 17:20, Stefan Behnel wrote:
>> the NumPy related tests use a file "numpy_common.pxi" that contains this
>> useless code:
>>
>> """
>> cdef extern from *:
>>   bint FALSE "0"
>>   void import_array()
>>   void import_umath()
>>
>> if FALSE:
>>    import_array()
>>    import_umath()
>> """
>>
>> Does this serve any purpose? It currently leads to build failures of the
>> tests in C++ mode because the "import_umath()" call seems to inline code
>> with a bare "return" statement, which is not allowed in the module init
>> function (which returns a reference to the module).
>>
>> Is there any reason why this can't just die?
> 
> Oh! long long ago Cython testsuite was able to run without a single C
> compiler warning... That numpy common file was a hack to silent
> "defined but not used" warnings about the import array/umath functions
> defined in  NumPy headers.

Ok - in that case, it should be enough to move the import calls into a cdef
function body and "avoiding" to call that.

And a comment in the file would be helpful ...

Stefan

From stefan_ml at behnel.de  Wed Mar 28 09:11:39 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Mar 2012 09:11:39 +0200
Subject: [Cython] [cython] Stopgap solution for NumPy 1.7 API changes
	(#99)
In-Reply-To: <cython/cython/pull/99@github.com>
References: <cython/cython/pull/99@github.com>
Message-ID: <4F72B9AB.3080100@behnel.de>

Dag Sverre Seljebotn, 21.03.2012 22:36:
> NumPy is starting to seriously deprecating access to the member fields
> in an ndarray (it was always frowned upon, but now it is starting to
> become enforced). To support the large body of Cython code out there
> accessing these fields (arr.shape[0] and so on), we special-case
> PyArrayObject in Cython, with special knowledge of the NumPy API.
>     
> Ideally, we may introduce features in Cython in the future that allows
> specifying this kind of magic with syntax in pxd files, and then we can
> move away from special-casing NumPy.

Given the comments in this change, wouldn't it make sense to emit a visible
"this will eventually stop working" warning when user code runs into the
special casing hacks?

Just because the fields are convenient doesn't mean code should rely on
them being there when the NumPy folks say they shouldn't be accessible. As
I understand it, it's encouraged to use the accessor functions instead. If
that's what the NumPy developers want, users should just do that, should
they not?

Stefan

From stefan_ml at behnel.de  Wed Mar 28 09:20:38 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Mar 2012 09:20:38 +0200
Subject: [Cython] Gsoc project
In-Reply-To: <4F727FF5.5010909@astro.uio.no>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
Message-ID: <4F72BBC6.9010401@behnel.de>

Dag Sverre Seljebotn, 28.03.2012 05:05:
>  - Very often one is not interested in the full header file. One really
> wants "the API", not a translation of the C header. This probably requires
> a) some heuristics, and b) the possibility for, as easily as possible,
> write some selectors/configuration for what should be included and not.
> Making that end-user-friendly is perhaps a challenge, I'm not sure.

It's usually not a problem when there's too much declared in a .pxd, but
users may want to leave specific things out. So I think a simple blacklist
configuration file would work just fine.

Stefan

From stefan_ml at behnel.de  Wed Mar 28 09:52:38 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Mar 2012 09:52:38 +0200
Subject: [Cython] Gsoc project
In-Reply-To: <4F72BBC6.9010401@behnel.de>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no> <4F72BBC6.9010401@behnel.de>
Message-ID: <4F72C346.40909@behnel.de>

Stefan Behnel, 28.03.2012 09:20:
> Dag Sverre Seljebotn, 28.03.2012 05:05:
>>  - Very often one is not interested in the full header file. One really
>> wants "the API", not a translation of the C header. This probably requires
>> a) some heuristics, and b) the possibility for, as easily as possible,
>> write some selectors/configuration for what should be included and not.
>> Making that end-user-friendly is perhaps a challenge, I'm not sure.
> 
> It's usually not a problem when there's too much declared in a .pxd, but
> users may want to leave specific things out. So I think a simple blacklist
> configuration file would work just fine.

Adding a bit to this: The other side of the use case, that you want to
override a specific definition that was auto-generated into a .pxd file, is
trivial in the sense that you can always add a hand-written .pxd file next
to it and use that instead. And for declarations that you need to override
but that are used in the .pxd file itself, a blacklist entry plus a cimport
from a hand-written .pxd file would do the trick.

So, basically, the generator would start by creating two .pxd files: one to
be manually edited and one that contains the generated declarations and
that cimports (or, maybe even better, includes) the editable file at the
top. From that point on, it only changes the generated file and uses a
blacklist to keep out certain declarations. It's then up to the user to
provide them in the hand written file (or not at all).

Sounds easy enough.

Stefan

From d.s.seljebotn at astro.uio.no  Wed Mar 28 16:40:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 28 Mar 2012 07:40:47 -0700
Subject: [Cython]
	=?utf-8?q?=5Bcython=5D_Stopgap_solution_for_NumPy_1=2E7_?=
	=?utf-8?b?QVBJIGNoYW5nZXMJKCM5OSk=?=
In-Reply-To: <4F72B9AB.3080100@behnel.de>
References: <cython/cython/pull/99@github.com> <4F72B9AB.3080100@behnel.de>
Message-ID: <ad1988cc-9d02-4e39-b180-a04db351d9d2@email.android.com>

The idea really was that it would never stop working, but could in time be replaced by other features that would make this implementable in the pxd (like inlineable properties on extension types, typed tuples or lists).

Consider it speeding up the ndarray Python API, not providing any C API.

Note that even before this, the 'dims' field was renamed 'shape', because it is shape in the Python API.

Dag
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Stefan Behnel <stefan_ml at behnel.de> wrote:

Dag Sverre Seljebotn, 21.03.2012 22:36:
> NumPy is starting to seriously deprecating access to the member fields
> in an ndarray (it was always frowned upon, but now it is starting to
> become enforced). To support the large body of Cython code out there
> accessing these fields (arr.shape[0] and so on), we special-case
> PyArrayObject in Cython, with special knowledge of the NumPy API.
> 
> Ideally, we may introduce features in Cython in the future that allows
> specifying this kind of magic with syntax in pxd files, and then we can
> move away from special-casing NumPy.

Given the comments in this change, wouldn't it make sense to emit a visible
"this will eventually stop working" warning when user code runs into the
special casing hacks?

Just because the fields are convenient doesn't mean code should rely on
them being there when the NumPy folks say they shouldn't be accessible. As
I understand it, it's encouraged to use the accessor functions instead. If
that's what the NumPy developers want, users should just do that, should
they not?

Stefan
_____________________________________________

cython-devel mailing list
cython-devel at python.org
http://mail.python.org/mailman/listinfo/cython-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120328/5b6850de/attachment.html>

From cross at ueh0.bank.gov.ua  Wed Mar 28 17:39:26 2012
From: cross at ueh0.bank.gov.ua (Oleksandr Kreshchenko)
Date: Wed, 28 Mar 2012 18:39:26 +0300
Subject: [Cython] test
Message-ID: <4F7330AE.7010703@ueh0.bank.gov.ua>


From robertwb at gmail.com  Wed Mar 28 21:57:10 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 28 Mar 2012 12:57:10 -0700
Subject: [Cython] Gsoc project
In-Reply-To: <4F72C346.40909@behnel.de>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no> <4F72BBC6.9010401@behnel.de>
	<4F72C346.40909@behnel.de>
Message-ID: <CADiQ+QCYCFtOtnJy4wb_eqoyzubThq9AeuSE974KdMfuXPJ83g@mail.gmail.com>

On Wed, Mar 28, 2012 at 12:52 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 28.03.2012 09:20:
>> Dag Sverre Seljebotn, 28.03.2012 05:05:
>>> ?- Very often one is not interested in the full header file. One really
>>> wants "the API", not a translation of the C header. This probably requires
>>> a) some heuristics, and b) the possibility for, as easily as possible,
>>> write some selectors/configuration for what should be included and not.
>>> Making that end-user-friendly is perhaps a challenge, I'm not sure.
>>
>> It's usually not a problem when there's too much declared in a .pxd, but
>> users may want to leave specific things out. So I think a simple blacklist
>> configuration file would work just fine.
>
> Adding a bit to this: The other side of the use case, that you want to
> override a specific definition that was auto-generated into a .pxd file, is
> trivial in the sense that you can always add a hand-written .pxd file next
> to it and use that instead. And for declarations that you need to override
> but that are used in the .pxd file itself, a blacklist entry plus a cimport
> from a hand-written .pxd file would do the trick.
>
> So, basically, the generator would start by creating two .pxd files: one to
> be manually edited and one that contains the generated declarations and
> that cimports (or, maybe even better, includes) the editable file at the
> top. From that point on, it only changes the generated file and uses a
> blacklist to keep out certain declarations. It's then up to the user to
> provide them in the hand written file (or not at all).
>
> Sounds easy enough.

Perhaps the blacklist could be (in part) based on the hand-generated
.pxd file, and the generated .pxd file would simply not emit
re-declarations.

- Robert

From robertwb at gmail.com  Wed Mar 28 22:08:53 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 28 Mar 2012 13:08:53 -0700
Subject: [Cython] Gsoc project
In-Reply-To: <4F727FF5.5010909@astro.uio.no>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
Message-ID: <CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>

On Tue, Mar 27, 2012 at 8:05 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 03/27/2012 02:17 PM, Philip Herron wrote:
>>
>> Hey
>>
>> I got linked to your idea
>>
>> http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
>> by David Malcolm on his plugin mailing list.
>>
>> I am looking to apply to Gsoc once again this year i have done gsoc
>> 2010 and 2011 on GCC implementing my own GCC front-end for python
>> which is still in very early stages since its a huge task. But i am
>> tempted to apply to this project to implement a more self contained
>> project to give back to the community more promptly while that hacking
>> on my own front-end on my own timer. And i think it would benefit me
>> to get to understand in more detail different aspects of python which
>> is what i need and would gain very much experience from.
>
>
> Excellent! After talking to lots of people at PyCon about Cython, it is
> obvious that auto-generation of pxd files is *the* most missed feature in
> Cython today. If you do this, lots of Cython users will be very grateful.

+1, this idea has been floated many times before, and I think it would
make a great GSoC project.

>> I was wondering if you could give me some more details on how this
>> could all work i am not 100% familiar with cython but i think i
>> understand it to a good extend from playing with it for most of my
>> evening. I just want to make sure i understand the basic use case of
>> this fully, When a user could have something like:
>>
>> -header foo.h
>>
>> extern int add (int, int);
>>
>> -source foo.c
>>
>> #include "foo.h"
>>
>> int add (int x, int y)
>> {
>> ? return x+y;
>> }
>>
>> We use the plugin to go over the decls created and create a pxd file like:
>>
>> cdef int add (int a, int b):
>> ? ? return a + b
>>
>> Although this is a really basic example i just want to make sure i
>> understand whats going on. Maybe some more of you have input? I guess
>> this would be best suited as a proposal for Python rather than GCC?
>
>
> This isn't quite what should be done. Cython generates C code that includes
> C header files; what the pxd files are needed for is to provide declarations
> for Cython about what is available on the C side (during the Cython->C
> translation/compilation).
>
> So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd
> like this:
>
> cdef extern from "foo.h":
> ? ?int add(int, int)
>
> Let us know if you have any question; you may want to look at examples for
> using Cython to wrap C code, such as
>
> https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd
>
> and the rest of the pyzmq code.
>
> Moving over to the idea of making this a GSoC:
>
> First, we have a policy of requiring patches from prospective students in
> addition to their application. Often, this has been to fix a bug or two in
> Cython. However, given that pxd generation can be done without much digging
> into Cython itself, I think that something like a crude prototype of the pxd
> generator (supporting only a subset of C) would be a better fit (other devs,
> what do you think?)

Yep, that would be sufficient for me.

> The project should contain at least:
>
> ?- The wrapper generator itself
> ?- Tests for it (including the task of figuring out how to test this,
> possibly both unit tests and integration tests)
> ?- A strategy for testing it for all relevant versions of gcc; one should
> probably set up Jenkins jobs for it
>
> Even then, I feel that this is rather small for a full GSoC, even when
> supporting the subset of C++ supported by Cython, I would estimate a month
> or so (and GSoC is two months). So it should be extended in one direction or
> another. Some ideas:
>
> ?- Very often one is not interested in the full header file. One really
> wants "the API", not a translation of the C header. This probably requires
> a) some heuristics, and b) the possibility for, as easily as possible, write
> some selectors/configuration for what should be included and not. Making
> that end-user-friendly is perhaps a challenge, I'm not sure.
>
> One idea here is to make possible an interplay where you look at the pyx
> file what needs to be wrapped. I.e. you first try to use a function in the
> pyx file as if it had already been declared, then run the pxd generator
> feeding in the pyx files (and .h files), and out comes the required pxd file
> bridging the two (containing only the used subset).
>
> ?- Support using clang to parse C code in addition
>
> ?- There's a problem in that an often-used Cython approach is:
>
> ?1) Generate C file from pyx and pxd files
> ?2) Ship to other computers
> ?3) Compile C file
>
> However, this is fragile when combined with auto-generated pxd files,
> because the resulting pxd may be different depending on whether -DFOO is
> given to gcc or not.
>
> The above 3 steps are possible because Cython often does not care about the
> exact type of something, just basic type and signedness. So if you do
>
> cdef extern from "foo.h":
> ? ?ctypedef int sometype_t
>
> then sometype_t can actually be a short or a char, and Cython doesn't care.
> (Similarly, not all fields of a struct needs to be exposed, only the ones
> that form part of the API.)
>
> However, I'm not sure if the quality of an auto-generated pxd file is good
> enough for this approach.
>
> So either a) the wrapper generator and Cython must be plugged into the
> typical setup.py build, or b) one figures out something clever (or, likely,
> more than one clever thing) which allows to continue using the above
> workflow.
>
> Either a) and b), or both, could be part of the project. a) essentially
> requires looking at Cython.Distutils. For b), it *may* involve hooking into
> gcc *before* the preprocessor is run and take into account #ifdef etc, if
> that is even possible, and new features in Cython for specifying in a pxd
> file that "there's an #ifdef here", and see if that can somehow result in
> intelligently generated C code.
>
> PS. I should stress that a pxd generator is *very* useful -- because it
> would do 90% of the job, and even if humans need to do the last 10% it is
> still a major timesaver.
>
> ?- More straightforward than the above: Parse Fortran through the gfortran
> GCC frontend. The Fwrap program (https://github.com/fwrap/fwrap) has been
> dormant in terms of development past couple of years, but is still the most
> promising way of bringing Fortran and Cython together.
>
> Part of Fwrap's problem is the existing parser. Changing to using the
> gfortran as the parser would be spectacular, and probably revive the
> project. It has a solid test suite, so one would basically replace the
> parser component of Fwrap, make sure the test suite passes, and that would
> be it.
>
> (Of course, few people outside the scientific community cares anything about
> Fortran.)
>
> Those are some ideas. Remember: This is *your* project, so make sure you
> focus on features you'd find fun to play with and implement. And do NOT take
> all of the above, that's way too much :-), just find one or two extra
> features that help make the GSoC application really appealing.

To make the proposal more concrete (and progress mesurable), I might
suggest listing 8-10 specific, non-trivial libraries that your pxd
generator should be able to handle (e.g. 65% coverage by midterm,
100%, or even 95% by final evaluation). E.g. the C++ stl would be a
good candidate.

I'd love to see this supported.

- Robert

From robertwb at gmail.com  Wed Mar 28 22:30:10 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 28 Mar 2012 13:30:10 -0700
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <4F72B471.1030804@behnel.de>
References: <4F71CCB0.8070700@behnel.de>
	<CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
	<4F72B471.1030804@behnel.de>
Message-ID: <CADiQ+QDAFwhHx-fLQmSHvNgiF0RUj5xm+HBn8dQHEi3Fs616jw@mail.gmail.com>

On Tue, Mar 27, 2012 at 11:49 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 27.03.2012 21:34:
>> On 27 March 2012 17:20, Stefan Behnel wrote:
>>> the NumPy related tests use a file "numpy_common.pxi" that contains this
>>> useless code:
>>>
>>> """
>>> cdef extern from *:
>>> ? bint FALSE "0"
>>> ? void import_array()
>>> ? void import_umath()
>>>
>>> if FALSE:
>>> ? ?import_array()
>>> ? ?import_umath()
>>> """
>>>
>>> Does this serve any purpose? It currently leads to build failures of the
>>> tests in C++ mode because the "import_umath()" call seems to inline code
>>> with a bare "return" statement, which is not allowed in the module init
>>> function (which returns a reference to the module).
>>>
>>> Is there any reason why this can't just die?
>>
>> Oh! long long ago Cython testsuite was able to run without a single C
>> compiler warning... That numpy common file was a hack to silent
>> "defined but not used" warnings about the import array/umath functions
>> defined in ?NumPy headers.
>
> Ok - in that case, it should be enough to move the import calls into a cdef
> function body and "avoiding" to call that.

And then gcc helpfully spits out an "unused function" warning, which
is what we're trying to avoid in the first place...

> And a comment in the file would be helpful ...

Done.

From redbrain at gcc.gnu.org  Thu Mar 29 04:58:21 2012
From: redbrain at gcc.gnu.org (Philip Herron)
Date: Thu, 29 Mar 2012 03:58:21 +0100
Subject: [Cython] Gsoc project
In-Reply-To: <CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
Message-ID: <CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>

Hey all

I am implemented a very crude and simplistic and very badly programmed
version of a pxd generator i think i understand what were after now
but i would appreciate if you look over what i did to make sure i have
grasped the basic idea for now:

So if i have:

#include "test.h"

int add (int x, int y)
{
  return x + y;
}

#ifndef __TEST_H__
#define __TEST_H__

extern int add (int, int);

struct foobar {
  int a;
  unsigned char * b;
};

#endif //__TEST_H_

We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c

And i output out.pxd:

cdef extern from "test.h":
	int add (int, int)

	ctypedef struct foobar:
		int a
		unsigned char * b

So far in a very crude way it understands structs and functions but
nothing else i can see how it could all work but i see now why you see
it could be a very small project David's plugin system has mostly
everything done for you but i would like to add to his plugin for some
tree access code etc...

Adding a config file for telling the plugin to not care about certain
things would be a nice addition. It seems interesting the clang idea,
it could be interesting porting this to other front-ends of gcc like
gccgo.

--Phil
-------------- next part --------------
A non-text attachment was scrubbed...
Name: walk.py
Type: text/x-python
Size: 2180 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120329/cdeb9453/attachment.py>

From d.s.seljebotn at astro.uio.no  Thu Mar 29 05:28:54 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 28 Mar 2012 20:28:54 -0700
Subject: [Cython] Gsoc project
In-Reply-To: <CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
Message-ID: <4F73D6F6.1010402@astro.uio.no>

On 03/28/2012 07:58 PM, Philip Herron wrote:
> Hey all
>
> I am implemented a very crude and simplistic and very badly programmed
> version of a pxd generator i think i understand what were after now
> but i would appreciate if you look over what i did to make sure i have
> grasped the basic idea for now:
>
> So if i have:
>
> #include "test.h"
>
> int add (int x, int y)
> {
>    return x + y;
> }
>
> #ifndef __TEST_H__
> #define __TEST_H__
>
> extern int add (int, int);
>
> struct foobar {
>    int a;
>    unsigned char * b;
> };
>
> #endif //__TEST_H_
>
> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
>
> And i output out.pxd:
>
> cdef extern from "test.h":
> 	int add (int, int)
>
> 	ctypedef struct foobar:
> 		int a
> 		unsigned char * b

Nice. But please use 4 spaces (see PEP 8) :-)

More ideas for project proposal:

Another slight complication is that you should ideally turn

#define FOO 3
#define BAR 4

into

cdef extern from "foo.h":
     enum:
         FOO
         BAR

so you need to hook in before the preprocessor and after the 
preprocessor and dig out different stuff.

Then what happens if you have

#ifdef FOO
#define BAR 3
#else
#define BAR 4
#endif

?? I'm not saying it is hard, but perhaps no longer completely trivial :-)

And like Robert hinted at, supporting all the aspects of C++ might take 
a little more work, there's just so much different syntax in C++, and 
there's several C++ features Cython just does not support and must be 
either ignored or hacked around (e.g., "const").

Supporting stuff like

#define MACRO(x) ((x)->field*2)

probably belongs in the category of "must be done manually".

>
> So far in a very crude way it understands structs and functions but
> nothing else i can see how it could all work but i see now why you see
> it could be a very small project David's plugin system has mostly
> everything done for you but i would like to add to his plugin for some
> tree access code etc...
>
> Adding a config file for telling the plugin to not care about certain
> things would be a nice addition. It seems interesting the clang idea,
> it could be interesting porting this to other front-ends of gcc like
> gccgo.

Does gccgo use the C ABI so that Cython could call it? If so, go for it!

(Fortran is actually very much in use in the Cython userbase and would 
get a lot more "customers" than Go, but if you have more of a CS 
background or similar I can see why you wouldn't be so interested in 
Fortran. I didn't believe people were still using Fortran either until I 
started doing astrophysics, and suddenly it seems to be the default tool 
everybody uses for everything.)

Dag

From stefan_ml at behnel.de  Thu Mar 29 08:51:24 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 29 Mar 2012 08:51:24 +0200
Subject: [Cython] Gsoc project
In-Reply-To: <4F73D6F6.1010402@astro.uio.no>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
	<4F73D6F6.1010402@astro.uio.no>
Message-ID: <4F74066C.2070609@behnel.de>

Dag Sverre Seljebotn, 29.03.2012 05:28:
> On 03/28/2012 07:58 PM, Philip Herron wrote:
>> I am implemented a very crude and simplistic and very badly programmed
>> version of a pxd generator i think i understand what were after now
>> but i would appreciate if you look over what i did to make sure i have
>> grasped the basic idea for now:
>>
>> So if i have:
>>
>> #include "test.h"
>>
>> int add (int x, int y)
>> {
>>    return x + y;
>> }
>>
>> #ifndef __TEST_H__
>> #define __TEST_H__
>>
>> extern int add (int, int);
>>
>> struct foobar {
>>    int a;
>>    unsigned char * b;
>> };
>>
>> #endif //__TEST_H_
>>
>> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
>>
>> And i output out.pxd:
>>
>> cdef extern from "test.h":
>>     int add (int, int)
>>
>>     ctypedef struct foobar:
>>         int a
>>         unsigned char * b
> 
> Nice.

Yep.


> But please use 4 spaces (see PEP 8) :-)

Yep.


> More ideas for project proposal:
> 
> Another slight complication is that you should ideally turn
> 
> #define FOO 3
> #define BAR 4
> 
> into
> 
> cdef extern from "foo.h":
>     enum:
>         FOO
>         BAR
> 
> so you need to hook in before the preprocessor and after the preprocessor
> and dig out different stuff.
> 
> Then what happens if you have
> 
> #ifdef FOO
> #define BAR 3
> #else
> #define BAR 4
> #endif
> 
> ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)

These things bring us close to what SWIG wants to achieve, though. I think
we should find a point where we should stop and leave it to users writing
their own supplemental .pxd file.

When generating the initial manual .pxd file, we could still move all stuff
in there that we think users should take care of. It's easy enough to
delete or comment out declarations once they're there - much easier than
wading through a header file trying to figure out what might work and what
might not.


> And like Robert hinted at, supporting all the aspects of C++ might take a
> little more work, there's just so much different syntax in C++, and there's
> several C++ features Cython just does not support and must be either
> ignored or hacked around (e.g., "const").

It would still be good to have them sort-of supported in the generator,
even if there's no declaration code coming out of it. In the specific case
of "const", we still want to support it in Cython at some point, and I'm
sure there are other things of that kind.


> Supporting stuff like
> 
> #define MACRO(x) ((x)->field*2)
> 
> probably belongs in the category of "must be done manually".

I think that's a good case for generating "something" into the editable
file (maybe even commented out) and letting the user make sense of it. We
could have standard comments for different constructs that we write into
the user file, e.g.

    ## please add types for macro MACRO, defined as
    ## MACRO(x) ((x)->field*2)
    # cdef ? MACRO(? x)

Probably also with some help text at the top of the file that shows how to
define different signatures for the same function declaration (using
explicit cnames) plus a link to the relevant docs.


>> it could be interesting porting this to other front-ends of gcc like
>> gccgo.
> 
> Does gccgo use the C ABI so that Cython could call it? If so, go for it!

Absolutely. We already have fwrap for Fortran, but if there's a way to use
GCC in order to generate glue code (i.e. not just .pxd files but also
header files and some adaptation code) that talks to even more languages,
that would be more than you could ever dream of as result of a GSoC. As
long as a language supports the C-ABI, it may even be easier to do than for
C/C++, because most languages simply don't have preprocessor macros and
similarly hard to analyse features. Even if it's just a "you have to write
your functions in this specific way in language X and only then the tool
can adapt them for you", that would still be extremely useful.

It certainly sounds like Philip has the right background for something like
that.

Stefan

From robertwb at gmail.com  Thu Mar 29 11:19:46 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 29 Mar 2012 02:19:46 -0700
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <4F72B471.1030804@behnel.de>
References: <4F71CCB0.8070700@behnel.de>
	<CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
	<4F72B471.1030804@behnel.de>
Message-ID: <CADiQ+QAxLBdjBFbY7BrxNGZOh-SYARFQ=xG6oH03dTNU3o5+kw@mail.gmail.com>

On Tue, Mar 27, 2012 at 11:49 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 27.03.2012 21:34:
>> On 27 March 2012 17:20, Stefan Behnel wrote:
>>> the NumPy related tests use a file "numpy_common.pxi" that contains this
>>> useless code:
>>>
>>> """
>>> cdef extern from *:
>>> ? bint FALSE "0"
>>> ? void import_array()
>>> ? void import_umath()
>>>
>>> if FALSE:
>>> ? ?import_array()
>>> ? ?import_umath()
>>> """
>>>
>>> Does this serve any purpose? It currently leads to build failures of the
>>> tests in C++ mode because the "import_umath()" call seems to inline code
>>> with a bare "return" statement, which is not allowed in the module init
>>> function (which returns a reference to the module).
>>>
>>> Is there any reason why this can't just die?
>>
>> Oh! long long ago Cython testsuite was able to run without a single C
>> compiler warning... That numpy common file was a hack to silent
>> "defined but not used" warnings about the import array/umath functions
>> defined in ?NumPy headers.
>
> Ok - in that case, it should be enough to move the import calls into a cdef
> function body and "avoiding" to call that.
>
> And a comment in the file would be helpful ...

I see you disabled this in the release branch (resulting in the
useless /numpy/core/include/numpy/__multiarray_api.h:1187: warning:
?int _import_array()? defined but not used). Was this intentional?

- Robert

From stefan_ml at behnel.de  Thu Mar 29 11:25:57 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 29 Mar 2012 11:25:57 +0200
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <CADiQ+QAxLBdjBFbY7BrxNGZOh-SYARFQ=xG6oH03dTNU3o5+kw@mail.gmail.com>
References: <4F71CCB0.8070700@behnel.de>
	<CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
	<4F72B471.1030804@behnel.de>
	<CADiQ+QAxLBdjBFbY7BrxNGZOh-SYARFQ=xG6oH03dTNU3o5+kw@mail.gmail.com>
Message-ID: <4F742AA5.3010008@behnel.de>

Robert Bradshaw, 29.03.2012 11:19:
>>> On 27 March 2012 17:20, Stefan Behnel wrote:
>>>> the NumPy related tests use a file "numpy_common.pxi" that contains this
>>>> useless code:
>>>>
>>>> """
>>>> cdef extern from *:
>>>>   bint FALSE "0"
>>>>   void import_array()
>>>>   void import_umath()
>>>>
>>>> if FALSE:
>>>>    import_array()
>>>>    import_umath()
>>>> """
>
> I see you disabled this in the release branch (resulting in the
> useless /numpy/core/include/numpy/__multiarray_api.h:1187: warning:
> ?int _import_array()? defined but not used). Was this intentional?

What's the alternative? It currently breaks the C++ tests - I think a
warning (in addition to those that are there already) is much better than
an entire test that fails to compile.

Stefan

From robertwb at gmail.com  Thu Mar 29 11:32:41 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 29 Mar 2012 02:32:41 -0700
Subject: [Cython] Question on "numpy_common.pxi" in NumPy tests
In-Reply-To: <4F742AA5.3010008@behnel.de>
References: <4F71CCB0.8070700@behnel.de>
	<CAEcYPwCjnVfDk56n6_2rf+=1z4DJ3vQtm2Sg017w93n=a8gm=g@mail.gmail.com>
	<4F72B471.1030804@behnel.de>
	<CADiQ+QAxLBdjBFbY7BrxNGZOh-SYARFQ=xG6oH03dTNU3o5+kw@mail.gmail.com>
	<4F742AA5.3010008@behnel.de>
Message-ID: <CADiQ+QAtU0jGhEMGkVYPOOzCNWXK3-gLi1F2Bbo9_zWSxLRbQw@mail.gmail.com>

On Thu, Mar 29, 2012 at 2:25 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 29.03.2012 11:19:
>>>> On 27 March 2012 17:20, Stefan Behnel wrote:
>>>>> the NumPy related tests use a file "numpy_common.pxi" that contains this
>>>>> useless code:
>>>>>
>>>>> """
>>>>> cdef extern from *:
>>>>> ? bint FALSE "0"
>>>>> ? void import_array()
>>>>> ? void import_umath()
>>>>>
>>>>> if FALSE:
>>>>> ? ?import_array()
>>>>> ? ?import_umath()
>>>>> """
>>
>> I see you disabled this in the release branch (resulting in the
>> useless /numpy/core/include/numpy/__multiarray_api.h:1187: warning:
>> ?int _import_array()? defined but not used). Was this intentional?
>
> What's the alternative? It currently breaks the C++ tests - I think a
> warning (in addition to those that are there already) is much better than
> an entire test that fails to compile.

Perhaps just comment out the problematic import_umath().

From markflorisson88 at gmail.com  Thu Mar 29 12:10:27 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 29 Mar 2012 11:10:27 +0100
Subject: [Cython] Gsoc project
In-Reply-To: <4F73D6F6.1010402@astro.uio.no>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
	<4F73D6F6.1010402@astro.uio.no>
Message-ID: <CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>

On 29 March 2012 04:28, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 03/28/2012 07:58 PM, Philip Herron wrote:
>>
>> Hey all
>>
>> I am implemented a very crude and simplistic and very badly programmed
>> version of a pxd generator i think i understand what were after now
>> but i would appreciate if you look over what i did to make sure i have
>> grasped the basic idea for now:
>>
>> So if i have:
>>
>> #include "test.h"
>>
>> int add (int x, int y)
>> {
>> ? return x + y;
>> }
>>
>> #ifndef __TEST_H__
>> #define __TEST_H__
>>
>> extern int add (int, int);
>>
>> struct foobar {
>> ? int a;
>> ? unsigned char * b;
>> };
>>
>> #endif //__TEST_H_
>>
>> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
>>
>> And i output out.pxd:
>>
>> cdef extern from "test.h":
>> ? ? ? ?int add (int, int)
>>
>> ? ? ? ?ctypedef struct foobar:
>> ? ? ? ? ? ? ? ?int a
>> ? ? ? ? ? ? ? ?unsigned char * b
>
>
> Nice. But please use 4 spaces (see PEP 8) :-)
>
> More ideas for project proposal:
>
> Another slight complication is that you should ideally turn
>
> #define FOO 3
> #define BAR 4
>
> into
>
> cdef extern from "foo.h":
> ? ?enum:
> ? ? ? ?FOO
> ? ? ? ?BAR
>
> so you need to hook in before the preprocessor and after the preprocessor
> and dig out different stuff.

David, I'm CCing you as this might be of interest to you.

I think the current GCC plugin support doesn't allow you to do much
with te preprocessor, it operates entirely after the C preprocessor
has run. So to support macros we have to consider that for this to
work the gcc plugin may have to be extended, which uses C to extend
GCC and Python, so it also requires knowledge of the CPython C API.
David, would you mind elaborating why C was used for this project and
not (partially) Cython, and would it be possible to extend the plugin
with Cython?

> Then what happens if you have
>
> #ifdef FOO
> #define BAR 3
> #else
> #define BAR 4
> #endif
>
> ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
>
> And like Robert hinted at, supporting all the aspects of C++ might take a
> little more work, there's just so much different syntax in C++, and there's
> several C++ features Cython just does not support and must be either ignored
> or hacked around (e.g., "const").
>
> Supporting stuff like
>
> #define MACRO(x) ((x)->field*2)
>
> probably belongs in the category of "must be done manually".
>
>
>>
>> So far in a very crude way it understands structs and functions but
>> nothing else i can see how it could all work but i see now why you see
>> it could be a very small project David's plugin system has mostly
>> everything done for you but i would like to add to his plugin for some
>> tree access code etc...
>>
>> Adding a config file for telling the plugin to not care about certain
>> things would be a nice addition. It seems interesting the clang idea,
>> it could be interesting porting this to other front-ends of gcc like
>> gccgo.
>
>
> Does gccgo use the C ABI so that Cython could call it? If so, go for it!
>
> (Fortran is actually very much in use in the Cython userbase and would get a
> lot more "customers" than Go, but if you have more of a CS background or
> similar I can see why you wouldn't be so interested in Fortran. I didn't
> believe people were still using Fortran either until I started doing
> astrophysics, and suddenly it seems to be the default tool everybody uses
> for everything.)
>
> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From sccolbert at gmail.com  Thu Mar 29 16:40:58 2012
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 29 Mar 2012 09:40:58 -0500
Subject: [Cython] Gsoc project
In-Reply-To: <CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
	<4F73D6F6.1010402@astro.uio.no>
	<CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>
Message-ID: <CAPc_zRX-pEA4V8bcCdTwZF0Jyn7fMgT4gZeN0JnxBMoQ6tVBXQ@mail.gmail.com>

Are you all aware of CWrap? It goes a long way to generating pxd files for
a C libary. It can, for example, generate valid pxd files for the whole of
the Intel IPP library.

This would likely be a good launching point for a GSOC project, unless you
really want to start from scratch.

https://github.com/enthought/cwrap

It's BSD licensed.


On Thu, Mar 29, 2012 at 5:10 AM, mark florisson
<markflorisson88 at gmail.com>wrote:

> On 29 March 2012 04:28, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
> wrote:
> > On 03/28/2012 07:58 PM, Philip Herron wrote:
> >>
> >> Hey all
> >>
> >> I am implemented a very crude and simplistic and very badly programmed
> >> version of a pxd generator i think i understand what were after now
> >> but i would appreciate if you look over what i did to make sure i have
> >> grasped the basic idea for now:
> >>
> >> So if i have:
> >>
> >> #include "test.h"
> >>
> >> int add (int x, int y)
> >> {
> >>   return x + y;
> >> }
> >>
> >> #ifndef __TEST_H__
> >> #define __TEST_H__
> >>
> >> extern int add (int, int);
> >>
> >> struct foobar {
> >>   int a;
> >>   unsigned char * b;
> >> };
> >>
> >> #endif //__TEST_H_
> >>
> >> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py
> test.c
> >>
> >> And i output out.pxd:
> >>
> >> cdef extern from "test.h":
> >>        int add (int, int)
> >>
> >>        ctypedef struct foobar:
> >>                int a
> >>                unsigned char * b
> >
> >
> > Nice. But please use 4 spaces (see PEP 8) :-)
> >
> > More ideas for project proposal:
> >
> > Another slight complication is that you should ideally turn
> >
> > #define FOO 3
> > #define BAR 4
> >
> > into
> >
> > cdef extern from "foo.h":
> >    enum:
> >        FOO
> >        BAR
> >
> > so you need to hook in before the preprocessor and after the preprocessor
> > and dig out different stuff.
>
> David, I'm CCing you as this might be of interest to you.
>
> I think the current GCC plugin support doesn't allow you to do much
> with te preprocessor, it operates entirely after the C preprocessor
> has run. So to support macros we have to consider that for this to
> work the gcc plugin may have to be extended, which uses C to extend
> GCC and Python, so it also requires knowledge of the CPython C API.
> David, would you mind elaborating why C was used for this project and
> not (partially) Cython, and would it be possible to extend the plugin
> with Cython?
>
> > Then what happens if you have
> >
> > #ifdef FOO
> > #define BAR 3
> > #else
> > #define BAR 4
> > #endif
> >
> > ?? I'm not saying it is hard, but perhaps no longer completely trivial
> :-)
> >
> > And like Robert hinted at, supporting all the aspects of C++ might take a
> > little more work, there's just so much different syntax in C++, and
> there's
> > several C++ features Cython just does not support and must be either
> ignored
> > or hacked around (e.g., "const").
> >
> > Supporting stuff like
> >
> > #define MACRO(x) ((x)->field*2)
> >
> > probably belongs in the category of "must be done manually".
> >
> >
> >>
> >> So far in a very crude way it understands structs and functions but
> >> nothing else i can see how it could all work but i see now why you see
> >> it could be a very small project David's plugin system has mostly
> >> everything done for you but i would like to add to his plugin for some
> >> tree access code etc...
> >>
> >> Adding a config file for telling the plugin to not care about certain
> >> things would be a nice addition. It seems interesting the clang idea,
> >> it could be interesting porting this to other front-ends of gcc like
> >> gccgo.
> >
> >
> > Does gccgo use the C ABI so that Cython could call it? If so, go for it!
> >
> > (Fortran is actually very much in use in the Cython userbase and would
> get a
> > lot more "customers" than Go, but if you have more of a CS background or
> > similar I can see why you wouldn't be so interested in Fortran. I didn't
> > believe people were still using Fortran either until I started doing
> > astrophysics, and suddenly it seems to be the default tool everybody uses
> > for everything.)
> >
> > Dag
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120329/cf2e9732/attachment.html>

From dmalcolm at redhat.com  Thu Mar 29 18:25:28 2012
From: dmalcolm at redhat.com (David Malcolm)
Date: Thu, 29 Mar 2012 12:25:28 -0400
Subject: [Cython] Gsoc project
In-Reply-To: <CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
	<4F73D6F6.1010402@astro.uio.no>
	<CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>
Message-ID: <1333038328.31165.44.camel@surprise>

On Thu, 2012-03-29 at 11:10 +0100, mark florisson wrote:

Thanks for CCing me; various comments inline below throughout.

> On 29 March 2012 04:28, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> > On 03/28/2012 07:58 PM, Philip Herron wrote:
> >>
> >> Hey all
> >>
> >> I am implemented a very crude and simplistic and very badly programmed
> >> version of a pxd generator i think i understand what were after now
> >> but i would appreciate if you look over what i did to make sure i have
> >> grasped the basic idea for now:

[...snip example sources...]

> >> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c

FWIW, the plugin has a helper script, so that you ought to be able to
simply run:
  ./gcc-with-python walk.py test.c
(paths permitting)

My primary use-case for the plugin is my libcpychecker code which
implements static analysis of refcount-handling, and for that I have
another helper script "gcc-with-cpychecker" that invokes my code so that
you can simply run:
  ./gcc-with-cpychecker -I/usr/include/python2.7 test.c

So you might want to do something similar for the pxd generation.

[...snip sample output...]

> > Another slight complication is that you should ideally turn
> >
> > #define FOO 3
> > #define BAR 4
> >
> > into
> >
> > cdef extern from "foo.h":
> >    enum:
> >        FOO
> >        BAR
> >
> > so you need to hook in before the preprocessor and after the preprocessor
> > and dig out different stuff.
> 
> David, I'm CCing you as this might be of interest to you.

Very much so - thanks!  (Hi everyone!)

FWIW, I happened to see Dag's earlier email via a google search, and
added the Cython idea to the list of "Ideas for using the GCC plugin"
here:
http://gcc-python-plugin.readthedocs.org/en/latest/getting-involved.html#ideas-for-using-the-plugin

> I think the current GCC plugin support doesn't allow you to do much
> with te preprocessor, it operates entirely after the C preprocessor
> has run. 
So far, yes.  I haven't explored GCC's C frontend as much as I have the
stages that follow.  The C preprocessor does run in-process; I don't
know yet to what extent it's amenable to hacking via a GCC plugin.  I
believe that aspects of its integration may have been rewritten somewhat
in GCC 4.7 (some of my colleagues tried to improve the line-numbering
capture in the presence of macros).

> So to support macros we have to consider that for this to
> work the gcc plugin may have to be extended, which uses C to extend
> GCC and Python, so it also requires knowledge of the CPython C API.
Yes; I'd expect you to have to go digging into the guts of the GCC C
preprocessor implementation, using GDB.

I don't know yet how feasible it is to get at the data from a plugin: it
might be anywhere from "easy" to "impossible".  You might need to get a
patch into GCC to expose the necessary information (if so, that would
probably be worthy of a GSoC slot, I think).

One issue is that although GCC has an API for plugins to use to register
themselves, it doesn't yet have an official API for plugins to use for
doing anything else, so we're somewhat at the mercy of future GCC
developments (hopefully Python will make it easier to survive future
internal interface changes though).

BTW, the Python plugin's API isn't 100% frozen yet: I still reserve the
right to tweak things if appropriate (I've only done this occasionally
though, and I've gone through all the code I know of when I do to
doublecheck if I'm about to break something).

> David, would you mind elaborating why C was used for this project and
> not (partially) Cython, and would it be possible to extend the plugin
> with Cython?
I did initial try using Cython: see early commits here:
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=4d62721d519008c325d7369f1330dc09080c0b51
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=9b5145955c823453404c49e4b295e8c739c5ff44

but GCC internals are just too, err, "baroque" (that's a euphemism): it
makes very heavy use of the C preprocessor (e.g. *all* field accesses go
through an access macro; there are garbage-collection annotations
thoughout); many of the types are declared by repeatedly
#include-ing .def files using macro definitions to expand the contents
in a variety of ways.

> > Then what happens if you have
> >
> > #ifdef FOO
> > #define BAR 3
> > #else
> > #define BAR 4
> > #endif
> >
> > ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
Yeah.  I have no idea to what extent the C preprocessor stuff is exposed
internally, and if the branching logic is preserved in any way that's
usable.

[...snip...]

> > Does gccgo use the C ABI so that Cython could call it? If so, go for it!
> >
> > (Fortran is actually very much in use in the Cython userbase and would get a
> > lot more "customers" than Go, but if you have more of a CS background or
> > similar I can see why you wouldn't be so interested in Fortran. I didn't
> > believe people were still using Fortran either until I started doing
> > astrophysics, and suddenly it seems to be the default tool everybody uses
> > for everything.)
I downloaded Philip's script from
http://mail.python.org/pipermail/cython-devel/attachments/20120329/cdeb9453/attachment.py

It's running immediately before "free_lang_data", which is the first
interprocedural "whole-file" optimization pass, after some per-function
passes have been run.

You can see a map of the passes here:
http://gcc-python-plugin.readthedocs.org/en/latest/tables-of-passes.html

[See also
http://gcc-python-plugin.readthedocs.org/en/latest/callbacks.html#gcc.PLUGIN_PASS_EXECUTION
for notes on how the sample code I showed Dag at PyCon works]

So my guess is that this code can be run for *all* languages that GCC
can handle: all of the language frontends feed in data near the top of
that map: so in theory this ought to work for Fortran, C++, Go, etc.

Having said that, I've been trying to get my libcpychecker code running
on C++ and I keep running into subtle difference in the exact data they
generate: e.g. the C++ frontend seems to add Nop statements for empty
functions, whereas the C frontend doesn't; type declarations get hidden
inside namespace objects in the C++ frontend; etc etc.

BTW, some stylistic nits on Philip's script:
   * don't match types based on strings: c.f.:
     if T == "<type 'gcc.FunctionDecl'>":
instead, use isinstance:
      if isinstance(decl.type, gcc.FunctionDecl)
so that you're not relying on repr() or str(), and so you match
subclasses, not just one class

   * "decl_location_get_file (decl)" jumps through lots of hoops to get
at the filename of a decl.location by parsing the repr().  But you can
simply look at the decl.location.file attribute:
http://gcc-python-plugin.readthedocs.org/en/latest/basics.html#gcc.Location.file

 * similar considerations apply to decl_identifier_node_to_string();
have a look at the dir() of the object (and if something is not
documented, file a bug, or a patch!).

Hope this is helpful; good luck!
Dave


From njs at pobox.com  Thu Mar 29 19:32:19 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 29 Mar 2012 18:32:19 +0100
Subject: [Cython] Gsoc project
In-Reply-To: <1333038328.31165.44.camel@surprise>
References: <CAEvRbeoyYrZvoMh9d7a1LR31SH1aiREbzxC1gKSz03W0MEW5xg@mail.gmail.com>
	<4F727FF5.5010909@astro.uio.no>
	<CADiQ+QCQx=UOonT75F_eptdLAweoea=mWv==v6t=vm2n=j9tTQ@mail.gmail.com>
	<CAEvRbeoTOhXyuezu=66tdPtLcX6qtsZf23TTz=_VAE5qdaNMuQ@mail.gmail.com>
	<4F73D6F6.1010402@astro.uio.no>
	<CANg26EWq6x3GQT+YeT7QOMpt7TK52XZeAAF4R==nneNJDx7jSQ@mail.gmail.com>
	<1333038328.31165.44.camel@surprise>
Message-ID: <CAPJVwB=VvLS7d2gbv_V7DtEQpVrvgz_2uGVC4QyunLvj9=UD=g@mail.gmail.com>

On Thu, Mar 29, 2012 at 5:25 PM, David Malcolm <dmalcolm at redhat.com> wrote:
> On Thu, 2012-03-29 at 11:10 +0100, mark florisson wrote:
>> > Then what happens if you have
>> >
>> > #ifdef FOO
>> > #define BAR 3
>> > #else
>> > #define BAR 4
>> > #endif
>> >
>> > ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
> Yeah. ?I have no idea to what extent the C preprocessor stuff is exposed
> internally, and if the branching logic is preserved in any way that's
> usable.

I'm pretty sure macros and such are gone before the C parser ever gets
to see the token stream.

But, it should be easy to pull out #define's on a second pass, using
something like
  gcc $CFLAGS -E -dM myfile.h
and then just scanning the output with a trivial "parser". (I'm not
sure which of -dM, -dD, -dU you want.)

-- Nathaniel

From stefan_ml at behnel.de  Fri Mar 30 13:19:37 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 30 Mar 2012 13:19:37 +0200
Subject: [Cython] weird code in argument unpacking (memoryview related?)
Message-ID: <4F7596C9.1000802@behnel.de>

Hi,

this code in Nodes.py around line 3950, at the end of the DefNodeWrapper's
method generate_keyword_unpacking_code(), was added as part of the memory
view changes, back in July last year:

"""
        # convert arg values to their final type and assign them
        for i, arg in enumerate(all_args):
            if arg.default and not arg.type.is_pyobject:
                code.putln("if (values[%d]) {" % i)
            if arg.default and not arg.type.is_pyobject:
                code.putln('} else {')
                code.putln(
                    "%s = %s;" % (
                        arg.entry.cname,
                        arg.calculate_default_value_code(code)))
                if arg.type.is_memoryviewslice:
                    code.put_incref_memoryviewslice(arg.entry.cname,
                                                    have_gil=True)
                code.putln('}')
"""

By being overly complicated, it hides rather well what it's actually meant
to achieve. It doesn't do at all what its comment says and also refers to
the default value of the argument, which is already assigned way before
this place in the code. Therefore, I'd be surprised if it generated
anything but dead C code, because the case that "values[i]" is NULL should
never happen.

Mark, could you comment on this? Is this just a left-over from a broken
merge or something?

Stefan

From markflorisson88 at gmail.com  Fri Mar 30 14:25:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 30 Mar 2012 13:25:28 +0100
Subject: [Cython] weird code in argument unpacking (memoryview related?)
In-Reply-To: <4F7596C9.1000802@behnel.de>
References: <4F7596C9.1000802@behnel.de>
Message-ID: <CANg26EUhFXnCx1ixCSATeMgZ3jorqNSYMch-UmRtkUoY=_uD_w@mail.gmail.com>

On 30 March 2012 12:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> this code in Nodes.py around line 3950, at the end of the DefNodeWrapper's
> method generate_keyword_unpacking_code(), was added as part of the memory
> view changes, back in July last year:
>
> """
> ? ? ? ?# convert arg values to their final type and assign them
> ? ? ? ?for i, arg in enumerate(all_args):
> ? ? ? ? ? ?if arg.default and not arg.type.is_pyobject:
> ? ? ? ? ? ? ? ?code.putln("if (values[%d]) {" % i)
> ? ? ? ? ? ?if arg.default and not arg.type.is_pyobject:
> ? ? ? ? ? ? ? ?code.putln('} else {')
> ? ? ? ? ? ? ? ?code.putln(
> ? ? ? ? ? ? ? ? ? ?"%s = %s;" % (
> ? ? ? ? ? ? ? ? ? ? ? ?arg.entry.cname,
> ? ? ? ? ? ? ? ? ? ? ? ?arg.calculate_default_value_code(code)))
> ? ? ? ? ? ? ? ?if arg.type.is_memoryviewslice:
> ? ? ? ? ? ? ? ? ? ?code.put_incref_memoryviewslice(arg.entry.cname,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?have_gil=True)
> ? ? ? ? ? ? ? ?code.putln('}')
> """
>
> By being overly complicated, it hides rather well what it's actually meant
> to achieve. It doesn't do at all what its comment says and also refers to
> the default value of the argument, which is already assigned way before
> this place in the code. Therefore, I'd be surprised if it generated
> anything but dead C code, because the case that "values[i]" is NULL should
> never happen.
>
> Mark, could you comment on this? Is this just a left-over from a broken
> merge or something?

Hm, IIRC I was fixing acquisition count errors for memoryview slices
(of which there were many) for arguments with default values, I was
cleaning up after the initial memoryview implementation.

All I did was add another case for memoryview slices in old code (I
only wrote the last 3 lines, you wrote the other ones in 2008). Very
similar but more recent code (probably copied and pasted) can be found
around line ~3730 in generate_tuple_and_keyword_parsing_code (instead
of generate_keyword_unpacking_code).

There are tests for default arguments and acquisition count errors
should fail immediately and hard, so I trust your judgement to remove
any or all of this.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Mar 30 17:22:05 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 30 Mar 2012 17:22:05 +0200
Subject: [Cython] weird code in argument unpacking (memoryview related?)
In-Reply-To: <CANg26EUhFXnCx1ixCSATeMgZ3jorqNSYMch-UmRtkUoY=_uD_w@mail.gmail.com>
References: <4F7596C9.1000802@behnel.de>
	<CANg26EUhFXnCx1ixCSATeMgZ3jorqNSYMch-UmRtkUoY=_uD_w@mail.gmail.com>
Message-ID: <4F75CF9D.3040801@behnel.de>

mark florisson, 30.03.2012 14:25:
> On 30 March 2012 12:19, Stefan Behnel wrote:
>> this code in Nodes.py around line 3950, at the end of the DefNodeWrapper's
>> method generate_keyword_unpacking_code(), was added as part of the memory
>> view changes, back in July last year:
>>
>> """
>>        # convert arg values to their final type and assign them
>>        for i, arg in enumerate(all_args):
>>            if arg.default and not arg.type.is_pyobject:
>>                code.putln("if (values[%d]) {" % i)
>>            if arg.default and not arg.type.is_pyobject:
>>                code.putln('} else {')
>>                code.putln(
>>                    "%s = %s;" % (
>>                        arg.entry.cname,
>>                        arg.calculate_default_value_code(code)))
>>                if arg.type.is_memoryviewslice:
>>                    code.put_incref_memoryviewslice(arg.entry.cname,
>>                                                    have_gil=True)
>>                code.putln('}')
>> """
>>
>> By being overly complicated, it hides rather well what it's actually meant
>> to achieve. It doesn't do at all what its comment says and also refers to
>> the default value of the argument, which is already assigned way before
>> this place in the code. Therefore, I'd be surprised if it generated
>> anything but dead C code, because the case that "values[i]" is NULL should
>> never happen.
>>
>> Mark, could you comment on this? Is this just a left-over from a broken
>> merge or something?
> 
> Hm, IIRC I was fixing acquisition count errors for memoryview slices
> (of which there were many) for arguments with default values, I was
> cleaning up after the initial memoryview implementation.
> 
> All I did was add another case for memoryview slices in old code (I
> only wrote the last 3 lines, you wrote the other ones in 2008).

Well, the entire block appeared in your commit (and it looked suspiciously
like my own code in retrospect), so I assumed that you copied it from
somewhere or that it accidentally slipped in while handling a merge conflict.


> Very
> similar but more recent code (probably copied and pasted) can be found
> around line ~3730 in generate_tuple_and_keyword_parsing_code (instead
> of generate_keyword_unpacking_code).

Right - that one makes sense. The "values[i] == NULL" case is actually used
for non-Python default values. I'll add a comment.

The first method calls the latter, so the correct code is being executed
after the block in question. That makes it look like the one above is just
a redundant quirk. The fact that it went undetected should hint at a
missing test or missing refnanny usage, though. A duplicated incref would
imply a reference leak, and the fact that it only affects default arguments
would make it very hard to detect. Might be visible by using memory views
as default arguments in closures, now that Vitja's fix correctly handles those.


> There are tests for default arguments and acquisition count errors
> should fail immediately and hard, so I trust your judgement to remove
> any or all of this.

Given that the correct code is in the right place, I've deleted the copy. I
did it in a branch because I was rewriting a part of that code anyway, so
any attempt to fix it directly in the master would just unnecessarily lead
to collisions. The whole code is here, commit #6 being the cleanup:

https://github.com/cython/cython/pull/107

Stefan

From markflorisson88 at gmail.com  Fri Mar 30 19:24:21 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 30 Mar 2012 18:24:21 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <4F71AB7B.7090007@behnel.de>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
Message-ID: <CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>

On 27 March 2012 12:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 27.03.2012 13:20:
>> I tested the release in my own branch and jenkins was blue, but
>> the release build seems to disagree.
>
> The release (and master) branch is tested against the "-ext" builds of
> CPython, which have some external packages installed, including NumPy. You
> have to change the config of your "tests" job to use those instead of the
> plain CPython builds. The build job can stay as it is.
>
> As for the inner workings, there's a normal "pyXY-hg" job to build CPython
> and a corresponding "pyXY-ext-hg" job that takes the build and installs a
> list of packages into it, then creates a new install archive. You can then
> reference either of the two archives in your build/test jobs by asking for
> a "pyXY" or "pyXY-ext" Python.
>
>
>> The py32 C++ build shows some refcount error:
>>
>> numpy_memoryview.cpp:16922: warning: ?void
>> __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject*, Py_buffer*)?
>> defined but not used
>
> These are legitimate warnings that are worth fixing (at some point), I
> think. They seem to originate from the buffer implementation in numpy.pxd.
> Those sort-of-external special methods shouldn't lead to the generation of
> a Python wrapper function.
>
>
>> python: Modules/gcmodule.c:327: visit_decref: Assertion
>> `gc->gc.gc_refs != 0' failed.
>
> It's surprising that that only occurs in one out of four Py3 test
> configurations. Maybe there's something indeterministic in those tests?
>
>
>> and some of the other python 3 tests are also failing.
>
> Yep, some of them look really funny (complaining about getting exactly the
> expected output), others are the typical Py3 problems (e.g. printing bytes).
>
> Note that the reason the py3k tests are not impacted is that it does not
> have NumPy. So the tests would equally fail in all Py3 versions.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

The release build looks good now, I'm thinking of pushing a second and
final beta out there tomorrow. If anyone wants to get something in,
now is the time to raise voice.

From robertwb at gmail.com  Fri Mar 30 19:50:52 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 30 Mar 2012 10:50:52 -0700
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
Message-ID: <CADiQ+QBc2+ejj8s3+_8VHB0x=fkN113zf9+KD4mH+OZ+rQGbtw@mail.gmail.com>

On Fri, Mar 30, 2012 at 10:24 AM, mark florisson
<markflorisson88 at gmail.com> wrote:

> The release build looks good now, I'm thinking of pushing a second and
> final beta out there tomorrow. If anyone wants to get something in,
> now is the time to raise voice.

Excellent. I personally don't know of anything that can't wait 'till
0.16.1. If you're confident enough, let's call it a release candidate
:).

- Robert

From vitja.makarov at gmail.com  Fri Mar 30 20:12:11 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 30 Mar 2012 22:12:11 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QBc2+ejj8s3+_8VHB0x=fkN113zf9+KD4mH+OZ+rQGbtw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
	<CADiQ+QBc2+ejj8s3+_8VHB0x=fkN113zf9+KD4mH+OZ+rQGbtw@mail.gmail.com>
Message-ID: <CAKGHGPQr4o+gu6tupMizFnr+6RmV3hit5xk=GuLvFNt58CiwCg@mail.gmail.com>

2012/3/30 Robert Bradshaw <robertwb at gmail.com>:
> On Fri, Mar 30, 2012 at 10:24 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>
>> The release build looks good now, I'm thinking of pushing a second and
>> final beta out there tomorrow. If anyone wants to get something in,
>> now is the time to raise voice.
>
> Excellent. I personally don't know of anything that can't wait 'till
> 0.16.1. If you're confident enough, let's call it a release candidate
> :).
>

I hope we're done with regressions now. I'm +1 for RC and feature freeze.

-- 
vitja.

From stefan_ml at behnel.de  Sat Mar 31 11:14:41 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 31 Mar 2012 11:14:41 +0200
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
Message-ID: <4F76CB01.8090800@behnel.de>

mark florisson, 30.03.2012 19:24:
> The release build looks good now, I'm thinking of pushing a second and
> final beta out there tomorrow. If anyone wants to get something in,
> now is the time to raise voice.

Note that Dag's NumPy specialisation broke the Sage build by introducing a
compiler crash.

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/sage-build/1064/consoleFull

The problem is that the new code is already triggered during type inference
(i.e. before type analysis) and then crashes the compiler because it
accesses types that are not known yet.

Stefan

From markflorisson88 at gmail.com  Sat Mar 31 11:31:24 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 31 Mar 2012 10:31:24 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <4F76CB01.8090800@behnel.de>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
	<4F76CB01.8090800@behnel.de>
Message-ID: <CANg26EVrD5yD70EhPb5_Lv30oh4JJ6Hd=FVq_GDJ4x=QQN=EyA@mail.gmail.com>

On 31 March 2012 10:14, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 30.03.2012 19:24:
>> The release build looks good now, I'm thinking of pushing a second and
>> final beta out there tomorrow. If anyone wants to get something in,
>> now is the time to raise voice.
>
> Note that Dag's NumPy specialisation broke the Sage build by introducing a
> compiler crash.
>
> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/sage-build/1064/consoleFull
>
> The problem is that the new code is already triggered during type inference
> (i.e. before type analysis) and then crashes the compiler because it
> accesses types that are not known yet.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Thanks for pointing that out Stefan. It would probably work if it
passed in 'obj_type' to the numpy_transform_attribute_node, and used
that instead of node.type. Alternatively, it could just wait until the
type is set through analyse_expressions.

From stefan_ml at behnel.de  Sat Mar 31 18:15:42 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 31 Mar 2012 18:15:42 +0200
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EVrD5yD70EhPb5_Lv30oh4JJ6Hd=FVq_GDJ4x=QQN=EyA@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
	<4F76CB01.8090800@behnel.de>
	<CANg26EVrD5yD70EhPb5_Lv30oh4JJ6Hd=FVq_GDJ4x=QQN=EyA@mail.gmail.com>
Message-ID: <4F772DAE.8090205@behnel.de>

mark florisson, 31.03.2012 11:31:
> On 31 March 2012 10:14, Stefan Behnel wrote:
>> mark florisson, 30.03.2012 19:24:
>>> The release build looks good now, I'm thinking of pushing a second and
>>> final beta out there tomorrow. If anyone wants to get something in,
>>> now is the time to raise voice.
>>
>> Note that Dag's NumPy specialisation broke the Sage build by introducing a
>> compiler crash.
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/sage-build/1064/consoleFull
>>
>> The problem is that the new code is already triggered during type inference
>> (i.e. before type analysis) and then crashes the compiler because it
>> accesses types that are not known yet.
> 
> Thanks for pointing that out Stefan. It would probably work if it
> passed in 'obj_type' to the numpy_transform_attribute_node, and used
> that instead of node.type. Alternatively, it could just wait until the
> type is set through analyse_expressions.

Thanks for fixing them. The release branch seems to be in good shape for a
final beta release now.

On a related note, the current master looks more like a 0.17 to me than a
0.16.1 when I compare it to the current release branch. We could just wait
for a couple of weeks to collect bugs in 0.16 from user reports (and
features as they come in), and then push out a 0.17. I hope the pending
PyPy changes will also be ready enough for a release by then, that would
round up the feature list quite nicely.

Stefan

From markflorisson88 at gmail.com  Sat Mar 31 18:24:30 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 31 Mar 2012 17:24:30 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <4F772DAE.8090205@behnel.de>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
	<CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>
	<CAKGHGPRO_S-y11zt3zTjT_6XU2_LYovQ=6u4VOwdVn5pe-TZKQ@mail.gmail.com>
	<CANg26EViPWNa23WiooHdWu9paKKDmOmWMaz02MvKwpeZVw8eGg@mail.gmail.com>
	<4F71AB7B.7090007@behnel.de>
	<CANg26EVqhkTgRRrNnm3XeO46hNvVTcPxApoFYpQsG8ksEeG8Fw@mail.gmail.com>
	<4F76CB01.8090800@behnel.de>
	<CANg26EVrD5yD70EhPb5_Lv30oh4JJ6Hd=FVq_GDJ4x=QQN=EyA@mail.gmail.com>
	<4F772DAE.8090205@behnel.de>
Message-ID: <CANg26EXk6ocd=+wOOZd6n2RE=Jw2FZMtkC6Zx7y=R=AR+4=BHg@mail.gmail.com>

On 31 March 2012 17:15, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 31.03.2012 11:31:
>> On 31 March 2012 10:14, Stefan Behnel wrote:
>>> mark florisson, 30.03.2012 19:24:
>>>> The release build looks good now, I'm thinking of pushing a second and
>>>> final beta out there tomorrow. If anyone wants to get something in,
>>>> now is the time to raise voice.
>>>
>>> Note that Dag's NumPy specialisation broke the Sage build by introducing a
>>> compiler crash.
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/sage-build/1064/consoleFull
>>>
>>> The problem is that the new code is already triggered during type inference
>>> (i.e. before type analysis) and then crashes the compiler because it
>>> accesses types that are not known yet.
>>
>> Thanks for pointing that out Stefan. It would probably work if it
>> passed in 'obj_type' to the numpy_transform_attribute_node, and used
>> that instead of node.type. Alternatively, it could just wait until the
>> type is set through analyse_expressions.
>
> Thanks for fixing them. The release branch seems to be in good shape for a
> final beta release now.
>
> On a related note, the current master looks more like a 0.17 to me than a
> 0.16.1 when I compare it to the current release branch. We could just wait
> for a couple of weeks to collect bugs in 0.16 from user reports (and
> features as they come in), and then push out a 0.17. I hope the pending
> PyPy changes will also be ready enough for a release by then, that would
> round up the feature list quite nicely.

Indeed, I was getting ready for a release, but I almost forgot to
merge over some fixes from the _fused_dispatch branch (to correctly
specialize np.ndarray and C++ templates). So I'm doing that now, which
also exposed a deepcopy python 2.4 bug. I'm planning an RC for
tonight.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sat Mar 31 22:59:49 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 31 Mar 2012 21:59:49 +0100
Subject: [Cython] Cython 0.16 Release Candidate
Message-ID: <CANg26EVzSDPqQ2_iyD2tEEAtJQ5jtzhYPnhgGv5-fzzqDaYBSw@mail.gmail.com>

We are happy to announce the first release candidate for Cython 0.16,
you can grab it from here: http://wiki.cython.org/ReleaseNotes-0.16

Since the beta last month several issues have been fixed, including
the NumPy array attribute access deprecation, which are now rewritten
to use the NumPy macros. Another serious bug in the reversed(range())
optimization was disabled, a C compiler bug on OS X Lion in OpenMP
code was circumvented, the documentation for fused types has been
improved, some bugs related to fused C++ and buffers types have been
fixed, and nogil was added to the included STL declarations.

I encourage everyone to give it a try, see if your project still
compiles and works, and if it doesn't to take up those torches and
pitchforks from your desks, you know where to find us.

Enjoy!