From dalcinl at gmail.com  Fri Apr  1 03:11:19 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 31 Mar 2011 22:11:19 -0300
Subject: [Cython] Cannot profile nogil function. Error or Warn?
In-Reply-To: <AANLkTinQ=wckoXWTcOL0p2BJ53Ke416OvNDYExk85HW3@mail.gmail.com>
References: <AANLkTinQ=wckoXWTcOL0p2BJ53Ke416OvNDYExk85HW3@mail.gmail.com>
Message-ID: <AANLkTinb4d58NPGNdV4M3Ug3YB6dyhU5ZzMe8wL9A-HL@mail.gmail.com>

On 29 March 2011 21:26, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>
> cdef int PyMPE_Raise(int ierr) except -1 with gil:
> ? ?__Pyx_Raise(RuntimeError, "MPE logging error [code: %d]" % ierr, NULL)
> ? ?return 0
>
> cdef inline int CHKERR(int ierr) nogil except -1:
> ? ?^
> ------------------------------------------------------------
>
> /home/dalcinl/Devel/mpi4py-dev/src/MPE/helpers.pxi:22:5: Cannot
> profile nogil function.
>
>
> Do we REALLY want this to be an error? Why not just a warning?
>

OK, I pushed a fix. Without this, using -X profile=True cannot work
with any pyx source that has nogil functions. Enabling profiling
should not force users to change source code.



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Fri Apr  1 10:11:56 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 01 Apr 2011 10:11:56 +0200
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
Message-ID: <4D9588CC.6000303@behnel.de>

Arthur de Souza Ribeiro, 29.03.2011 09:11:
> Hello everybody,
>
> My name is Arthur de Souza Ribeiro and I'm a fourth-year student of Computer
> Science in Federal University of Campina Grande, Brazil. I'm a python
> programmer and have knowledge of other languages too, like Java, C, C++, Qt,
> Grails and ActionScript (used in Flex framework of Adobe).
>
> I saw Cython project and got really interested in contributing to it. By the
> way, I saw that the project is trying to participate of GSoC under Python
> Software Foundation umbrella. I know the student application period have
> already started, but, I'd really enjoy to participate of GSoC 2011 as a
> Cython's student. Until day 8 I could work really hard to show you that I
> can be selected as a GSoC student for Cython. I looked for an Ideas Page of
> the project but didn't find it, Is there any idea that you have to submit a
> project in GSoC?
>
> If possible, please tell me things that I can start doing to help the
> project.

Hi Arthur,

sorry for the late response and thank you for your application. We are 
always happy about contributions.

The Cython project is currently running a workshop that may yield further 
possible GSoC tasks, but the one we already have identified is IMHO quite a 
nice and self-contained one. The goal is to rewrite modules in CPython's 
standard library in Cython that are currently written in C. The intention 
is a) to simplify the implementation to make it easier for CPython 
developers to maintain their code base and b) to try to make the modules 
even faster than they are to show off Cython's optimisation capabilities 
(in that order, I think).

A related task could be to take existing Python modules in the stdlib, to 
profile them, and to add external type annotations to optimise them when 
being compiled with Cython.

Both the task of showing Cython's ability to efficiently (and compatibly) 
implement or compile parts of the stdlib, and the resulting testing of 
Cython (and bug reporting/fixing) against real world Python code would be 
very valuable to our project.

If you're interested, you could start by writing a short proposal including 
the modules that you would like to rewrite and what makes them interesting. 
Both "itertools" and "math" are certainly hot candidates, but there are 
definitely others, and your interest may change the priorities.

If you think that's not a good project for you, please bug us again, we may 
be able to come up with other projects as well.

Stefan

From dalcinl at gmail.com  Fri Apr  1 18:57:08 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Apr 2011 13:57:08 -0300
Subject: [Cython] implementation of cdef functions with default arguments
Message-ID: <BANLkTi=Sir3u_LFGvkeCR-+opCyoHr404A@mail.gmail.com>

Perhaps I'm missing something, but why we need the intermediate
struct? Why not generate a regular C function with all the args, and
then generate the call providing arguments? We could even extend this
to support kwargs for calling functions in Cython,

1 - The implementation would be cleaner, IMHO.
2 - These functions cannot be easily used in external C code (or
course, C code should provide all the args)
3 - We could define default args for "cdef extern" C functions, Cython
would provide the arg values on call.
4 - We could add support to pass values as kwargs (well, we could do
that with the current implementation).
5 - Faster code?

Comments?

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From arthurdesribeiro at gmail.com  Sat Apr  2 03:52:40 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Fri, 1 Apr 2011 22:52:40 -0300
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <4D9588CC.6000303@behnel.de>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
	<4D9588CC.6000303@behnel.de>
Message-ID: <AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>

HI Stefan, thank you very much for responding my e-mail to cython's list.

About the proposal, I'd be very happy in helping the cython community doing
the task 'rewrite modules in CPython's standard library in Cython that are
currently written in C'. I didn't think about any special modules, but I'm
going to start doing it, in my opinion, both modules you've mentioned are
really good examples.

I think this project could be very important, but, I don't know CPython very
well, are there any examples you could suggest me to understand CPython
better? I think I could do a good effort to understand this as fast as I can
and we discuss more the proposal.

Waiting for your reply...

Best Regards..

[]s

Arthur

2011/4/1 Stefan Behnel <stefan_ml at behnel.de>

> Arthur de Souza Ribeiro, 29.03.2011 09:11:
>
>  Hello everybody,
>>
>> My name is Arthur de Souza Ribeiro and I'm a fourth-year student of
>> Computer
>> Science in Federal University of Campina Grande, Brazil. I'm a python
>> programmer and have knowledge of other languages too, like Java, C, C++,
>> Qt,
>> Grails and ActionScript (used in Flex framework of Adobe).
>>
>> I saw Cython project and got really interested in contributing to it. By
>> the
>> way, I saw that the project is trying to participate of GSoC under Python
>> Software Foundation umbrella. I know the student application period have
>> already started, but, I'd really enjoy to participate of GSoC 2011 as a
>> Cython's student. Until day 8 I could work really hard to show you that I
>> can be selected as a GSoC student for Cython. I looked for an Ideas Page
>> of
>> the project but didn't find it, Is there any idea that you have to submit
>> a
>> project in GSoC?
>>
>> If possible, please tell me things that I can start doing to help the
>> project.
>>
>
> Hi Arthur,
>
> sorry for the late response and thank you for your application. We are
> always happy about contributions.
>
> The Cython project is currently running a workshop that may yield further
> possible GSoC tasks, but the one we already have identified is IMHO quite a
> nice and self-contained one. The goal is to rewrite modules in CPython's
> standard library in Cython that are currently written in C. The intention is
> a) to simplify the implementation to make it easier for CPython developers
> to maintain their code base and b) to try to make the modules even faster
> than they are to show off Cython's optimisation capabilities (in that order,
> I think).
>
> A related task could be to take existing Python modules in the stdlib, to
> profile them, and to add external type annotations to optimise them when
> being compiled with Cython.
>
> Both the task of showing Cython's ability to efficiently (and compatibly)
> implement or compile parts of the stdlib, and the resulting testing of
> Cython (and bug reporting/fixing) against real world Python code would be
> very valuable to our project.
>
> If you're interested, you could start by writing a short proposal including
> the modules that you would like to rewrite and what makes them interesting.
> Both "itertools" and "math" are certainly hot candidates, but there are
> definitely others, and your interest may change the priorities.
>
> If you think that's not a good project for you, please bug us again, we may
> be able to come up with other projects as well.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110401/23cf3542/attachment.html>

From stefan_ml at behnel.de  Sat Apr  2 08:50:38 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 02 Apr 2011 08:50:38 +0200
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
	<4D9588CC.6000303@behnel.de>
	<AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>
Message-ID: <4D96C73E.4080600@behnel.de>

Hi Arthur,

Arthur de Souza Ribeiro, 02.04.2011 03:52:
> HI Stefan, thank you very much for responding my e-mail to cython's list.
>
> About the proposal, I'd be very happy in helping the cython community doing
> the task 'rewrite modules in CPython's standard library in Cython that are
> currently written in C'. I didn't think about any special modules, but I'm
> going to start doing it, in my opinion, both modules you've mentioned are
> really good examples.

Cool.


> I think this project could be very important, but, I don't know CPython very
> well, are there any examples you could suggest me to understand CPython
> better? I think I could do a good effort to understand this as fast as I can
> and we discuss more the proposal.

The nice thing about this task is that you don't have to be an expert of 
CPython's C-API, nor a core developer of Cython. You will have to read the 
C code of the modules, and you will have to look up and understand what the 
C-API calls in the code are doing, but most of them have rather 
understandable names.

However, you will have to program efficiently in Cython, and write fast 
code in it. Writing Cython code that is easy to read and maintain, and at 
the same time fast enough to replace the existing manually tuned C code is 
the challenging bit here.

So my advice would be to get going in Cython programming (take a look 
through our tutorials), and to start reading the source code of a couple of 
CPython stdlib modules to get an idea of what you need to translate.

It would certainly help your application if you could reimplement one 
reasonably sized and self-contained function in a stdlib C module of your 
choice, and present that on the cython-users mailing list to get feedback. 
A couple of benchmark or profiling results comparing it to the original 
CPython function would round this up very nicely.

Stefan

From arthurdesribeiro at gmail.com  Sun Apr  3 04:17:37 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Sat, 2 Apr 2011 23:17:37 -0300
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <4D96C73E.4080600@behnel.de>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
	<4D9588CC.6000303@behnel.de>
	<AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>
	<4D96C73E.4080600@behnel.de>
Message-ID: <AANLkTin_EXE-vo+vWz38BSGFG2DZvtndksqhQWqaF=Ne@mail.gmail.com>

Hi Stefan, well, i took a look at CPython's source code and as you said, if
we use Cython in there we could get a very more readable code without losing
performance (I suppose).

I took a look especially in cmathmodule.c that composes Python 3.2 source
code (more recent stable version). As you said, depending on how fast I
create the Cython code, we could add more modules (like socket one, for
example).

An example on how code would be more readable (in my opinion, please correct
me if I'm wrong) is that we wouldn't have to have configuration code about
what functions would compose the module, for example, in math module we
have:

static PyMethodDef cmath_methods[] = {
    {"acos",   cmath_acos,  METH_VARARGS, c_acos_doc},
    {"acosh",  cmath_acosh, METH_VARARGS, c_acosh_doc},
    {"asin",   cmath_asin,  METH_VARARGS, c_asin_doc},
    {"asinh",  cmath_asinh, METH_VARARGS, c_asinh_doc},
    {"atan",   cmath_atan,  METH_VARARGS, c_atan_doc},
    {"atanh",  cmath_atanh, METH_VARARGS, c_atanh_doc},
    {"cos",    cmath_cos,   METH_VARARGS, c_cos_doc},
    {"cosh",   cmath_cosh,  METH_VARARGS, c_cosh_doc},
    {"exp",    cmath_exp,   METH_VARARGS, c_exp_doc},
    {"isfinite", cmath_isfinite, METH_VARARGS, cmath_isfinite_doc},
    {"isinf",  cmath_isinf, METH_VARARGS, cmath_isinf_doc},
    {"isnan",  cmath_isnan, METH_VARARGS, cmath_isnan_doc},
    {"log",    cmath_log,   METH_VARARGS, cmath_log_doc},
    {"log10",  cmath_log10, METH_VARARGS, c_log10_doc},
    {"phase",  cmath_phase, METH_VARARGS, cmath_phase_doc},
    {"polar",  cmath_polar, METH_VARARGS, cmath_polar_doc},
    {"rect",   cmath_rect,  METH_VARARGS, cmath_rect_doc},
    {"sin",    cmath_sin,   METH_VARARGS, c_sin_doc},
    {"sinh",   cmath_sinh,  METH_VARARGS, c_sinh_doc},
    {"sqrt",   cmath_sqrt,  METH_VARARGS, c_sqrt_doc},
    {"tan",    cmath_tan,   METH_VARARGS, c_tan_doc},
    {"tanh",   cmath_tanh,  METH_VARARGS, c_tanh_doc},
    {NULL,              NULL}           /* sentinel */
};


static struct PyModuleDef cmathmodule = {
    PyModuleDef_HEAD_INIT,
    "cmath",
    module_doc,
    -1,
    cmath_methods,
    NULL,
    NULL,
    NULL,
    NULL
};

And the init function after it, as I saw in cython (and implemented some
examples), we would just have to implement the functions that the
compilation would generate the object files that would be imported.

But, I noticed a problem that may likely appears that is the configuration
part. I mean, Cython code is compiled differently than CPython's one right?
If yes, would you have an idea on how we could work on this?

Another stuff that I'm getting in trouble in this initial part is how we
would translate functions like PyArg_ParseTuple, any clue? I'm studing ways
to replace too.

As you suggested, I'm practicing Cython to create functions and get more and
more familiar with the language, so that I can create a very efficient
cython code that would meet our expectations. I'm also reading CPython's
code to see where cython can be applied.

Thank you.

Best Regards.

[]s

Arthur

2011/4/2 Stefan Behnel <stefan_ml at behnel.de>

> Hi Arthur,
>
> Arthur de Souza Ribeiro, 02.04.2011 03:52:
>
>  HI Stefan, thank you very much for responding my e-mail to cython's list.
>>
>> About the proposal, I'd be very happy in helping the cython community
>> doing
>> the task 'rewrite modules in CPython's standard library in Cython that are
>> currently written in C'. I didn't think about any special modules, but I'm
>> going to start doing it, in my opinion, both modules you've mentioned are
>> really good examples.
>>
>
> Cool.
>
>
>
>  I think this project could be very important, but, I don't know CPython
>> very
>> well, are there any examples you could suggest me to understand CPython
>> better? I think I could do a good effort to understand this as fast as I
>> can
>> and we discuss more the proposal.
>>
>
> The nice thing about this task is that you don't have to be an expert of
> CPython's C-API, nor a core developer of Cython. You will have to read the C
> code of the modules, and you will have to look up and understand what the
> C-API calls in the code are doing, but most of them have rather
> understandable names.
>
> However, you will have to program efficiently in Cython, and write fast
> code in it. Writing Cython code that is easy to read and maintain, and at
> the same time fast enough to replace the existing manually tuned C code is
> the challenging bit here.
>
> So my advice would be to get going in Cython programming (take a look
> through our tutorials), and to start reading the source code of a couple of
> CPython stdlib modules to get an idea of what you need to translate.
>
> It would certainly help your application if you could reimplement one
> reasonably sized and self-contained function in a stdlib C module of your
> choice, and present that on the cython-users mailing list to get feedback. A
> couple of benchmark or profiling results comparing it to the original
> CPython function would round this up very nicely.
>
> Stefan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110402/d5db1719/attachment.html>

From sturla at molden.no  Mon Apr  4 01:49:37 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 04 Apr 2011 01:49:37 +0200
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <AANLkTin_EXE-vo+vWz38BSGFG2DZvtndksqhQWqaF=Ne@mail.gmail.com>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>	<4D9588CC.6000303@behnel.de>	<AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>	<4D96C73E.4080600@behnel.de>
	<AANLkTin_EXE-vo+vWz38BSGFG2DZvtndksqhQWqaF=Ne@mail.gmail.com>
Message-ID: <4D990791.6080301@molden.no>

Den 03.04.2011 04:17, skrev Arthur de Souza Ribeiro:
>
> static PyMethodDef cmath_methods[] = {
>     {"acos",   cmath_acos,  METH_VARARGS, c_acos_doc},
>     {"acosh",  cmath_acosh, METH_VARARGS, c_acosh_doc},
>     {"asin",   cmath_asin,  METH_VARARGS, c_asin_doc},
>     {"asinh",  cmath_asinh, METH_VARARGS, c_asinh_doc},
>     {"atan",   cmath_atan,  METH_VARARGS, c_atan_doc},
>     {"atanh",  cmath_atanh, METH_VARARGS, c_atanh_doc},
>     {"cos",    cmath_cos,   METH_VARARGS, c_cos_doc},
>     {"cosh",   cmath_cosh,  METH_VARARGS, c_cosh_doc},
>     {"exp",    cmath_exp,   METH_VARARGS, c_exp_doc},
>     {"isfinite", cmath_isfinite, METH_VARARGS, cmath_isfinite_doc},
>     {"isinf",  cmath_isinf, METH_VARARGS, cmath_isinf_doc},
>     {"isnan",  cmath_isnan, METH_VARARGS, cmath_isnan_doc},
>     {"log",    cmath_log,   METH_VARARGS, cmath_log_doc},
>     {"log10",  cmath_log10, METH_VARARGS, c_log10_doc},
>     {"phase",  cmath_phase, METH_VARARGS, cmath_phase_doc},
>     {"polar",  cmath_polar, METH_VARARGS, cmath_polar_doc},
>     {"rect",   cmath_rect,  METH_VARARGS, cmath_rect_doc},
>     {"sin",    cmath_sin,   METH_VARARGS, c_sin_doc},
>     {"sinh",   cmath_sinh,  METH_VARARGS, c_sinh_doc},
>     {"sqrt",   cmath_sqrt,  METH_VARARGS, c_sqrt_doc},
>     {"tan",    cmath_tan,   METH_VARARGS, c_tan_doc},
>     {"tanh",   cmath_tanh,  METH_VARARGS, c_tanh_doc},
>     {NULL,              NULL}           /* sentinel */
> };
>
>
> static struct PyModuleDef cmathmodule = {
>     PyModuleDef_HEAD_INIT,
>     "cmath",
>     module_doc,
>     -1,
>     cmath_methods,
>     NULL,
>     NULL,
>     NULL,
>     NULL
> };
>

Cython will make this, do not care about it. You don't have to set up 
jump tables to make Python get the right function from Cython generated 
C code. If you have 22 Python-callable functions (i.e. declared def or 
cpdef), Cython will make a jump table for those as above.



> Another stuff that I'm getting in trouble in this initial part is how 
> we would translate functions like PyArg_ParseTuple, any clue? I'm 
> studing ways to replace too.
>

Do not care about PyArg_ParseTuple either. It's what C Python needs to 
parse function call arguments from a tuple into C primitives. Cython 
will do this, which is some of the raison d'etre for using Cython.

Also observe that any initialisation done in PyInit_cmathshould go as a 
module level function call in Cython, i.e. PyInit_cmathis called on 
import just like module level Python and Cython code.

I don't have time to implement all of cmathmodule.c, but here is a 
starter (not tested & not complete).

It might actually be that we should use "cdef complex z" instead of 
"cdef double complex z". There might be a distinction in the generated 
C/C++ code between those types, e.g. Py_complex for complex and "double 
_Complex" or "std::complex<double>" for double complex,  even though 
they are binary equivalent. I'm not sure about the state of Cython with 
respect to complex numbers, so just try it and see which works better :-)

Also observe that we do not release the GIL here. That is not because 
these functions are not thread-safe, they are, but yielding the GIL will 
slow things terribly.

Sturla





cimport math
cimport stdlib


cdef extern from "_math.h":
     int Py_IS_FINITE(double)
     int Py_IS_NAN(double)
     double copysign(double,double)


cdef enum special_types:
     ST_NINF  # 0, negative infinity
     ST_NEG   # 1, negative finite number (nonzero)
     ST_NZERO # 2, -0.
     ST_PZERO # 3, +0.
     ST_POS   # 4, positive finite number (nonzero)
     ST_PINF  # 5, positive infinity
     ST_NAN   # 6, Not a Number


cdef inline special_types special_type(double d):
     if (Py_IS_FINITE(d)):
         if (d != 0):
             if (copysign(1., d) == 1.):
                 return ST_POS
             else
                 return ST_NEG
         else:
             if (copysign(1., d) == 1.):
                 return ST_PZERO
             else
                 return ST_NZERO
     if (Py_IS_NAN(d)):
         return ST_NAN
     if (copysign(1., d) == 1.):
         return ST_PINF
     else:
         return ST_NINF


cdef void INIT_SPECIAL_VALUES( double complex *table,
                                double complex arc[][7]):
     stdlib.memcpy(table, &(src[0][0]), 7*7*sizeof(double complex))



cdef inline double complex *SPECIAL_VALUE(double complex z, double 
complex table[][7]):
     if (not Py_IS_FINITE(z.real)) or (not Py_IS_FINITE(z.imag)):
         errno = 0
         return &(table[special_type(z.real)][special_type(z.imag)])
     else:
         return NULL



cdef double complex acosh_special_values[7][7]

INIT_SPECIAL_VALUES( acos_special_values, {
  C(P34,INF) C(P,INF) C(P,INF) C(P,-INF) C(P,-INF) C(P34,-INF) C(N,INF)
  C(P12,INF) C(U,U) C(U,U) C(U,U) C(U,U) C(P12,-INF) C(N,N)
  C(P12,INF) C(U,U) C(P12,0.) C(P12,-0.) C(U,U) C(P12,-INF) C(P12,N)
  C(P12,INF) C(U,U) C(P12,0.) C(P12,-0.) C(U,U) C(P12,-INF) C(P12,N)
  C(P12,INF) C(U,U) C(U,U) C(U,U) C(U,U) C(P12,-INF) C(N,N)
  C(P14,INF) C(0.,INF) C(0.,INF) C(0.,-INF) C(0.,-INF) C(P14,-INF) C(N,INF)
  C(N,INF) C(N,N) C(N,N) C(N,N) C(N,N) C(N,-INF) C(N,N)
  })



cdef double complex cmath_acosh(double complex z):

     cdef double complex s1, s2, r, *psv

     psv = SPECIAL_VALUE(z, acosh_special_values)
     if (psv != NULL):
         return psv[0]

     if (math.fabs(z.real) > CM_LARGE_DOUBLE or math.fabs(z.imag) > 
CM_LARGE_DOUBLE):
         # avoid unnecessary overflow for large arguments
         r.real = math.log(hypot(z.real/2., z.imag/2.)) + M_LN2*2.
         r.imag = math.atan2(z.imag, z.real)
     else:
         s1.real = z.real - 1.
         s1.imag = z.imag
         s1 = cmath_sqrt(s1) # cdef double complex cmath_sqrt(double 
complex z)
         s2.real = z.real + 1.
         s2.imag = z.imag
         s2 = cmath_sqrt(s2)
         r.real = math.asinh(s1.real*s2.real + s1.imag*s2.imag)
         r.imag = 2.*math.atan2(s1.imag, s2.real)
     errno = 0
     return r


def acosh(object arg):
     """acos(x)

Return the arc cosine of x."""
     double complex z
     z = cmath_acosh(<double complex> arg)
     return complex(z)





























From sturla at molden.no  Mon Apr  4 01:53:57 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 04 Apr 2011 01:53:57 +0200
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <4D990791.6080301@molden.no>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>	<4D9588CC.6000303@behnel.de>	<AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>	<4D96C73E.4080600@behnel.de>	<AANLkTin_EXE-vo+vWz38BSGFG2DZvtndksqhQWqaF=Ne@mail.gmail.com>
	<4D990791.6080301@molden.no>
Message-ID: <4D990895.60609@molden.no>

Den 04.04.2011 01:49, skrev Sturla Molden:
> Also observe that we do not release the GIL here. That is not because 
> these functions are not thread-safe, they are, but yielding the GIL 
> will slow things terribly.

Oh, actually they are not thread-safe because we set errno... Sorry.

Sturla



From d.s.seljebotn at astro.uio.no  Mon Apr  4 12:17:56 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 12:17:56 +0200
Subject: [Cython] CEP: prange for parallel loops
Message-ID: <4D999AD4.8080609@astro.uio.no>

CEP up at http://wiki.cython.org/enhancements/prange

"""
This spec is the result of a number of discussions at Cython workshop 1. 
Quite a few different ways of expressing parallelism was looked at, and 
finally we decided to split the problem in two:

  * A simple and friendly solution that covers, perhaps, 80% of the 
cases, based on simply replacing range with prange.

  * Less friendly solutions for the remaining cases. These cases may 
well not even require language support in Cython, or only in indirect 
ways (e.g., cdef closures if normal closures are too expensive).

This document focuses exclusively on the former solution and does not 
intend to cover all use-cases for parallel programming, only the most 
common ones.
"""

Note that me and Mark talked some more on the way to the airport, and 
also I got a couple of more ideas afterwards, so everybody interested 
should probably take a read even if you were there for discussions.

Main post-workshop changes:

  * cython.parallel.firstiteration()/lastiteration # for in-loop if-test 
for thread setup/teardown blocks

  * An idea for how to implement numthreads(), so that we can drop the 
rather complex Context idea.

  * More thoughts on firstprivate/lastprivate


Dag Sverre

From d.s.seljebotn at astro.uio.no  Mon Apr  4 11:43:37 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 11:43:37 +0200
Subject: [Cython] CEP: prange for parallel loops
Message-ID: <4D9992C9.4060605@student.matnat.uio.no>

CEP up at http://wiki.cython.org/enhancements/prange

"""
This spec is the result of a number of discussions at Cython workshop 1. 
Quite a few different ways of expressing parallelism was looked at, and 
finally we decided to split the problem in two:

  * A simple and friendly solution that covers, perhaps, 80% of the 
cases, based on simply replacing range with prange.

  * Less friendly solutions for the remaining cases. These cases may 
well not even require language support in Cython, or only in indirect 
ways (e.g., cdef closures if normal closures are too expensive).

This document focuses exclusively on the former solution and does not 
intend to cover all use-cases for parallel programming, only the most 
common ones.
"""

Note that me and Mark talked some more on the way to the airport, and 
also I got a couple of more ideas afterwards, so everybody interested 
should probably take a read even if you were there for discussions.

Dag Sverre

From d.s.seljebotn at astro.uio.no  Mon Apr  4 11:47:06 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 11:47:06 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9992C9.4060605@student.matnat.uio.no>
References: <4D9992C9.4060605@student.matnat.uio.no>
Message-ID: <4D99939A.5050001@student.matnat.uio.no>

On 04/04/2011 11:43 AM, Dag Sverre Seljebotn wrote:
> CEP up at http://wiki.cython.org/enhancements/prange
>
> """
> This spec is the result of a number of discussions at Cython workshop 
> 1. Quite a few different ways of expressing parallelism was looked at, 
> and finally we decided to split the problem in two:
>
>  * A simple and friendly solution that covers, perhaps, 80% of the 
> cases, based on simply replacing range with prange.
>
>  * Less friendly solutions for the remaining cases. These cases may 
> well not even require language support in Cython, or only in indirect 
> ways (e.g., cdef closures if normal closures are too expensive).
>
> This document focuses exclusively on the former solution and does not 
> intend to cover all use-cases for parallel programming, only the most 
> common ones.
> """
>
> Note that me and Mark talked some more on the way to the airport, and 
> also I got a couple of more ideas afterwards, so everybody interested 
> should probably take a read even if you were there for discussions.

To be more specific, here's the main post-workshop changes:

  * if cython.parallel.firstthreaditer()/lastthreaditer() # Use if-test 
in loop for thread setup/teardown

  * An idea for implementing threadnum() in a way so that we can drop 
the rather complex Context idea.

  * More thoughts on firstprivate/lastprivate

Dag Sverre

From stefan_ml at behnel.de  Mon Apr  4 13:23:50 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 04 Apr 2011 13:23:50 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D999AD4.8080609@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no>
Message-ID: <4D99AA46.7020600@behnel.de>

Dag Sverre Seljebotn, 04.04.2011 12:17:
> CEP up at http://wiki.cython.org/enhancements/prange

"""
Variable handling

Rather than explicit declaration of shared/private variables we rely on 
conventions:

     * Thread-shared: Variables that are only read and not written in the 
loop body are shared across threads. Variables that are only used in the 
else block are considered shared as well.

     * Thread-private: Variables that are assigned to in the loop body are 
thread-private. Obviously, the iteration counter is thread-private as well.

     * Reduction: Variables that only used on the LHS of an inplace 
operator, such as s above, are marked as targets for reduction. If the 
variable is also used in other ways (LHS of assignment or in an expression) 
it does instead turn into a thread-private variable. Note: This means that 
if one, e.g., inserts printf(... s) above, s is turned into a thread-local 
variable. OTOH, there is simply no way to correctly emulate the effect 
printf(... s) would have in a sequential loop, so such code must be 
discouraged anyway.
"""

What about simply (ab-)using Python semantics and creating a new inner 
scope for the prange loop body? That would basically make the loop behave 
like a closure function, but with the looping header at the 'right' place 
rather than after the closure.

Also, in the example, the local variable declaration of "tmp" outside of 
the loop looks somewhat misplaced, although it's precedented by 
comprehensions (which also have their own local scope in Cython).

Stefan

From d.s.seljebotn at astro.uio.no  Mon Apr  4 13:53:16 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 13:53:16 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99AA46.7020600@behnel.de>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
Message-ID: <4D99B12C.10201@astro.uio.no>

On 04/04/2011 01:23 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 04.04.2011 12:17:
>> CEP up at http://wiki.cython.org/enhancements/prange
>
> """
> Variable handling
>
> Rather than explicit declaration of shared/private variables we rely 
> on conventions:
>
>     * Thread-shared: Variables that are only read and not written in 
> the loop body are shared across threads. Variables that are only used 
> in the else block are considered shared as well.
>
>     * Thread-private: Variables that are assigned to in the loop body 
> are thread-private. Obviously, the iteration counter is thread-private 
> as well.
>
>     * Reduction: Variables that only used on the LHS of an inplace 
> operator, such as s above, are marked as targets for reduction. If the 
> variable is also used in other ways (LHS of assignment or in an 
> expression) it does instead turn into a thread-private variable. Note: 
> This means that if one, e.g., inserts printf(... s) above, s is turned 
> into a thread-local variable. OTOH, there is simply no way to 
> correctly emulate the effect printf(... s) would have in a sequential 
> loop, so such code must be discouraged anyway.
> """
>
> What about simply (ab-)using Python semantics and creating a new inner 
> scope for the prange loop body? That would basically make the loop 
> behave like a closure function, but with the looping header at the 
> 'right' place rather than after the closure.

I'm not quite sure what the concrete changes to the CEP this would lead 
to (assuming you mean this as a proposal for alternative semantics, and 
not an implementation detail).

How would we treat reduction variables? They need to be supported, and 
there's nothing in Python semantics to support reduction variables, they 
are a rather special case everywhere. I suppose keeping the reduction 
clause above, or use the "nonlocal" keyword in the loop body...

Also there's the else:-block, although we could make that part of the 
scope. And the "lastprivate" functionality, although that could be 
dropped without much loss.

>
> Also, in the example, the local variable declaration of "tmp" outside 
> of the loop looks somewhat misplaced, although it's precedented by 
> comprehensions (which also have their own local scope in Cython).

Well, depending on the decision of lastprivate, the declaration would 
need to be outside; I really like the idea of moving "cdef", and am 
prepared to drop lastprivate for this.

Being explicit about thread-local variables does make things a lot safer 
to use.

(One problem is that switching between serial and parallel one needs to 
move variable declarations. But that only happens once, and one can use 
"nthreads=1" to disable parallel after that.)

An example would then be:

def f(np.ndarray[double] x, double alpha):
     cdef double s = 0, globtmp
     with nogil:
         for i in prange(x.shape[0]):
             cdef double tmp # thread-private
             tmp = alpha * i # alpha available from global scope
             s += x[i] * tmp # still automatic reduction for inplace 
operators
             # printf(...s) -> now leads to error, since s is not 
declared thread-private but is read
         else:
             # tmp still available here...looks a bit strange, but useful
             s += tmp * 10
             globtmp = tmp # we save tmp for later
         # tmp not available here, globtmp is
     return s

Or, we just drop support for the else block on these loops.

Dag Sverre

From stefan_ml at behnel.de  Mon Apr  4 15:04:11 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 04 Apr 2011 15:04:11 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99B12C.10201@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
Message-ID: <4D99C1CB.2060400@behnel.de>

Dag Sverre Seljebotn, 04.04.2011 13:53:
> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>> CEP up at http://wiki.cython.org/enhancements/prange
>>
>> """
>> Variable handling
>>
>> Rather than explicit declaration of shared/private variables we rely on
>> conventions:
>>
>> * Thread-shared: Variables that are only read and not written in the loop
>> body are shared across threads. Variables that are only used in the else
>> block are considered shared as well.
>>
>> * Thread-private: Variables that are assigned to in the loop body are
>> thread-private. Obviously, the iteration counter is thread-private as well.
>>
>> * Reduction: Variables that only used on the LHS of an inplace operator,
>> such as s above, are marked as targets for reduction. If the variable is
>> also used in other ways (LHS of assignment or in an expression) it does
>> instead turn into a thread-private variable. Note: This means that if
>> one, e.g., inserts printf(... s) above, s is turned into a thread-local
>> variable. OTOH, there is simply no way to correctly emulate the effect
>> printf(... s) would have in a sequential loop, so such code must be
>> discouraged anyway.
>> """
>>
>> What about simply (ab-)using Python semantics and creating a new inner
>> scope for the prange loop body? That would basically make the loop behave
>> like a closure function, but with the looping header at the 'right' place
>> rather than after the closure.
>
> I'm not quite sure what the concrete changes to the CEP this would lead to
> (assuming you mean this as a proposal for alternative semantics, and not an
> implementation detail).

What I would like to avoid is having to tell users "and now for something 
completely different". It looks like a loop, but then there's a whole page 
of new semantics for it. And this also cannot be used in plain Python code 
due to the differing scoping behaviour.


> How would we treat reduction variables? They need to be supported, and
> there's nothing in Python semantics to support reduction variables, they
> are a rather special case everywhere. I suppose keeping the reduction
> clause above, or use the "nonlocal" keyword in the loop body...

That's what I thought, yes. It looks unexpected, sure. That's the clear 
advantage of using inner functions, which do not add anything new at all. 
But if we want to add something that looks more like a loop, we should at 
least make it behave like something that's easy to explain.

Sorry for not taking the opportunity to articulate my scepticism in the 
workshop discussion. Skipping through the CEP now, I think this feature 
adds quite some complexity to the language, and I'm not sure it's worth 
that when compared to the existing closures. The equivalent 
closure+decorator syntax is certainly easier to explain, and could 
translate into exactly the same code. But with the clear advantage that the 
scope of local, nonlocal and thread-configuring variables is immediately 
obvious.

Basically, your example would become

def f(np.ndarray[double] x, double alpha):
     cdef double s = 0

     with cython.nogil:
         @cython.run_parallel_for_loop( range(x.shape[0]) )
         cdef threaded_loop(i):    # 'nogil' is inherited
             cdef double tmp = alpha * i
             nonlocal s
             s += x[i] * tmp
         s += alpha * (x.shape[0] - 1)
     return s

We likely agree that this is not beautiful. It's also harder to implement 
than a "simple" for-in-prange loop. But I find it at least easier to 
explain and semantically 'obvious'. And it would allow us to write a pure 
mode implementation for this based on the threading module.


> Also there's the else:-block, although we could make that part of the
> scope.

Since that's supposed to run single-threaded anyway, it can be written 
after the loop, right? Or is there really a use case where one of the 
threads has to do something in parallel, especially based on its local 
thread state, that the others don't do?


> And the "lastprivate" functionality, although that could be dropped
> without much loss.

I'm not sure how the "else" block and "lastprivate" could be integrated 
into the closures approach.

Stefan

From njs at pobox.com  Mon Apr  4 15:27:50 2011
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 4 Apr 2011 06:27:50 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D999AD4.8080609@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no>
Message-ID: <BANLkTikHz6Yk0kDgDOPtV614KWjZYEUB5Q@mail.gmail.com>

On Mon, Apr 4, 2011 at 3:17 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> ?* A simple and friendly solution that covers, perhaps, 80% of the cases,
> based on simply replacing range with prange.

This is a "merely" aesthetic objection, while remaining agnostic on
the larger discussion, but -- 'for i in prange(...)' looks Just Wrong.
This is not a regular loop over a funny range, it's a funny loop over
a regular range. Surely it should be 'pfor i in range(...)'. Or better
yet, spell it 'parallel_for'.

-- Nathaniel

From d.s.seljebotn at astro.uio.no  Mon Apr  4 15:33:02 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 15:33:02 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99C1CB.2060400@behnel.de>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>
	<4D99C1CB.2060400@behnel.de>
Message-ID: <4D99C88E.6030004@astro.uio.no>

On 04/04/2011 03:04 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 04.04.2011 13:53:
>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>
>>> """
>>> Variable handling
>>>
>>> Rather than explicit declaration of shared/private variables we rely on
>>> conventions:
>>>
>>> * Thread-shared: Variables that are only read and not written in the 
>>> loop
>>> body are shared across threads. Variables that are only used in the 
>>> else
>>> block are considered shared as well.
>>>
>>> * Thread-private: Variables that are assigned to in the loop body are
>>> thread-private. Obviously, the iteration counter is thread-private 
>>> as well.
>>>
>>> * Reduction: Variables that only used on the LHS of an inplace 
>>> operator,
>>> such as s above, are marked as targets for reduction. If the 
>>> variable is
>>> also used in other ways (LHS of assignment or in an expression) it does
>>> instead turn into a thread-private variable. Note: This means that if
>>> one, e.g., inserts printf(... s) above, s is turned into a thread-local
>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>> printf(... s) would have in a sequential loop, so such code must be
>>> discouraged anyway.
>>> """
>>>
>>> What about simply (ab-)using Python semantics and creating a new inner
>>> scope for the prange loop body? That would basically make the loop 
>>> behave
>>> like a closure function, but with the looping header at the 'right' 
>>> place
>>> rather than after the closure.
>>
>> I'm not quite sure what the concrete changes to the CEP this would 
>> lead to
>> (assuming you mean this as a proposal for alternative semantics, and 
>> not an
>> implementation detail).
>
> What I would like to avoid is having to tell users "and now for 
> something completely different". It looks like a loop, but then 
> there's a whole page of new semantics for it. And this also cannot be 
> used in plain Python code due to the differing scoping behaviour.

Well, at least it's better than the 300 pages of semantics for OpenMP :-)

>
>
>> How would we treat reduction variables? They need to be supported, and
>> there's nothing in Python semantics to support reduction variables, they
>> are a rather special case everywhere. I suppose keeping the reduction
>> clause above, or use the "nonlocal" keyword in the loop body...
>
> That's what I thought, yes. It looks unexpected, sure. That's the 
> clear advantage of using inner functions, which do not add anything 
> new at all. But if we want to add something that looks more like a 
> loop, we should at least make it behave like something that's easy to 
> explain.
>
> Sorry for not taking the opportunity to articulate my scepticism in 
> the workshop discussion.


I like the idea of considering cdef/nonlocal in the prange blocks. But, 
yes, I do feel that opposing a parallel loop construct in general is 
rather late, or at least could have been done at a more convenient time...

All I know and care about is that a decorator-and-closure solution will 
be a lot more obscure among non-CS people who have no clue what a 
closure or decorator is, and those are exactly the people who need this 
kind of simple 80%-solution.  You and me don't really need any support 
from Cython at all to write multithreaded apps (leaving aesthetics and 
number of keystrokes to the side).

It'd be good to hear Robert's and Mark's opinions before going further, 
let's economise this thread a bit.

Dag Sverre

From sturla at molden.no  Mon Apr  4 15:33:52 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 04 Apr 2011 15:33:52 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99C1CB.2060400@behnel.de>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>
	<4D99C1CB.2060400@behnel.de>
Message-ID: <4D99C8C0.2020700@molden.no>

Den 04.04.2011 15:04, skrev Stefan Behnel:
>
> What I would like to avoid is having to tell users "and now for 
> something completely different". It looks like a loop, but then 
> there's a whole page of new semantics for it. And this also cannot be 
> used in plain Python code due to the differing scoping behaviour.
>

I've been working on something similar, which does not involve any 
changes to Cython, and will work from Python as well. It's been 
discussed before, basically it involves wrapping a loop in a closure, 
and then normal Python scoping rules applies.

cdef int n
@parallel
def _parallel_loop(parallel_env):
      cdef int i, s0, s1
      for s0,s1 in parallel_env.range(n):
          for i in range(s0,s1):
              pass

I am not happy about the verbosity of the wrapper compared to

for i in prange(n):
     pass

but this is the best I can do without changing the compiler. Notice e.g. 
that the loop becomes two nested loops, which is required for efficient 
work scheduling.

Progress is mainly limited by lack of time and personal need. If I ned 
parallel computing I use Fortran or an optimized LAPACK library (e.g. ACML).

Sturla

From d.s.seljebotn at astro.uio.no  Mon Apr  4 15:51:23 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 15:51:23 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTikHz6Yk0kDgDOPtV614KWjZYEUB5Q@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<BANLkTikHz6Yk0kDgDOPtV614KWjZYEUB5Q@mail.gmail.com>
Message-ID: <4D99CCDB.7000406@astro.uio.no>

On 04/04/2011 03:27 PM, Nathaniel Smith wrote:
> On Mon, Apr 4, 2011 at 3:17 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>>   * A simple and friendly solution that covers, perhaps, 80% of the cases,
>> based on simply replacing range with prange.
> This is a "merely" aesthetic objection, while remaining agnostic on
> the larger discussion, but -- 'for i in prange(...)' looks Just Wrong.
> This is not a regular loop over a funny range, it's a funny loop over
> a regular range. Surely it should be 'pfor i in range(...)'. Or better
> yet, spell it 'parallel_for'.

I don't mind calling it "parallel_for" myself, if only a good place to 
provide scheduling parameters (numthreads, dynamic vs. static 
scheduling, chunksize) can be found. That would make it more obvious 
that scoping rules are different too.

No sense in discussing this further until the higher-level discussion on 
whether to do it or not has completed though.

Dag Sverre

From markflorisson88 at gmail.com  Mon Apr  4 17:22:20 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 4 Apr 2011 17:22:20 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99B12C.10201@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
Message-ID: <BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>

On 4 April 2011 13:53, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>
>>> CEP up at http://wiki.cython.org/enhancements/prange
>>
>> """
>> Variable handling
>>
>> Rather than explicit declaration of shared/private variables we rely on
>> conventions:
>>
>> ? ?* Thread-shared: Variables that are only read and not written in the
>> loop body are shared across threads. Variables that are only used in the
>> else block are considered shared as well.
>>
>> ? ?* Thread-private: Variables that are assigned to in the loop body are
>> thread-private. Obviously, the iteration counter is thread-private as well.
>>
>> ? ?* Reduction: Variables that only used on the LHS of an inplace
>> operator, such as s above, are marked as targets for reduction. If the
>> variable is also used in other ways (LHS of assignment or in an expression)
>> it does instead turn into a thread-private variable. Note: This means that
>> if one, e.g., inserts printf(... s) above, s is turned into a thread-local
>> variable. OTOH, there is simply no way to correctly emulate the effect
>> printf(... s) would have in a sequential loop, so such code must be
>> discouraged anyway.
>> """
>>
>> What about simply (ab-)using Python semantics and creating a new inner
>> scope for the prange loop body? That would basically make the loop behave
>> like a closure function, but with the looping header at the 'right' place
>> rather than after the closure.
>
> I'm not quite sure what the concrete changes to the CEP this would lead to
> (assuming you mean this as a proposal for alternative semantics, and not an
> implementation detail).
>
> How would we treat reduction variables? They need to be supported, and
> there's nothing in Python semantics to support reduction variables, they are
> a rather special case everywhere. I suppose keeping the reduction clause
> above, or use the "nonlocal" keyword in the loop body...
>
> Also there's the else:-block, although we could make that part of the scope.
> And the "lastprivate" functionality, although that could be dropped without
> much loss.
>
>>
>> Also, in the example, the local variable declaration of "tmp" outside of
>> the loop looks somewhat misplaced, although it's precedented by
>> comprehensions (which also have their own local scope in Cython).
>
> Well, depending on the decision of lastprivate, the declaration would need
> to be outside; I really like the idea of moving "cdef", and am prepared to
> drop lastprivate for this.
>
> Being explicit about thread-local variables does make things a lot safer to
> use.
>
> (One problem is that switching between serial and parallel one needs to move
> variable declarations. But that only happens once, and one can use
> "nthreads=1" to disable parallel after that.)
>
> An example would then be:
>
> def f(np.ndarray[double] x, double alpha):
> ? ?cdef double s = 0, globtmp
> ? ?with nogil:
> ? ? ? ?for i in prange(x.shape[0]):
> ? ? ? ? ? ?cdef double tmp # thread-private
> ? ? ? ? ? ?tmp = alpha * i # alpha available from global scope
> ? ? ? ? ? ?s += x[i] * tmp # still automatic reduction for inplace operators
> ? ? ? ? ? ?# printf(...s) -> now leads to error, since s is not declared
> thread-private but is read
> ? ? ? ?else:
> ? ? ? ? ? ?# tmp still available here...looks a bit strange, but useful
> ? ? ? ? ? ?s += tmp * 10
> ? ? ? ? ? ?globtmp = tmp # we save tmp for later
> ? ? ? ?# tmp not available here, globtmp is
> ? ?return s
>
> Or, we just drop support for the else block on these loops.

I think since we are disallowing break (yet) we shouldn't support the
else clause. Basically, I think we can make the CEP a tad more simple.

I think we could declare everything outside of the prange body. Then,
in the prange loop body:

    if a variable is assigned to anywhere -> make it lastprivate
        - if a variable is read before assigned to -> make it
firstprivate in addition to lastprivate (raise compiler error if the
variable is not initialized outside of the loop body)

    if a variable is only ever read -> make it shared (the default for OpenMP)

    if a variable has an inplace operator -> make it a reduction

There is really no reason to disallow reading of the reduction
variable (in e.g. a printf). The reduction should also be initialized
outside of the prange body.

Then prange() could be implemented in pure mode as simply the
sequential version, i.e. range() which some more arguments.

For any scratch space buffers etc, I'd prefer something like


with cython.parallel:
    cdef char *buf = malloc(100)

    for i in prange(n):
        use buf

    free(buf)

At least it fits my brain pretty well :) (this code does however
assume that malloc is thread-safe).

Anyway, I'm not sure I just covered all cases, but what do you think?

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Mon Apr  4 19:01:46 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 19:01:46 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99C1CB.2060400@behnel.de>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>
	<4D99C1CB.2060400@behnel.de>
Message-ID: <4D99F97A.70003@astro.uio.no>

On 04/04/2011 03:04 PM, Stefan Behnel wrote:
>
> That's what I thought, yes. It looks unexpected, sure. That's the 
> clear advantage of using inner functions, which do not add anything 
> new at all. But if we want to add something that looks more like a 
> loop, we should at least make it behave like something that's easy to 
> explain.
>
> Sorry for not taking the opportunity to articulate my scepticism in 
> the workshop discussion. Skipping through the CEP now, I think this 
> feature adds quite some complexity to the language, and I'm not sure 
> it's worth that when compared to the existing closures. The equivalent 
> closure+decorator syntax is certainly easier to explain, and could 
> translate into exactly the same code. But with the clear advantage 
> that the scope of local, nonlocal and thread-configuring variables is 
> immediately obvious.
>
> Basically, your example would become
>
> def f(np.ndarray[double] x, double alpha):
>     cdef double s = 0
>
>     with cython.nogil:
>         @cython.run_parallel_for_loop( range(x.shape[0]) )
>         cdef threaded_loop(i):    # 'nogil' is inherited
>             cdef double tmp = alpha * i
>             nonlocal s
>             s += x[i] * tmp
>         s += alpha * (x.shape[0] - 1)
>     return s
>
> We likely agree that this is not beautiful. It's also harder to 
> implement than a "simple" for-in-prange loop. But I find it at least 
> easier to explain and semantically 'obvious'. And it would allow us to 
> write a pure mode implementation for this based on the threading module.

Short clarification on this example: There is still magic going on here 
in the reduction variable -- one must have a version of "s" for each 
thread, and then reduce at the end.

(Stefan: I realize that you may know this, I'm just making sure 
everything is stated clearly in this discussion.)

Dag Sverre

From d.s.seljebotn at astro.uio.no  Mon Apr  4 19:18:40 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 19:18:40 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
Message-ID: <4D99FD70.6010408@astro.uio.no>

On 04/04/2011 05:22 PM, mark florisson wrote:
> On 4 April 2011 13:53, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>> """
>>> Variable handling
>>>
>>> Rather than explicit declaration of shared/private variables we rely on
>>> conventions:
>>>
>>>     * Thread-shared: Variables that are only read and not written in the
>>> loop body are shared across threads. Variables that are only used in the
>>> else block are considered shared as well.
>>>
>>>     * Thread-private: Variables that are assigned to in the loop body are
>>> thread-private. Obviously, the iteration counter is thread-private as well.
>>>
>>>     * Reduction: Variables that only used on the LHS of an inplace
>>> operator, such as s above, are marked as targets for reduction. If the
>>> variable is also used in other ways (LHS of assignment or in an expression)
>>> it does instead turn into a thread-private variable. Note: This means that
>>> if one, e.g., inserts printf(... s) above, s is turned into a thread-local
>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>> printf(... s) would have in a sequential loop, so such code must be
>>> discouraged anyway.
>>> """
>>>
>>> What about simply (ab-)using Python semantics and creating a new inner
>>> scope for the prange loop body? That would basically make the loop behave
>>> like a closure function, but with the looping header at the 'right' place
>>> rather than after the closure.
>> I'm not quite sure what the concrete changes to the CEP this would lead to
>> (assuming you mean this as a proposal for alternative semantics, and not an
>> implementation detail).
>>
>> How would we treat reduction variables? They need to be supported, and
>> there's nothing in Python semantics to support reduction variables, they are
>> a rather special case everywhere. I suppose keeping the reduction clause
>> above, or use the "nonlocal" keyword in the loop body...
>>
>> Also there's the else:-block, although we could make that part of the scope.
>> And the "lastprivate" functionality, although that could be dropped without
>> much loss.
>>
>>> Also, in the example, the local variable declaration of "tmp" outside of
>>> the loop looks somewhat misplaced, although it's precedented by
>>> comprehensions (which also have their own local scope in Cython).
>> Well, depending on the decision of lastprivate, the declaration would need
>> to be outside; I really like the idea of moving "cdef", and am prepared to
>> drop lastprivate for this.
>>
>> Being explicit about thread-local variables does make things a lot safer to
>> use.
>>
>> (One problem is that switching between serial and parallel one needs to move
>> variable declarations. But that only happens once, and one can use
>> "nthreads=1" to disable parallel after that.)
>>
>> An example would then be:
>>
>> def f(np.ndarray[double] x, double alpha):
>>     cdef double s = 0, globtmp
>>     with nogil:
>>         for i in prange(x.shape[0]):
>>             cdef double tmp # thread-private
>>             tmp = alpha * i # alpha available from global scope
>>             s += x[i] * tmp # still automatic reduction for inplace operators
>>             # printf(...s) ->  now leads to error, since s is not declared
>> thread-private but is read
>>         else:
>>             # tmp still available here...looks a bit strange, but useful
>>             s += tmp * 10
>>             globtmp = tmp # we save tmp for later
>>         # tmp not available here, globtmp is
>>     return s
>>
>> Or, we just drop support for the else block on these loops.
> I think since we are disallowing break (yet) we shouldn't support the
> else clause. Basically, I think we can make the CEP a tad more simple.
>
> I think we could declare everything outside of the prange body. Then,
> in the prange loop body:
>
>      if a variable is assigned to anywhere ->  make it lastprivate
>          - if a variable is read before assigned to ->  make it
> firstprivate in addition to lastprivate (raise compiler error if the
> variable is not initialized outside of the loop body)
>
>      if a variable is only ever read ->  make it shared (the default for OpenMP)
>
>      if a variable has an inplace operator ->  make it a reduction
>
> There is really no reason to disallow reading of the reduction
> variable (in e.g. a printf). The reduction should also be initialized
> outside of the prange body.

The reason for disallowing reading the reduction variable is that 
otherwise you have a contradiction above, since a reduction variable may 
also be a thread-local variable. Or, you disable inplace operators for 
thread-local variables? (ugh)

That's the main reason I'm leaning towards explicit declaring local 
variables using "cdef".

If we're reducing complexity BTW, I'd rather remove 
firstprivate/lastprivate alltogether, see below.

> Then prange() could be implemented in pure mode as simply the
> sequential version, i.e. range() which some more arguments.
>
> For any scratch space buffers etc, I'd prefer something like
>
>
> with cython.parallel:
>      cdef char *buf = malloc(100)
>
>      for i in prange(n):
>          use buf
>
>      free(buf)
>
> At least it fits my brain pretty well :) (this code does however
> assume that malloc is thread-safe).

Yes...perhaps a cython.parellel block will make everybody happy:

  - It's more obvious that we create a new scope, which at least answers 
some of Stefan's complaints

  - We can use normal "for i in range", and put scheduling params on 
parallel(), which makes Nathaniel happy

In this case I'd say we simply do not support firstprivate, all 
thread-local variables must be declared in the block, and for 
firstprivate behaviour you just initialize them yourself which is more 
explicit and Pythonic. The "else:"-block on loops is still useful for 
lastprivate behaviour -- the point of executing the else block in one of 
the threads is that you can then copy thread-local variables of the 
"last" thread into shared variables to get lastprivate behaviour (again, 
more explicit and Python).

If we allow "with cython.nogil, cython.parallel" we can keep the same 
number of indentation levels in some cases.

Also, I think there's still a use for my num_threads_that_would_spawn(), 
so that the malloc can be moved out to a GIL-holding section if one 
wants to -- I may want to allocate with a NumPy array instead of malloc.

Dag Sverre

From markflorisson88 at gmail.com  Mon Apr  4 21:26:34 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 4 Apr 2011 21:26:34 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99FD70.6010408@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
Message-ID: <BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>

On 4 April 2011 19:18, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/04/2011 05:22 PM, mark florisson wrote:
>>
>> On 4 April 2011 13:53, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>>>
>>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>>>
>>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>>
>>>> """
>>>> Variable handling
>>>>
>>>> Rather than explicit declaration of shared/private variables we rely on
>>>> conventions:
>>>>
>>>> ? ?* Thread-shared: Variables that are only read and not written in the
>>>> loop body are shared across threads. Variables that are only used in the
>>>> else block are considered shared as well.
>>>>
>>>> ? ?* Thread-private: Variables that are assigned to in the loop body are
>>>> thread-private. Obviously, the iteration counter is thread-private as
>>>> well.
>>>>
>>>> ? ?* Reduction: Variables that only used on the LHS of an inplace
>>>> operator, such as s above, are marked as targets for reduction. If the
>>>> variable is also used in other ways (LHS of assignment or in an
>>>> expression)
>>>> it does instead turn into a thread-private variable. Note: This means
>>>> that
>>>> if one, e.g., inserts printf(... s) above, s is turned into a
>>>> thread-local
>>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>>> printf(... s) would have in a sequential loop, so such code must be
>>>> discouraged anyway.
>>>> """
>>>>
>>>> What about simply (ab-)using Python semantics and creating a new inner
>>>> scope for the prange loop body? That would basically make the loop
>>>> behave
>>>> like a closure function, but with the looping header at the 'right'
>>>> place
>>>> rather than after the closure.
>>>
>>> I'm not quite sure what the concrete changes to the CEP this would lead
>>> to
>>> (assuming you mean this as a proposal for alternative semantics, and not
>>> an
>>> implementation detail).
>>>
>>> How would we treat reduction variables? They need to be supported, and
>>> there's nothing in Python semantics to support reduction variables, they
>>> are
>>> a rather special case everywhere. I suppose keeping the reduction clause
>>> above, or use the "nonlocal" keyword in the loop body...
>>>
>>> Also there's the else:-block, although we could make that part of the
>>> scope.
>>> And the "lastprivate" functionality, although that could be dropped
>>> without
>>> much loss.
>>>
>>>> Also, in the example, the local variable declaration of "tmp" outside of
>>>> the loop looks somewhat misplaced, although it's precedented by
>>>> comprehensions (which also have their own local scope in Cython).
>>>
>>> Well, depending on the decision of lastprivate, the declaration would
>>> need
>>> to be outside; I really like the idea of moving "cdef", and am prepared
>>> to
>>> drop lastprivate for this.
>>>
>>> Being explicit about thread-local variables does make things a lot safer
>>> to
>>> use.
>>>
>>> (One problem is that switching between serial and parallel one needs to
>>> move
>>> variable declarations. But that only happens once, and one can use
>>> "nthreads=1" to disable parallel after that.)
>>>
>>> An example would then be:
>>>
>>> def f(np.ndarray[double] x, double alpha):
>>> ? ?cdef double s = 0, globtmp
>>> ? ?with nogil:
>>> ? ? ? ?for i in prange(x.shape[0]):
>>> ? ? ? ? ? ?cdef double tmp # thread-private
>>> ? ? ? ? ? ?tmp = alpha * i # alpha available from global scope
>>> ? ? ? ? ? ?s += x[i] * tmp # still automatic reduction for inplace
>>> operators
>>> ? ? ? ? ? ?# printf(...s) -> ?now leads to error, since s is not declared
>>> thread-private but is read
>>> ? ? ? ?else:
>>> ? ? ? ? ? ?# tmp still available here...looks a bit strange, but useful
>>> ? ? ? ? ? ?s += tmp * 10
>>> ? ? ? ? ? ?globtmp = tmp # we save tmp for later
>>> ? ? ? ?# tmp not available here, globtmp is
>>> ? ?return s
>>>
>>> Or, we just drop support for the else block on these loops.
>>
>> I think since we are disallowing break (yet) we shouldn't support the
>> else clause. Basically, I think we can make the CEP a tad more simple.
>>
>> I think we could declare everything outside of the prange body. Then,
>> in the prange loop body:
>>
>> ? ? if a variable is assigned to anywhere -> ?make it lastprivate
>> ? ? ? ? - if a variable is read before assigned to -> ?make it
>> firstprivate in addition to lastprivate (raise compiler error if the
>> variable is not initialized outside of the loop body)
>>
>> ? ? if a variable is only ever read -> ?make it shared (the default for
>> OpenMP)
>>
>> ? ? if a variable has an inplace operator -> ?make it a reduction
>>
>> There is really no reason to disallow reading of the reduction
>> variable (in e.g. a printf). The reduction should also be initialized
>> outside of the prange body.
>
> The reason for disallowing reading the reduction variable is that otherwise
> you have a contradiction above, since a reduction variable may also be a
> thread-local variable. Or, you disable inplace operators for thread-local
> variables? (ugh)

Yes, an inplace operator would make it a reduction variable, just like
assigning something makes it lastprivate, only reading makes it shared
and reading before writing makes it firstprivate in addition to
lastprivate. This is all implicit.

Alternatively, if you want it more explicit, then instead of the
inplace operator you could allow something like

    sum = cython.parallel.reduction('+', sum) + var1 * var2

instead of

    sum += var1 * var2

> That's the main reason I'm leaning towards explicit declaring local
> variables using "cdef".
>
> If we're reducing complexity BTW, I'd rather remove firstprivate/lastprivate
> alltogether, see below.
>> Then prange() could be implemented in pure mode as simply the
>> sequential version, i.e. range() which some more arguments.
>>
>> For any scratch space buffers etc, I'd prefer something like
>>
>>
>> with cython.parallel:
>> ? ? cdef char *buf = malloc(100)
>>
>> ? ? for i in prange(n):
>> ? ? ? ? use buf
>>
>> ? ? free(buf)
>>
>> At least it fits my brain pretty well :) (this code does however
>> assume that malloc is thread-safe).
>
> Yes...perhaps a cython.parellel block will make everybody happy:
>
> ?- It's more obvious that we create a new scope, which at least answers some
> of Stefan's complaints
>
> ?- We can use normal "for i in range", and put scheduling params on
> parallel(), which makes Nathaniel happy

That doesn't sound intuitive, as the scheduling pertains to the
worksharing 'for' construct, and not the entire parallel region. So
scheduling parameters should be provided to e.g.
cython.parallel.range() (or cython.prange, cython.parallel_range,
whatever).

Then if cython.parallel.range() is in a 'with cython.parallel' block,
it would have '#pragma omp for' semantics (considering OpenMP),
whereas it would be a '#pragma omp parallel for' if not closely nested
in such a block.

> In this case I'd say we simply do not support firstprivate, all thread-local
> variables must be declared in the block, and for firstprivate behaviour you
> just initialize them yourself which is more explicit and Pythonic. The
> "else:"-block on loops is still useful for lastprivate behaviour -- the
> point of executing the else block in one of the threads is that you can then
> copy thread-local variables of the "last" thread into shared variables to
> get lastprivate behaviour (again, more explicit and Python).

Why? They are entirely implicit in my proposal, and intuitively so.
Having the parallel range match the sequential range semantics in this
way feel much more Pythonic than having to copy things over in an else
block and having to declare and define simple variables in a special
place.

So basically you keep your options open: a simple and very concise way
to do a parallel range, and a slightly more convoluted way if you need
to initialize some thread-local buffers.
And the good thing is, you can move back to the sequential range by
simply renaming cython.parallel.range to range.


> If we allow "with cython.nogil, cython.parallel" we can keep the same number
> of indentation levels in some cases.

Yeah that would be nice. We could also make cython.parallel implicitly
nogil, but your approach is more flexible if we want to allow this
construct with the gil in the future.

> Also, I think there's still a use for my num_threads_that_would_spawn(), so
> that the malloc can be moved out to a GIL-holding section if one wants to --
> I may want to allocate with a NumPy array instead of malloc.

Yeah we can keep that in for full flexibility.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

For clarity, I'll add an example:

def f(np.ndarray[double] x, double alpha):
    cdef double s = 0
    cdef double tmp = 2
    cdef double other = 6.6

    with nogil:
        for i in prange(x.shape[0]):
            # reading 'tmp' makes it firstprivate in addition to lastprivate
            # 'other' is only ever read, so it's shared
            printf("%lf %lf %lf\n", tmp, s, other)

            # assigning 'tmp' makes it lastprivate
            tmp = alpha * i

            # using += on 's' makes it a reduction variable with operator '+'
            s += x[i] * tmp

    # at this point, all variables s, tmp and other are well defined

    return s

NOTE: any variable that is determined firstprivate, shared or
reduction must be defined, so there is no place for implicit behaviour
biting you in the behind

From vitja.makarov at gmail.com  Mon Apr  4 22:06:41 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 4 Apr 2011 22:06:41 +0200
Subject: [Cython] problem building master with python3
In-Reply-To: <BANLkTinC_h4XHVFdzmUq9VayCs8dJWamKw@mail.gmail.com>
References: <BANLkTi=a5u_wEheS4hfiWWZ2RD8ynZGbUQ@mail.gmail.com>
	<BANLkTinC_h4XHVFdzmUq9VayCs8dJWamKw@mail.gmail.com>
Message-ID: <BANLkTinocoL7N9dBibyTM0ee=wCdfCf2kw@mail.gmail.com>

2011/4/4 Darren Dale <dsdale24 at gmail.com>:
> On Mon, Apr 4, 2011 at 3:32 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>> I'm attempting to install cython from the git repository to benefit
>> from this fix: http://trac.cython.org/cython_trac/ticket/597 . When I
>> run "python3 setup.py install --user", I get an error:
>>
>> cythoning /Users/darren/Projects/cython/Cython/Compiler/Code.py to
>> /Users/darren/Projects/cython/Cython/Compiler/Code.c
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ? ? ?self.cname = cname
>> ? ? ? ?self.text = text
>> ? ? ? ?self.escaped_value = StringEncoding.escape_byte_string(byte_string)
>> ? ? ? ?self.py_strings = None
>>
>> ? ?def get_py_string_const(self, encoding, identifier=None, is_str=False):
>> ? ^
>> ------------------------------------------------------------
>>
>> Cython/Compiler/Code.py:320:4: Signature not compatible with previous
>> declaration
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ?cdef public object text
>> ? ?cdef public object escaped_value
>> ? ?cdef public dict py_strings
>>
>> ? ?@cython.locals(intern=bint, is_str=bint, is_unicode=bint)
>> ? ?cpdef get_py_string_const(self, encoding, identifier=*, is_str=*)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>> ------------------------------------------------------------
>>
>> Cython/Compiler/Code.pxd:64:30: Previous declaration is here
>> building 'Cython.Compiler.Code' extension
>> /usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g
>> -fwrapv -O3 -Wall -Wstrict-prototypes -O2
>> -I/opt/local/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m
>> -c /Users/darren/Projects/cython/Cython/Compiler/Code.c -o
>> build/temp.macosx-10.6-x86_64-3.2/Users/darren/Projects/cython/Cython/Compiler/Code.o
>> /Users/darren/Projects/cython/Cython/Compiler/Code.c:1:2: error:
>> #error Do not use this file, it is the result of a failed Cython
>> compilation.
>> error: command '/usr/bin/gcc-4.2' failed with exit status 1
>>
>
> Actually, I get this same error when I try to build with python-2.7 as well.
>
> Darren

This one fails too :(

Generators branch is okay. But upstream after merge isn't :(

vitja at vitja-laptop:~/work/cython.git$ cat ttt.py
def foo(is_str=False):
    pass
vitja at vitja-laptop:~/work/cython.git$ cat ttt.pxd
cimport cython

@cython.locals(is_str=cython.bint)
cdef foo(is_str=*)





-- 
vitja.

From d.s.seljebotn at astro.uio.no  Mon Apr  4 22:43:11 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 04 Apr 2011 22:43:11 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
Message-ID: <4D9A2D5F.60503@astro.uio.no>

On 04/04/2011 09:26 PM, mark florisson wrote:
> On 4 April 2011 19:18, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/04/2011 05:22 PM, mark florisson wrote:
>>> On 4 April 2011 13:53, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>   wrote:
>>>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>>> """
>>>>> Variable handling
>>>>>
>>>>> Rather than explicit declaration of shared/private variables we rely on
>>>>> conventions:
>>>>>
>>>>>     * Thread-shared: Variables that are only read and not written in the
>>>>> loop body are shared across threads. Variables that are only used in the
>>>>> else block are considered shared as well.
>>>>>
>>>>>     * Thread-private: Variables that are assigned to in the loop body are
>>>>> thread-private. Obviously, the iteration counter is thread-private as
>>>>> well.
>>>>>
>>>>>     * Reduction: Variables that only used on the LHS of an inplace
>>>>> operator, such as s above, are marked as targets for reduction. If the
>>>>> variable is also used in other ways (LHS of assignment or in an
>>>>> expression)
>>>>> it does instead turn into a thread-private variable. Note: This means
>>>>> that
>>>>> if one, e.g., inserts printf(... s) above, s is turned into a
>>>>> thread-local
>>>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>>>> printf(... s) would have in a sequential loop, so such code must be
>>>>> discouraged anyway.
>>>>> """
>>>>>
>>>>> What about simply (ab-)using Python semantics and creating a new inner
>>>>> scope for the prange loop body? That would basically make the loop
>>>>> behave
>>>>> like a closure function, but with the looping header at the 'right'
>>>>> place
>>>>> rather than after the closure.
>>>> I'm not quite sure what the concrete changes to the CEP this would lead
>>>> to
>>>> (assuming you mean this as a proposal for alternative semantics, and not
>>>> an
>>>> implementation detail).
>>>>
>>>> How would we treat reduction variables? They need to be supported, and
>>>> there's nothing in Python semantics to support reduction variables, they
>>>> are
>>>> a rather special case everywhere. I suppose keeping the reduction clause
>>>> above, or use the "nonlocal" keyword in the loop body...
>>>>
>>>> Also there's the else:-block, although we could make that part of the
>>>> scope.
>>>> And the "lastprivate" functionality, although that could be dropped
>>>> without
>>>> much loss.
>>>>
>>>>> Also, in the example, the local variable declaration of "tmp" outside of
>>>>> the loop looks somewhat misplaced, although it's precedented by
>>>>> comprehensions (which also have their own local scope in Cython).
>>>> Well, depending on the decision of lastprivate, the declaration would
>>>> need
>>>> to be outside; I really like the idea of moving "cdef", and am prepared
>>>> to
>>>> drop lastprivate for this.
>>>>
>>>> Being explicit about thread-local variables does make things a lot safer
>>>> to
>>>> use.
>>>>
>>>> (One problem is that switching between serial and parallel one needs to
>>>> move
>>>> variable declarations. But that only happens once, and one can use
>>>> "nthreads=1" to disable parallel after that.)
>>>>
>>>> An example would then be:
>>>>
>>>> def f(np.ndarray[double] x, double alpha):
>>>>     cdef double s = 0, globtmp
>>>>     with nogil:
>>>>         for i in prange(x.shape[0]):
>>>>             cdef double tmp # thread-private
>>>>             tmp = alpha * i # alpha available from global scope
>>>>             s += x[i] * tmp # still automatic reduction for inplace
>>>> operators
>>>>             # printf(...s) ->    now leads to error, since s is not declared
>>>> thread-private but is read
>>>>         else:
>>>>             # tmp still available here...looks a bit strange, but useful
>>>>             s += tmp * 10
>>>>             globtmp = tmp # we save tmp for later
>>>>         # tmp not available here, globtmp is
>>>>     return s
>>>>
>>>> Or, we just drop support for the else block on these loops.
>>> I think since we are disallowing break (yet) we shouldn't support the
>>> else clause. Basically, I think we can make the CEP a tad more simple.
>>>
>>> I think we could declare everything outside of the prange body. Then,
>>> in the prange loop body:
>>>
>>>      if a variable is assigned to anywhere ->    make it lastprivate
>>>          - if a variable is read before assigned to ->    make it
>>> firstprivate in addition to lastprivate (raise compiler error if the
>>> variable is not initialized outside of the loop body)
>>>
>>>      if a variable is only ever read ->    make it shared (the default for
>>> OpenMP)
>>>
>>>      if a variable has an inplace operator ->    make it a reduction
>>>
>>> There is really no reason to disallow reading of the reduction
>>> variable (in e.g. a printf). The reduction should also be initialized
>>> outside of the prange body.
>> The reason for disallowing reading the reduction variable is that otherwise
>> you have a contradiction above, since a reduction variable may also be a
>> thread-local variable. Or, you disable inplace operators for thread-local
>> variables? (ugh)
> Yes, an inplace operator would make it a reduction variable, just like
> assigning something makes it lastprivate, only reading makes it shared
> and reading before writing makes it firstprivate in addition to
> lastprivate. This is all implicit.
>
> Alternatively, if you want it more explicit, then instead of the
> inplace operator you could allow something like
>
>      sum = cython.parallel.reduction('+', sum) + var1 * var2
>
> instead of
>
>      sum += var1 * var2
>
>> That's the main reason I'm leaning towards explicit declaring local
>> variables using "cdef".
>>
>> If we're reducing complexity BTW, I'd rather remove firstprivate/lastprivate
>> alltogether, see below.
>>> Then prange() could be implemented in pure mode as simply the
>>> sequential version, i.e. range() which some more arguments.
>>>
>>> For any scratch space buffers etc, I'd prefer something like
>>>
>>>
>>> with cython.parallel:
>>>      cdef char *buf = malloc(100)
>>>
>>>      for i in prange(n):
>>>          use buf
>>>
>>>      free(buf)
>>>
>>> At least it fits my brain pretty well :) (this code does however
>>> assume that malloc is thread-safe).
>> Yes...perhaps a cython.parellel block will make everybody happy:
>>
>>   - It's more obvious that we create a new scope, which at least answers some
>> of Stefan's complaints
>>
>>   - We can use normal "for i in range", and put scheduling params on
>> parallel(), which makes Nathaniel happy
> That doesn't sound intuitive, as the scheduling pertains to the
> worksharing 'for' construct, and not the entire parallel region. So
> scheduling parameters should be provided to e.g.
> cython.parallel.range() (or cython.prange, cython.parallel_range,
> whatever).
>
> Then if cython.parallel.range() is in a 'with cython.parallel' block,
> it would have '#pragma omp for' semantics (considering OpenMP),
> whereas it would be a '#pragma omp parallel for' if not closely nested
> in such a block.
>
>> In this case I'd say we simply do not support firstprivate, all thread-local
>> variables must be declared in the block, and for firstprivate behaviour you
>> just initialize them yourself which is more explicit and Pythonic. The
>> "else:"-block on loops is still useful for lastprivate behaviour -- the
>> point of executing the else block in one of the threads is that you can then
>> copy thread-local variables of the "last" thread into shared variables to
>> get lastprivate behaviour (again, more explicit and Python).
> Why? They are entirely implicit in my proposal, and intuitively so.
> Having the parallel range match the sequential range semantics in this
> way feel much more Pythonic than having to copy things over in an else
> block and having to declare and define simple variables in a special
> place.

I'm just afraid the risk of creating bugs is too high with inplace 
operators being as magic as you propose. We must not only judge the 
convenience when done right, but also the chance of using it in the 
wrong way.

"import this" says "Explicit is better than implicit". I'll likely draft 
a different CEP tomorrow just to explore more -- in the end I may still 
well favour your approach.

Dag Sverre

From greg.ewing at canterbury.ac.nz  Mon Apr  4 23:18:03 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 05 Apr 2011 09:18:03 +1200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTikHz6Yk0kDgDOPtV614KWjZYEUB5Q@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<BANLkTikHz6Yk0kDgDOPtV614KWjZYEUB5Q@mail.gmail.com>
Message-ID: <4D9A358B.7040302@canterbury.ac.nz>

Nathaniel Smith wrote:
> Surely it should be 'pfor i in range(...)'.

Or 'pfhor', just to let you know it's really something out of
this world.

http://marathongame.wikia.com/wiki/Pfhor_%28Race%29

-- 
Greg

From robertwb at math.washington.edu  Tue Apr  5 07:05:54 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Apr 2011 22:05:54 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D99C1CB.2060400@behnel.de>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no> <4D99C1CB.2060400@behnel.de>
Message-ID: <BANLkTinaTp3JkVa-TgHt21-yQCkbwbjrAg@mail.gmail.com>

On Mon, Apr 4, 2011 at 6:04 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 04.04.2011 13:53:
>>
>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>>
>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>>
>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>
>>> """
>>> Variable handling
>>>
>>> Rather than explicit declaration of shared/private variables we rely on
>>> conventions:
>>>
>>> * Thread-shared: Variables that are only read and not written in the loop
>>> body are shared across threads. Variables that are only used in the else
>>> block are considered shared as well.
>>>
>>> * Thread-private: Variables that are assigned to in the loop body are
>>> thread-private. Obviously, the iteration counter is thread-private as
>>> well.
>>>
>>> * Reduction: Variables that only used on the LHS of an inplace operator,
>>> such as s above, are marked as targets for reduction. If the variable is
>>> also used in other ways (LHS of assignment or in an expression) it does
>>> instead turn into a thread-private variable. Note: This means that if
>>> one, e.g., inserts printf(... s) above, s is turned into a thread-local
>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>> printf(... s) would have in a sequential loop, so such code must be
>>> discouraged anyway.
>>> """
>>>
>>> What about simply (ab-)using Python semantics and creating a new inner
>>> scope for the prange loop body? That would basically make the loop behave
>>> like a closure function, but with the looping header at the 'right' place
>>> rather than after the closure.
>>
>> I'm not quite sure what the concrete changes to the CEP this would lead to
>> (assuming you mean this as a proposal for alternative semantics, and not
>> an
>> implementation detail).
>
> What I would like to avoid is having to tell users "and now for something
> completely different". It looks like a loop, but then there's a whole page
> of new semantics for it. And this also cannot be used in plain Python code
> due to the differing scoping behaviour.

The same could be said of OpenMP--it looks exactly like a loop except
for a couple of pragmas.

The proposed (as I'm reading the CEP now) semantics of what's shared
and first/last private and reduction would give it the semantics of a
normal, sequential loop (and if your final result changes based on how
many threads were involved then you've got incorrect code). Perhaps
reading of the reduction variable could be fine (though obviously
ill-defined, suitable only for debugging).

>> How would we treat reduction variables? They need to be supported, and
>> there's nothing in Python semantics to support reduction variables, they
>> are a rather special case everywhere. I suppose keeping the reduction
>> clause above, or use the "nonlocal" keyword in the loop body...
>
> That's what I thought, yes. It looks unexpected, sure. That's the clear
> advantage of using inner functions, which do not add anything new at all.
> But if we want to add something that looks more like a loop, we should at
> least make it behave like something that's easy to explain.
>
> Sorry for not taking the opportunity to articulate my scepticism in the
> workshop discussion. Skipping through the CEP now, I think this feature adds
> quite some complexity to the language, and I'm not sure it's worth that when
> compared to the existing closures. The equivalent closure+decorator syntax
> is certainly easier to explain, and could translate into exactly the same
> code. But with the clear advantage that the scope of local, nonlocal and
> thread-configuring variables is immediately obvious.
>
> Basically, your example would become
>
> def f(np.ndarray[double] x, double alpha):
> ? ?cdef double s = 0
>
> ? ?with cython.nogil:
> ? ? ? ?@cython.run_parallel_for_loop( range(x.shape[0]) )
> ? ? ? ?cdef threaded_loop(i): ? ?# 'nogil' is inherited
> ? ? ? ? ? ?cdef double tmp = alpha * i
> ? ? ? ? ? ?nonlocal s
> ? ? ? ? ? ?s += x[i] * tmp
> ? ? ? ?s += alpha * (x.shape[0] - 1)
> ? ?return s
>
> We likely agree that this is not beautiful. It's also harder to implement
> than a "simple" for-in-prange loop. But I find it at least easier to explain
> and semantically 'obvious'. And it would allow us to write a pure mode
> implementation for this based on the threading module.

I'm not opposed to having something like this, it's a whole lot of
code and extra refactoring for the basic usecase. I think a nice,
clean syntax is worthwhile and requires at lest some level of language
support. In some ways it's like buffer support--what goes on under the
hood does take some explaining, but most of the time it works as
expected (i.e. as if you hadn't declared the type), just faster. The
inner workings of prange may be a bit magical, but the intent is not,
and the latter is what users care about.

- Robert

From arthurdesribeiro at gmail.com  Tue Apr  5 07:54:33 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Tue, 5 Apr 2011 02:54:33 -0300
Subject: [Cython] Interest in contributing to the project
In-Reply-To: <4D990895.60609@molden.no>
References: <AANLkTini6eu7dJD-6FS17vhr6oQzuNnKHEPhPYS_Rzsd@mail.gmail.com>
	<4D9588CC.6000303@behnel.de>
	<AANLkTim5ougkQVTFQniAx-YTpr-+bcNtavzsZQakn1bq@mail.gmail.com>
	<4D96C73E.4080600@behnel.de>
	<AANLkTin_EXE-vo+vWz38BSGFG2DZvtndksqhQWqaF=Ne@mail.gmail.com>
	<4D990791.6080301@molden.no> <4D990895.60609@molden.no>
Message-ID: <BANLkTi=rNp67yb0uErtjf5BJoZCXZB94rg@mail.gmail.com>

Thanks for clarification Sturla, that's just the way I was thinking about
some things...

I realized that you used some C code that is in _math.h header file, I mean,
I was thinking that in the project I should rewrite code that belongs to
this file too right?

I started coding but I got stucked in functions like Py_Is_Infinite and
Py_Is_NaN... I saw Sturla e-mail but I thought this would be wrote in a
different way, was I wrong?

I also started to write a proposal for this project and hope to publish it
here tomorrow for your evaluation.

Another point that I'm thinking about is how the profile results should be
organized. Is there any template for this?

Best Regards

[]s

Arthur

2011/4/3 Sturla Molden <sturla at molden.no>

> Den 04.04.2011 01:49, skrev Sturla Molden:
>
>  Also observe that we do not release the GIL here. That is not because
>> these functions are not thread-safe, they are, but yielding the GIL will
>> slow things terribly.
>>
>
> Oh, actually they are not thread-safe because we set errno... Sorry.
>
> Sturla
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110405/f36decbf/attachment.html>

From d.s.seljebotn at astro.uio.no  Tue Apr  5 08:16:59 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 08:16:59 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTinaTp3JkVa-TgHt21-yQCkbwbjrAg@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>
	<4D99C1CB.2060400@behnel.de>
	<BANLkTinaTp3JkVa-TgHt21-yQCkbwbjrAg@mail.gmail.com>
Message-ID: <4D9AB3DB.7090706@astro.uio.no>

On 04/05/2011 07:05 AM, Robert Bradshaw wrote:
> On Mon, Apr 4, 2011 at 6:04 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Dag Sverre Seljebotn, 04.04.2011 13:53:
>>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>> """
>>>> Variable handling
>>>>
>>>> Rather than explicit declaration of shared/private variables we rely on
>>>> conventions:
>>>>
>>>> * Thread-shared: Variables that are only read and not written in the loop
>>>> body are shared across threads. Variables that are only used in the else
>>>> block are considered shared as well.
>>>>
>>>> * Thread-private: Variables that are assigned to in the loop body are
>>>> thread-private. Obviously, the iteration counter is thread-private as
>>>> well.
>>>>
>>>> * Reduction: Variables that only used on the LHS of an inplace operator,
>>>> such as s above, are marked as targets for reduction. If the variable is
>>>> also used in other ways (LHS of assignment or in an expression) it does
>>>> instead turn into a thread-private variable. Note: This means that if
>>>> one, e.g., inserts printf(... s) above, s is turned into a thread-local
>>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>>> printf(... s) would have in a sequential loop, so such code must be
>>>> discouraged anyway.
>>>> """
>>>>
>>>> What about simply (ab-)using Python semantics and creating a new inner
>>>> scope for the prange loop body? That would basically make the loop behave
>>>> like a closure function, but with the looping header at the 'right' place
>>>> rather than after the closure.
>>> I'm not quite sure what the concrete changes to the CEP this would lead to
>>> (assuming you mean this as a proposal for alternative semantics, and not
>>> an
>>> implementation detail).
>> What I would like to avoid is having to tell users "and now for something
>> completely different". It looks like a loop, but then there's a whole page
>> of new semantics for it. And this also cannot be used in plain Python code
>> due to the differing scoping behaviour.
> The same could be said of OpenMP--it looks exactly like a loop except
> for a couple of pragmas.
>
> The proposed (as I'm reading the CEP now) semantics of what's shared
> and first/last private and reduction would give it the semantics of a
> normal, sequential loop (and if your final result changes based on how
> many threads were involved then you've got incorrect code). Perhaps
> reading of the reduction variable could be fine (though obviously
> ill-defined, suitable only for debugging).

So would you disable inplace operators for thread-private variables? 
Otherwise a variable could be both a reduction variable and 
thread-private...

There's a reason I disabled reading the reduction variable (which I 
should have written down).

Dag Sverre

From d.s.seljebotn at astro.uio.no  Tue Apr  5 09:21:43 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 09:21:43 +0200
Subject: [Cython] Another CEP: Parallel block
Message-ID: <4D9AC307.4010006@astro.uio.no>

There's a (much shorter) proposal for a more explicit parallelism 
construct at

http://wiki.cython.org/enhancements/parallelblock

This is a little more verbose for the simplest case, but makes the 
medium-cases that needs work buffers much simpler, and is also more 
explicit and difficult to get wrong.

I am not sure myself which one I prefer of this and prange.

Justification for Cython-specific syntax: This is something that is 
really only useful if you can release the GIL *outside* of the loop. So 
I feel this is an area where a custom Cython solution is natural, sort 
of like "cdef extern", and the buffer access.

Since a similar pure-Python solution is rather useless, I also think 
there's less incentive for making something that works well in 
pure-Python mode.

Dag Sverre

From markflorisson88 at gmail.com  Tue Apr  5 10:26:40 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 10:26:40 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <4D9AC307.4010006@astro.uio.no>
References: <4D9AC307.4010006@astro.uio.no>
Message-ID: <BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>

On 5 April 2011 09:21, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> There's a (much shorter) proposal for a more explicit parallelism construct
> at
>
> http://wiki.cython.org/enhancements/parallelblock
>
> This is a little more verbose for the simplest case, but makes the
> medium-cases that needs work buffers much simpler, and is also more explicit
> and difficult to get wrong.

I actually think your else block really complicates matters. In this
example even your index variable is not well-defined right after the
loop, because it's not "declared lastprivate through the else block".
There is really no reason to make variables private instead of
lastprivate (and additionally firstprivate if needed) by default.

I think we should allow at least both options, so if the variables are
declared in the parallel nogil block they can only be used inside that
block (but are still lastprivate, as the first loop may be followed by
other code). But the user will also still be able to declare and
define stuff outside of the block and omit the with parallel block
entirely.

And again, you will want something like cython.parallel.range instead
of just range, as you will want to pass scheduling parameters to the
range(), and not the parallel.

So e.g. you can still write something like this:

cdef Py_ssize_t i
for i in cython.parallel.range(..., schedule='dynamic', nogil=True):
    do something

print i # i is well-defined here

My point is, implicit first- and lastprivate can be implicit because
it works the exact same way as the sequential python version does. The
only remaining pitfall is the in-place operator which declares a
reduction.

> I am not sure myself which one I prefer of this and prange.
>
> Justification for Cython-specific syntax: This is something that is really
> only useful if you can release the GIL *outside* of the loop. So I feel this
> is an area where a custom Cython solution is natural, sort of like "cdef
> extern", and the buffer access.
>
> Since a similar pure-Python solution is rather useless, I also think there's
> less incentive for making something that works well in pure-Python mode.

Which feature is Cython specific here? The 'with a, b as c:' thing?

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Tue Apr  5 10:34:19 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 05 Apr 2011 10:34:19 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>
References: <4D9AC307.4010006@astro.uio.no>
	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>
Message-ID: <4D9AD40B.20407@behnel.de>

mark florisson, 05.04.2011 10:26:
> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>> Justification for Cython-specific syntax: This is something that is really
>> only useful if you can release the GIL *outside* of the loop. So I feel this
>> is an area where a custom Cython solution is natural, sort of like "cdef
>> extern", and the buffer access.
>>
>> Since a similar pure-Python solution is rather useless, I also think there's
>> less incentive for making something that works well in pure-Python mode.
>
> Which feature is Cython specific here? The 'with a, b as c:' thing?

No, the syntax is just Python. It's the scoping that's Cython specific, 
including the local variable declarations inside of the "with" block.

Stefan

From markflorisson88 at gmail.com  Tue Apr  5 10:44:41 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 10:44:41 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <4D9AD40B.20407@behnel.de>
References: <4D9AC307.4010006@astro.uio.no>
	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>
	<4D9AD40B.20407@behnel.de>
Message-ID: <BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>

On 5 April 2011 10:34, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 05.04.2011 10:26:
>>
>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>
>>> Justification for Cython-specific syntax: This is something that is
>>> really
>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>> this
>>> is an area where a custom Cython solution is natural, sort of like "cdef
>>> extern", and the buffer access.
>>>
>>> Since a similar pure-Python solution is rather useless, I also think
>>> there's
>>> less incentive for making something that works well in pure-Python mode.
>>
>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>
> No, the syntax is just Python. It's the scoping that's Cython specific,
> including the local variable declarations inside of the "with" block.

Hmm, but you can use cython.declare() for that, no?

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Apr  5 10:45:32 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 10:45:32 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
References: <4D9AC307.4010006@astro.uio.no>
	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>
	<4D9AD40B.20407@behnel.de>
	<BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
Message-ID: <BANLkTikhbU9pqUmO+S05O5gbP8EK1wB73A@mail.gmail.com>

On 5 April 2011 10:44, mark florisson <markflorisson88 at gmail.com> wrote:
> On 5 April 2011 10:34, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 05.04.2011 10:26:
>>>
>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>
>>>> Justification for Cython-specific syntax: This is something that is
>>>> really
>>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>>> this
>>>> is an area where a custom Cython solution is natural, sort of like "cdef
>>>> extern", and the buffer access.
>>>>
>>>> Since a similar pure-Python solution is rather useless, I also think
>>>> there's
>>>> less incentive for making something that works well in pure-Python mode.
>>>
>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>
>> No, the syntax is just Python. It's the scoping that's Cython specific,
>> including the local variable declarations inside of the "with" block.
>
> Hmm, but you can use cython.declare() for that, no?

(disregarding the malloc() and pointer arithmetic, of course :)

>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

From stefan_ml at behnel.de  Tue Apr  5 11:01:02 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 05 Apr 2011 11:01:02 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
References: <4D9AC307.4010006@astro.uio.no>	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>	<4D9AD40B.20407@behnel.de>
	<BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
Message-ID: <4D9ADA4E.4050005@behnel.de>

mark florisson, 05.04.2011 10:44:
> On 5 April 2011 10:34, Stefan Behnel wrote:
>> mark florisson, 05.04.2011 10:26:
>>>
>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>
>>>> Justification for Cython-specific syntax: This is something that is
>>>> really
>>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>>> this
>>>> is an area where a custom Cython solution is natural, sort of like "cdef
>>>> extern", and the buffer access.
>>>>
>>>> Since a similar pure-Python solution is rather useless, I also think
>>>> there's
>>>> less incentive for making something that works well in pure-Python mode.
>>>
>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>
>> No, the syntax is just Python. It's the scoping that's Cython specific,
>> including the local variable declarations inside of the "with" block.
>
> Hmm, but you can use cython.declare() for that, no?

cython.declare() is a no-op (or just a plain assignment) in Python. But the 
thread-local scoping of these variables cannot be emulated in Python. So 
this would be a feature that cannot be used in pure Python mode, unlike 
closures.

Stefan

From markflorisson88 at gmail.com  Tue Apr  5 11:05:26 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 11:05:26 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <4D9ADA4E.4050005@behnel.de>
References: <4D9AC307.4010006@astro.uio.no>
	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>
	<4D9AD40B.20407@behnel.de>
	<BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
	<4D9ADA4E.4050005@behnel.de>
Message-ID: <BANLkTikZT9=c3UtW3xjDMFs0kff3tn2gjQ@mail.gmail.com>

On 5 April 2011 11:01, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 05.04.2011 10:44:
>>
>> On 5 April 2011 10:34, Stefan Behnel wrote:
>>>
>>> mark florisson, 05.04.2011 10:26:
>>>>
>>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> Justification for Cython-specific syntax: This is something that is
>>>>> really
>>>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>>>> this
>>>>> is an area where a custom Cython solution is natural, sort of like
>>>>> "cdef
>>>>> extern", and the buffer access.
>>>>>
>>>>> Since a similar pure-Python solution is rather useless, I also think
>>>>> there's
>>>>> less incentive for making something that works well in pure-Python
>>>>> mode.
>>>>
>>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>>
>>> No, the syntax is just Python. It's the scoping that's Cython specific,
>>> including the local variable declarations inside of the "with" block.
>>
>> Hmm, but you can use cython.declare() for that, no?
>
> cython.declare() is a no-op (or just a plain assignment) in Python. But the
> thread-local scoping of these variables cannot be emulated in Python. So
> this would be a feature that cannot be used in pure Python mode, unlike
> closures.

Sure, but the Python version would just be serial, it wouldn't use
threads at all. That's the great thing about OpenMP's philosophy is
that it can be either serial or parallel, the only difference is
speed. If you want speed, use Cython.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Tue Apr  5 11:08:43 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 11:08:43 +0200
Subject: [Cython] Another CEP: Parallel block
In-Reply-To: <4D9ADA4E.4050005@behnel.de>
References: <4D9AC307.4010006@astro.uio.no>	<BANLkTinZSVWHGYYtukk1BkpQwZaNdqQX7g@mail.gmail.com>	<4D9AD40B.20407@behnel.de>	<BANLkTimV1-bujaPeP1mj0rRRoRPr+cU0dw@mail.gmail.com>
	<4D9ADA4E.4050005@behnel.de>
Message-ID: <4D9ADC1B.30302@astro.uio.no>

On 04/05/2011 11:01 AM, Stefan Behnel wrote:
> mark florisson, 05.04.2011 10:44:
>> On 5 April 2011 10:34, Stefan Behnel wrote:
>>> mark florisson, 05.04.2011 10:26:
>>>>
>>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> Justification for Cython-specific syntax: This is something that is
>>>>> really
>>>>> only useful if you can release the GIL *outside* of the loop. So I 
>>>>> feel
>>>>> this
>>>>> is an area where a custom Cython solution is natural, sort of like 
>>>>> "cdef
>>>>> extern", and the buffer access.
>>>>>
>>>>> Since a similar pure-Python solution is rather useless, I also think
>>>>> there's
>>>>> less incentive for making something that works well in pure-Python 
>>>>> mode.
>>>>
>>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>>
>>> No, the syntax is just Python. It's the scoping that's Cython specific,
>>> including the local variable declarations inside of the "with" block.
>>
>> Hmm, but you can use cython.declare() for that, no?
>
> cython.declare() is a no-op (or just a plain assignment) in Python. 
> But the thread-local scoping of these variables cannot be emulated in 
> Python. So this would be a feature that cannot be used in pure Python 
> mode, unlike closures.

The intention of prange was certainly to fall back to a normal 
single-threaded range in Python mode.

Because of the GIL there would rarely be any benefit in running the loop 
in parallel -- only if you immediately dispatch to a long-running task 
that itself releases the GIL, but in those cases you should rather stick 
to pure Python in the first place and not bother with prange.

I think the chance of seeing real-life code that both requires prange to 
run optimally in Cython, and that would not be made slower by more than 
one thread in Python, is pretty close to zero.

Dag Sverre

From stefan_ml at behnel.de  Tue Apr  5 12:51:33 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 05 Apr 2011 12:51:33 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
Message-ID: <4D9AF435.5080905@behnel.de>

mark florisson, 04.04.2011 21:26:
> For clarity, I'll add an example:
>
> def f(np.ndarray[double] x, double alpha):
>      cdef double s = 0
>      cdef double tmp = 2
>      cdef double other = 6.6
>
>      with nogil:
>          for i in prange(x.shape[0]):
>              # reading 'tmp' makes it firstprivate in addition to lastprivate
>              # 'other' is only ever read, so it's shared
>              printf("%lf %lf %lf\n", tmp, s, other)

So, adding a printf() to your code can change the semantics of your 
variables? That sounds like a really bad design to me.

Stefan

From markflorisson88 at gmail.com  Tue Apr  5 13:52:20 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 13:52:20 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9AF435.5080905@behnel.de>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<4D9AF435.5080905@behnel.de>
Message-ID: <BANLkTimSruROeXh-kxuB4Buo1T923+nb-g@mail.gmail.com>

On 5 April 2011 12:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 04.04.2011 21:26:
>>
>> For clarity, I'll add an example:
>>
>> def f(np.ndarray[double] x, double alpha):
>> ? ? cdef double s = 0
>> ? ? cdef double tmp = 2
>> ? ? cdef double other = 6.6
>>
>> ? ? with nogil:
>> ? ? ? ? for i in prange(x.shape[0]):
>> ? ? ? ? ? ? # reading 'tmp' makes it firstprivate in addition to
>> lastprivate
>> ? ? ? ? ? ? # 'other' is only ever read, so it's shared
>> ? ? ? ? ? ? printf("%lf %lf %lf\n", tmp, s, other)
>
> So, adding a printf() to your code can change the semantics of your
> variables? That sounds like a really bad design to me.

I agree, I think we should refrain from the firstprivate() entirely,
as it wouldn't have the same semantics as serial execution (as 'tmp'
would have the original value with parallel execution and the value
from previous iterations with serial execution). So basically we
should allow reading of private variables only after they are assigned
to in the loop body.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From pav at iki.fi  Tue Apr  5 14:55:36 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 5 Apr 2011 12:55:36 +0000 (UTC)
Subject: [Cython] CEP: prange for parallel loops
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
Message-ID: <inf3g8$283$1@dough.gmane.org>

Mon, 04 Apr 2011 21:26:34 +0200, mark florisson wrote:
[clip]
> For clarity, I'll add an example:
[clip]

How about making all the special declarations explicit? The automatic 
inference of variables has a problem in that a small change in a part of 
the code can have somewhat unintuitive non-local effects, as the private/
shared/reduction status of the variable changes in the whole function 
scope (if Python scoping is retained).

Like so with explicit declarations:

def f(np.ndarray[double] x, double alpha):
    cdef double alpha = 6.6
    cdef char *ptr = something()

    # Parallel variables are declared beforehand;
    # the exact syntax could also be something else
    cdef cython.parallel.private[int] tmp = 2, tmp2
    cdef cython.parallel.reduction[int] s = 0

    # Act like ordinary cdef outside prange(); in the prange they are
    # firstprivate if initialized or written to outside the loop anywhere
    # in the scope. Or, they could be firstprivate always, if this
    # has a negligible performance impact.
    tmp = 3

    with nogil:
        s = 9

        for i in prange(x.shape[0]):
            if cython.parallel.first_iteration(i):
                # whatever initialization; Cython is in principle allowed
                # to move this outside the loop, at least if it is
                # the first thing here
                pass

            # tmp2 is not firstprivate, as it's not written to outside
            # the loop body; also, it's also not lastprivate as it's not
            # read outside the loop
            tmp2 = 99

            # Increment a private variable
            tmp += 2*tmp

            # Add stuff to reduction
            s += alpha*i

            # The following raise a compilation error -- the reduction
            # variable cannot be assigned to, and can be only operated on
            # with only a single reduction operation inside prange
            s *= 9
            s = 8

            # It can be read, however, provided openmp supports this
            tmp = s

            # Assignment to non-private variables causes a compile-time
            # error; this avoids common mistakes, such as forgetting to
            # declare the reduction variable.
            alpha += 42
            alpha123 = 9
            ptr = 94

            # These, however, need to be allowed:
            # the users are on their own to make sure they don't clobber
            # non-local variables
            x[i] = 123
            (ptr + i)[0] = 123
            some_routine(x, ptr, i)
        else:
            # private variables are lastprivate if read outside the loop
            foo = tmp

        # The else: block can be added, but actually has no effect
        # as it is always executed --- the code here could as well
        # be written after the for loop
        foo = tmp  # <- same result

    with nogil:
        # Suppose Cython allowed cdef inside blocks with usual scoping
        # rules
        cdef cython.parallel.reduction[double] r = 0

        # the same variables can be used again in a second parallel loop
        for i in prange(x.shape[0]):
            r += 1.5
            s -= i
            tmp = 9

        # also the iteration variable is available after the loop
        count = i

    # As per usual Cython scoping rules
    return r, s

What did I miss here? As far as I see, the above would have the same 
semantics and scoping as a single-threaded Python implementation.

The only change required to make things parallel is replacing range() by 
prange() and adding the variable declarations.

-- 
Pauli Virtanen


From pav at iki.fi  Tue Apr  5 15:10:55 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 5 Apr 2011 13:10:55 +0000 (UTC)
Subject: [Cython] CEP: prange for parallel loops
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<inf3g8$283$1@dough.gmane.org>
Message-ID: <inf4cv$283$2@dough.gmane.org>

Tue, 05 Apr 2011 12:55:36 +0000, Pauli Virtanen wrote:
[clip]
>             # Assignment to non-private variables causes a compile-time
>             # error; this avoids common mistakes, such as forgetting to
>             # declare the reduction variable.
>             alpha += 42
>             alpha123 = 9
>             ptr = 94

Actually, I'm not sure this is absolutely necessary -- life is tough, 
especially if you are programming in parallel, and there are limits to 
hand-holding.

However, an explicit declaration could be added for turning the error off 
for the (rare) cases where this makes sense (e.g. setting a shared flag)

	cdef cython.parallel.shared[double] some_flag

-- 
Pauli Virtanen


From markflorisson88 at gmail.com  Tue Apr  5 15:56:55 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 15:56:55 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <inf3g8$283$1@dough.gmane.org>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<inf3g8$283$1@dough.gmane.org>
Message-ID: <BANLkTimXtyGr8Zd_=QaEhWKsuZmXJdtcOQ@mail.gmail.com>

On 5 April 2011 14:55, Pauli Virtanen <pav at iki.fi> wrote:
>
> Mon, 04 Apr 2011 21:26:34 +0200, mark florisson wrote:
> [clip]
> > For clarity, I'll add an example:
> [clip]
>
> How about making all the special declarations explicit? The automatic
> inference of variables has a problem in that a small change in a part of
> the code can have somewhat unintuitive non-local effects, as the private/
> shared/reduction status of the variable changes in the whole function
> scope (if Python scoping is retained).
>
> Like so with explicit declarations:
>
> def f(np.ndarray[double] x, double alpha):
> ? ?cdef double alpha = 6.6
> ? ?cdef char *ptr = something()
>
> ? ?# Parallel variables are declared beforehand;
> ? ?# the exact syntax could also be something else
> ? ?cdef cython.parallel.private[int] tmp = 2, tmp2
> ? ?cdef cython.parallel.reduction[int] s = 0
>
> ? ?# Act like ordinary cdef outside prange(); in the prange they are
> ? ?# firstprivate if initialized or written to outside the loop anywhere
> ? ?# in the scope. Or, they could be firstprivate always, if this
> ? ?# has a negligible performance impact.
> ? ?tmp = 3

The problem with firstprivate() is that it doesn't give you the same
semantics as in the sequential version. That's why I think it would be
best to forget about firstprivate entirely and allow reading of
private variables only after they are assigned to in the loop body.

>
> ? ?with nogil:
> ? ? ? ?s = 9
>
> ? ? ? ?for i in prange(x.shape[0]):
> ? ? ? ? ? ?if cython.parallel.first_iteration(i):
> ? ? ? ? ? ? ? ?# whatever initialization; Cython is in principle allowed
> ? ? ? ? ? ? ? ?# to move this outside the loop, at least if it is
> ? ? ? ? ? ? ? ?# the first thing here
> ? ? ? ? ? ? ? ?pass

For this I prefer the aforementioned 'with cython.parallel:' block.

>
> ? ? ? ? ? ?# tmp2 is not firstprivate, as it's not written to outside
> ? ? ? ? ? ?# the loop body; also, it's also not lastprivate as it's not
> ? ? ? ? ? ?# read outside the loop
> ? ? ? ? ? ?tmp2 = 99
>
> ? ? ? ? ? ?# Increment a private variable
> ? ? ? ? ? ?tmp += 2*tmp
>
> ? ? ? ? ? ?# Add stuff to reduction
> ? ? ? ? ? ?s += alpha*i
>
> ? ? ? ? ? ?# The following raise a compilation error -- the reduction
> ? ? ? ? ? ?# variable cannot be assigned to, and can be only operated on
> ? ? ? ? ? ?# with only a single reduction operation inside prange
> ? ? ? ? ? ?s *= 9
> ? ? ? ? ? ?s = 8

I think OpenMP allows arbitrary assignments and expressions to the
reduction variable, all the spec says "usually it will be of the form
'x <binop>= ...'".

>
> ? ? ? ? ? ?# It can be read, however, provided openmp supports this
> ? ? ? ? ? ?tmp = s
>
> ? ? ? ? ? ?# Assignment to non-private variables causes a compile-time
> ? ? ? ? ? ?# error; this avoids common mistakes, such as forgetting to
> ? ? ? ? ? ?# declare the reduction variable.
> ? ? ? ? ? ?alpha += 42
> ? ? ? ? ? ?alpha123 = 9
> ? ? ? ? ? ?ptr = 94
>
> ? ? ? ? ? ?# These, however, need to be allowed:
> ? ? ? ? ? ?# the users are on their own to make sure they don't clobber
> ? ? ? ? ? ?# non-local variables
> ? ? ? ? ? ?x[i] = 123
> ? ? ? ? ? ?(ptr + i)[0] = 123
> ? ? ? ? ? ?some_routine(x, ptr, i)

Indeed. They could be either shared or firstprivate (as the pointer
would be firstprivate, and not the entire array, unless it was
declared as a C array of certain size).

> ? ? ? ?else:
> ? ? ? ? ? ?# private variables are lastprivate if read outside the loop
> ? ? ? ? ? ?foo = tmp
>
> ? ? ? ?# The else: block can be added, but actually has no effect
> ? ? ? ?# as it is always executed --- the code here could as well
> ? ? ? ?# be written after the for loop
> ? ? ? ?foo = tmp ?# <- same result
>
> ? ?with nogil:
> ? ? ? ?# Suppose Cython allowed cdef inside blocks with usual scoping
> ? ? ? ?# rules
> ? ? ? ?cdef cython.parallel.reduction[double] r = 0
>
> ? ? ? ?# the same variables can be used again in a second parallel loop
> ? ? ? ?for i in prange(x.shape[0]):
> ? ? ? ? ? ?r += 1.5
> ? ? ? ? ? ?s -= i
> ? ? ? ? ? ?tmp = 9
>
> ? ? ? ?# also the iteration variable is available after the loop
> ? ? ? ?count = i
>
> ? ?# As per usual Cython scoping rules
> ? ?return r, s
>
> What did I miss here? As far as I see, the above would have the same
> semantics and scoping as a single-threaded Python implementation.
>
> The only change required to make things parallel is replacing range() by
> prange() and adding the variable declarations.

Basically, I like your approach. It's only slightly more verbose as
the implicit way, as you need to declare the type of each variable
anyway.

I also still like the implicit way, but it has a couple of problems:
     - inplace operators suddenly declare a reduction
     - assigning to a variable has implicit (last)private semantics,
whereas assigning to an element in a buffer has shared semantics

Your explicit version solves both these problems. So I'm +1.

> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Apr  5 15:57:50 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 15:57:50 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <inf4cv$283$2@dough.gmane.org>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<inf3g8$283$1@dough.gmane.org> <inf4cv$283$2@dough.gmane.org>
Message-ID: <BANLkTinGvWAYkvA24V-ruhmDaGLozPOV+A@mail.gmail.com>

On 5 April 2011 15:10, Pauli Virtanen <pav at iki.fi> wrote:
> Tue, 05 Apr 2011 12:55:36 +0000, Pauli Virtanen wrote:
> [clip]
>> ? ? ? ? ? ? # Assignment to non-private variables causes a compile-time
>> ? ? ? ? ? ? # error; this avoids common mistakes, such as forgetting to
>> ? ? ? ? ? ? # declare the reduction variable.
>> ? ? ? ? ? ? alpha += 42
>> ? ? ? ? ? ? alpha123 = 9
>> ? ? ? ? ? ? ptr = 94
>
> Actually, I'm not sure this is absolutely necessary -- life is tough,
> especially if you are programming in parallel, and there are limits to
> hand-holding.
>
> However, an explicit declaration could be added for turning the error off
> for the (rare) cases where this makes sense (e.g. setting a shared flag)
>
> ? ? ? ?cdef cython.parallel.shared[double] some_flag

I think that unless we add support for critical, single or master
sections, or the atomic construct, we should also disallow assigning
to shared variables entirely.

> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Tue Apr  5 16:53:33 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 5 Apr 2011 07:53:33 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9AF435.5080905@behnel.de>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<4D9AF435.5080905@behnel.de>
Message-ID: <BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>

On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 04.04.2011 21:26:
>>
>> For clarity, I'll add an example:
>>
>> def f(np.ndarray[double] x, double alpha):
>> ? ? cdef double s = 0
>> ? ? cdef double tmp = 2
>> ? ? cdef double other = 6.6
>>
>> ? ? with nogil:
>> ? ? ? ? for i in prange(x.shape[0]):
>> ? ? ? ? ? ? # reading 'tmp' makes it firstprivate in addition to
>> lastprivate
>> ? ? ? ? ? ? # 'other' is only ever read, so it's shared
>> ? ? ? ? ? ? printf("%lf %lf %lf\n", tmp, s, other)
>
> So, adding a printf() to your code can change the semantics of your
> variables? That sounds like a really bad design to me.

That's what I was thinking. Basically, if you do an inlace operation,
then it's a reduction variable, no matter what else you do to it
(including possibly a direct assignment, though we could make that a
compile-time error).

- Robert

From d.s.seljebotn at astro.uio.no  Tue Apr  5 16:58:01 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 16:58:01 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>	<4D99FD70.6010408@astro.uio.no>	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>	<4D9AF435.5080905@behnel.de>
	<BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>
Message-ID: <4D9B2DF9.8040305@astro.uio.no>

On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> mark florisson, 04.04.2011 21:26:
>>> For clarity, I'll add an example:
>>>
>>> def f(np.ndarray[double] x, double alpha):
>>>      cdef double s = 0
>>>      cdef double tmp = 2
>>>      cdef double other = 6.6
>>>
>>>      with nogil:
>>>          for i in prange(x.shape[0]):
>>>              # reading 'tmp' makes it firstprivate in addition to
>>> lastprivate
>>>              # 'other' is only ever read, so it's shared
>>>              printf("%lf %lf %lf\n", tmp, s, other)
>> So, adding a printf() to your code can change the semantics of your
>> variables? That sounds like a really bad design to me.
> That's what I was thinking. Basically, if you do an inlace operation,
> then it's a reduction variable, no matter what else you do to it
> (including possibly a direct assignment, though we could make that a
> compile-time error).

-1, I think that's too obscure. Not being able to use inplace operators 
for certain variables will be at the very least be nagging.

I think we need to explicitly declare something. Either a simple 
prange(..., reduce="s:+"), or all-out declaration of thread-local variables.

Reduction isn't *that* common, so perhaps that is what should be 
explicit, unlike my other proposal...

Dag

From robertwb at math.washington.edu  Tue Apr  5 17:01:16 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 5 Apr 2011 08:01:16 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTimSruROeXh-kxuB4Buo1T923+nb-g@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<4D9AF435.5080905@behnel.de>
	<BANLkTimSruROeXh-kxuB4Buo1T923+nb-g@mail.gmail.com>
Message-ID: <BANLkTimhAFCbW-84TzOX8w85P8Eb2fYHgA@mail.gmail.com>

On Tue, Apr 5, 2011 at 4:52 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 5 April 2011 12:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 04.04.2011 21:26:
>>>
>>> For clarity, I'll add an example:
>>>
>>> def f(np.ndarray[double] x, double alpha):
>>> ? ? cdef double s = 0
>>> ? ? cdef double tmp = 2
>>> ? ? cdef double other = 6.6
>>>
>>> ? ? with nogil:
>>> ? ? ? ? for i in prange(x.shape[0]):
>>> ? ? ? ? ? ? # reading 'tmp' makes it firstprivate in addition to
>>> lastprivate
>>> ? ? ? ? ? ? # 'other' is only ever read, so it's shared
>>> ? ? ? ? ? ? printf("%lf %lf %lf\n", tmp, s, other)
>>
>> So, adding a printf() to your code can change the semantics of your
>> variables? That sounds like a really bad design to me.
>
> I agree, I think we should refrain from the firstprivate() entirely,
> as it wouldn't have the same semantics as serial execution (as 'tmp'
> would have the original value with parallel execution and the value
> from previous iterations with serial execution). So basically we
> should allow reading of private variables only after they are assigned
> to in the loop body.

Unless I'm miss-understanding the meaning of firstprivate (it's
initialized per-thread, not per-iteration), for single-threaded
execution, it would have exactly the same semantics as serial
execution. As I mentioned before, if your code functions differently
for single or multiple threads, then it's incorrect. I think it's
natural that a parallel loop would behave like

tmp = global_value
if fork():
   # do first half of the loop, with tmp starting as global_value
else:
   # do last half of the loop, with tmp starting as global_value
# reduction magic

- Robert

From d.s.seljebotn at astro.uio.no  Tue Apr  5 17:02:26 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 17:02:26 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9B2DF9.8040305@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no>	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>	<4D99FD70.6010408@astro.uio.no>	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>	<4D9AF435.5080905@behnel.de>	<BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>
	<4D9B2DF9.8040305@astro.uio.no>
Message-ID: <4D9B2F02.5050301@astro.uio.no>

On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel<stefan_ml at behnel.de>  
>> wrote:
>>> mark florisson, 04.04.2011 21:26:
>>>> For clarity, I'll add an example:
>>>>
>>>> def f(np.ndarray[double] x, double alpha):
>>>>      cdef double s = 0
>>>>      cdef double tmp = 2
>>>>      cdef double other = 6.6
>>>>
>>>>      with nogil:
>>>>          for i in prange(x.shape[0]):
>>>>              # reading 'tmp' makes it firstprivate in addition to
>>>> lastprivate
>>>>              # 'other' is only ever read, so it's shared
>>>>              printf("%lf %lf %lf\n", tmp, s, other)
>>> So, adding a printf() to your code can change the semantics of your
>>> variables? That sounds like a really bad design to me.
>> That's what I was thinking. Basically, if you do an inlace operation,
>> then it's a reduction variable, no matter what else you do to it
>> (including possibly a direct assignment, though we could make that a
>> compile-time error).
>
> -1, I think that's too obscure. Not being able to use inplace 
> operators for certain variables will be at the very least be nagging.
>
> I think we need to explicitly declare something. Either a simple 
> prange(..., reduce="s:+"), or all-out declaration of thread-local 
> variables.

Sorry: prange(..., reduce="s"), or perhaps &s or cython.address(s). The 
+ is of course still specified in code.

Dag Sverre

From robertwb at math.washington.edu  Tue Apr  5 17:14:38 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 5 Apr 2011 08:14:38 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <inf3g8$283$1@dough.gmane.org>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<inf3g8$283$1@dough.gmane.org>
Message-ID: <BANLkTimeCzs_K9vdjQHRVVw1rcqrK1O3sQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:55 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Mon, 04 Apr 2011 21:26:34 +0200, mark florisson wrote:
> [clip]
>> For clarity, I'll add an example:
> [clip]
>
> How about making all the special declarations explicit? The automatic
> inference of variables has a problem in that a small change in a part of
> the code can have somewhat unintuitive non-local effects, as the private/
> shared/reduction status of the variable changes in the whole function
> scope (if Python scoping is retained).
>
> Like so with explicit declarations:

That's an interesting idea. It's a bit odd specifying the scope as
part of the type, but may work. However, I'm still not convinced that
we can't safely infer this information.

> def f(np.ndarray[double] x, double alpha):
> ? ?cdef double alpha = 6.6
> ? ?cdef char *ptr = something()
>
> ? ?# Parallel variables are declared beforehand;
> ? ?# the exact syntax could also be something else
> ? ?cdef cython.parallel.private[int] tmp = 2, tmp2
> ? ?cdef cython.parallel.reduction[int] s = 0
>
> ? ?# Act like ordinary cdef outside prange(); in the prange they are
> ? ?# firstprivate if initialized or written to outside the loop anywhere
> ? ?# in the scope. Or, they could be firstprivate always, if this
> ? ?# has a negligible performance impact.
> ? ?tmp = 3
>
> ? ?with nogil:
> ? ? ? ?s = 9
>
> ? ? ? ?for i in prange(x.shape[0]):
> ? ? ? ? ? ?if cython.parallel.first_iteration(i):
> ? ? ? ? ? ? ? ?# whatever initialization; Cython is in principle allowed
> ? ? ? ? ? ? ? ?# to move this outside the loop, at least if it is
> ? ? ? ? ? ? ? ?# the first thing here
> ? ? ? ? ? ? ? ?pass
>
> ? ? ? ? ? ?# tmp2 is not firstprivate, as it's not written to outside
> ? ? ? ? ? ?# the loop body; also, it's also not lastprivate as it's not
> ? ? ? ? ? ?# read outside the loop
> ? ? ? ? ? ?tmp2 = 99
>
> ? ? ? ? ? ?# Increment a private variable
> ? ? ? ? ? ?tmp += 2*tmp
>
> ? ? ? ? ? ?# Add stuff to reduction
> ? ? ? ? ? ?s += alpha*i
>
> ? ? ? ? ? ?# The following raise a compilation error -- the reduction
> ? ? ? ? ? ?# variable cannot be assigned to, and can be only operated on
> ? ? ? ? ? ?# with only a single reduction operation inside prange
> ? ? ? ? ? ?s *= 9
> ? ? ? ? ? ?s = 8
>
> ? ? ? ? ? ?# It can be read, however, provided openmp supports this
> ? ? ? ? ? ?tmp = s
>
> ? ? ? ? ? ?# Assignment to non-private variables causes a compile-time
> ? ? ? ? ? ?# error; this avoids common mistakes, such as forgetting to
> ? ? ? ? ? ?# declare the reduction variable.
> ? ? ? ? ? ?alpha += 42
> ? ? ? ? ? ?alpha123 = 9
> ? ? ? ? ? ?ptr = 94
>
> ? ? ? ? ? ?# These, however, need to be allowed:
> ? ? ? ? ? ?# the users are on their own to make sure they don't clobber
> ? ? ? ? ? ?# non-local variables
> ? ? ? ? ? ?x[i] = 123
> ? ? ? ? ? ?(ptr + i)[0] = 123
> ? ? ? ? ? ?some_routine(x, ptr, i)
> ? ? ? ?else:
> ? ? ? ? ? ?# private variables are lastprivate if read outside the loop
> ? ? ? ? ? ?foo = tmp
>
> ? ? ? ?# The else: block can be added, but actually has no effect
> ? ? ? ?# as it is always executed --- the code here could as well
> ? ? ? ?# be written after the for loop
> ? ? ? ?foo = tmp ?# <- same result
>
> ? ?with nogil:
> ? ? ? ?# Suppose Cython allowed cdef inside blocks with usual scoping
> ? ? ? ?# rules
> ? ? ? ?cdef cython.parallel.reduction[double] r = 0
>
> ? ? ? ?# the same variables can be used again in a second parallel loop
> ? ? ? ?for i in prange(x.shape[0]):
> ? ? ? ? ? ?r += 1.5
> ? ? ? ? ? ?s -= i
> ? ? ? ? ? ?tmp = 9
>
> ? ? ? ?# also the iteration variable is available after the loop
> ? ? ? ?count = i
>
> ? ?# As per usual Cython scoping rules
> ? ?return r, s
>
> What did I miss here? As far as I see, the above would have the same
> semantics and scoping as a single-threaded Python implementation.

One thing is that it's forcing the scope of the variable to be
consistant throughout the entire function body, so, for example, a
reduction variable in one loop could not be used as a shared in
another (without having to declare a new variable), which is a
different form of non-locality.

> The only change required to make things parallel is replacing range() by
> prange() and adding the variable declarations.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Tue Apr  5 17:26:24 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 5 Apr 2011 08:26:24 -0700
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9B2F02.5050301@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<4D9AF435.5080905@behnel.de>
	<BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>
	<4D9B2DF9.8040305@astro.uio.no> <4D9B2F02.5050301@astro.uio.no>
Message-ID: <BANLkTikWjU04CeDjOkd-HvDUsRcm+ZrAVA@mail.gmail.com>

On Tue, Apr 5, 2011 at 8:02 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
>>
>> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>>>
>>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel<stefan_ml at behnel.de>
>>> ?wrote:
>>>>
>>>> mark florisson, 04.04.2011 21:26:
>>>>>
>>>>> For clarity, I'll add an example:
>>>>>
>>>>> def f(np.ndarray[double] x, double alpha):
>>>>> ? ? cdef double s = 0
>>>>> ? ? cdef double tmp = 2
>>>>> ? ? cdef double other = 6.6
>>>>>
>>>>> ? ? with nogil:
>>>>> ? ? ? ? for i in prange(x.shape[0]):
>>>>> ? ? ? ? ? ? # reading 'tmp' makes it firstprivate in addition to
>>>>> lastprivate
>>>>> ? ? ? ? ? ? # 'other' is only ever read, so it's shared
>>>>> ? ? ? ? ? ? printf("%lf %lf %lf\n", tmp, s, other)
>>>>
>>>> So, adding a printf() to your code can change the semantics of your
>>>> variables? That sounds like a really bad design to me.
>>>
>>> That's what I was thinking. Basically, if you do an inlace operation,
>>> then it's a reduction variable, no matter what else you do to it
>>> (including possibly a direct assignment, though we could make that a
>>> compile-time error).
>>
>> -1, I think that's too obscure. Not being able to use inplace operators
>> for certain variables will be at the very least be nagging.

You could still use inplace operators to your hearts content--just
don't bother using the reduced variable outside the loop. (I guess I'm
assuming reducing a variable has negligible performance overhead,
which it should.) For the rare cases that you want the non-aggregated
private, make an assignment to another variable, or use non-inplace
operations.

Not being able to mix inplace operators might be an annoyance. We
could also allow explicit declarations, as per Pauli's suggestion, but
not require them. Essentially, as long as we have

1) Sequential behavior == one thread scheduled (by semantics)
2) one thread scheduled == multiple threads scheduled (user's
responsibility, as it must be)

then I think we should be fine.

>> I think we need to explicitly declare something. Either a simple
>> prange(..., reduce="s:+"), or all-out declaration of thread-local variables.
>
> Sorry: prange(..., reduce="s"), or perhaps &s or cython.address(s). The + is
> of course still specified in code.
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Tue Apr  5 18:32:25 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 18:32:25 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <BANLkTikWjU04CeDjOkd-HvDUsRcm+ZrAVA@mail.gmail.com>
References: <4D999AD4.8080609@astro.uio.no>
	<4D99AA46.7020600@behnel.de>	<4D99B12C.10201@astro.uio.no>	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>	<4D99FD70.6010408@astro.uio.no>	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>	<4D9AF435.5080905@behnel.de>	<BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>	<4D9B2DF9.8040305@astro.uio.no>
	<4D9B2F02.5050301@astro.uio.no>
	<BANLkTikWjU04CeDjOkd-HvDUsRcm+ZrAVA@mail.gmail.com>
Message-ID: <4D9B4419.5030804@astro.uio.no>

On 04/05/2011 05:26 PM, Robert Bradshaw wrote:
> On Tue, Apr 5, 2011 at 8:02 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
>>> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>>>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel<stefan_ml at behnel.de>
>>>>   wrote:
>>>>> mark florisson, 04.04.2011 21:26:
>>>>>> For clarity, I'll add an example:
>>>>>>
>>>>>> def f(np.ndarray[double] x, double alpha):
>>>>>>      cdef double s = 0
>>>>>>      cdef double tmp = 2
>>>>>>      cdef double other = 6.6
>>>>>>
>>>>>>      with nogil:
>>>>>>          for i in prange(x.shape[0]):
>>>>>>              # reading 'tmp' makes it firstprivate in addition to
>>>>>> lastprivate
>>>>>>              # 'other' is only ever read, so it's shared
>>>>>>              printf("%lf %lf %lf\n", tmp, s, other)
>>>>> So, adding a printf() to your code can change the semantics of your
>>>>> variables? That sounds like a really bad design to me.
>>>> That's what I was thinking. Basically, if you do an inlace operation,
>>>> then it's a reduction variable, no matter what else you do to it
>>>> (including possibly a direct assignment, though we could make that a
>>>> compile-time error).
>>> -1, I think that's too obscure. Not being able to use inplace operators
>>> for certain variables will be at the very least be nagging.
> You could still use inplace operators to your hearts content--just
> don't bother using the reduced variable outside the loop. (I guess I'm
> assuming reducing a variable has negligible performance overhead,
> which it should.) For the rare cases that you want the non-aggregated
> private, make an assignment to another variable, or use non-inplace
> operations.

Ahh! Of course! With some control flow analysis we could even eliminate 
the reduction if the variable isn't used after the loop, although I 
agree the cost should be trivial.


> Not being able to mix inplace operators might be an annoyance. We
> could also allow explicit declarations, as per Pauli's suggestion, but
> not require them. Essentially, as long as we have

I think you should be able to mix them, but if you do a reduction 
doesn't happen. This is slightly uncomfortable, but I believe control 
flow analysis and disabling firstprivate can solve it, see below.

I believe I'm back in the implicit-camp. And the CEP can probably be 
simplified a bit too, I'll try to do that tomorrow.

Two things:

  * It'd still be nice with something like a parallel block for thread 
setup/teardown rather than "if firstthreaditeration():". So, a prange 
for the 50% simplest cases, followed by a parallel-block for the next 30%.

  * Control flow analysis can help us tight it up a bit: For loops where 
you actually depend on values of thread-private variables computed in 
the previous iteration (beyond reduction), it'd be nice to raise a 
warning unless the variable is explicitly declared thread-local or 
similar. There are uses for such variables but they'd be rather rare, 
and such a hint could be very helpful.

I'm still not sure if we want firstprivate, even if we can do it. It'd 
be good to see a usecase for it. I'd rather have NaN and 0x7FFFFFFF 
personally, as relying on the firstprivate value is likely a bug -- yes, 
it makes the sequential case work, but that is exactly in the case where 
parallelizing the sequential case would be wrong!!

Grepping through 30000 lines of heavily OpenMP-ified Fortran code here 
there's no mention of firstprivate or lastprivate (although we certainly 
want lastprivate to align with the sequential case).

Dag Sverre


From markflorisson88 at gmail.com  Tue Apr  5 19:33:42 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 5 Apr 2011 19:33:42 +0200
Subject: [Cython] CEP: prange for parallel loops
In-Reply-To: <4D9B4419.5030804@astro.uio.no>
References: <4D999AD4.8080609@astro.uio.no> <4D99AA46.7020600@behnel.de>
	<4D99B12C.10201@astro.uio.no>
	<BANLkTimb-s3H18KZV6AXddgq=NKqCsGZYw@mail.gmail.com>
	<4D99FD70.6010408@astro.uio.no>
	<BANLkTin-USRY_+6AY5FDNkqQYsQUSU8KXg@mail.gmail.com>
	<4D9AF435.5080905@behnel.de>
	<BANLkTinGZ=gnNPwU7oOBgrRxcpMB+JNBkw@mail.gmail.com>
	<4D9B2DF9.8040305@astro.uio.no> <4D9B2F02.5050301@astro.uio.no>
	<BANLkTikWjU04CeDjOkd-HvDUsRcm+ZrAVA@mail.gmail.com>
	<4D9B4419.5030804@astro.uio.no>
Message-ID: <BANLkTik-qaMpiB6uCev95XpML1V4UUJUDQ@mail.gmail.com>

On 5 April 2011 18:32, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/05/2011 05:26 PM, Robert Bradshaw wrote:
>>
>> On Tue, Apr 5, 2011 at 8:02 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
>>>>
>>>> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>>>>>
>>>>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel<stefan_ml at behnel.de>
>>>>> ?wrote:
>>>>>>
>>>>>> mark florisson, 04.04.2011 21:26:
>>>>>>>
>>>>>>> For clarity, I'll add an example:
>>>>>>>
>>>>>>> def f(np.ndarray[double] x, double alpha):
>>>>>>> ? ? cdef double s = 0
>>>>>>> ? ? cdef double tmp = 2
>>>>>>> ? ? cdef double other = 6.6
>>>>>>>
>>>>>>> ? ? with nogil:
>>>>>>> ? ? ? ? for i in prange(x.shape[0]):
>>>>>>> ? ? ? ? ? ? # reading 'tmp' makes it firstprivate in addition to
>>>>>>> lastprivate
>>>>>>> ? ? ? ? ? ? # 'other' is only ever read, so it's shared
>>>>>>> ? ? ? ? ? ? printf("%lf %lf %lf\n", tmp, s, other)
>>>>>>
>>>>>> So, adding a printf() to your code can change the semantics of your
>>>>>> variables? That sounds like a really bad design to me.
>>>>>
>>>>> That's what I was thinking. Basically, if you do an inlace operation,
>>>>> then it's a reduction variable, no matter what else you do to it
>>>>> (including possibly a direct assignment, though we could make that a
>>>>> compile-time error).
>>>>
>>>> -1, I think that's too obscure. Not being able to use inplace operators
>>>> for certain variables will be at the very least be nagging.
>>
>> You could still use inplace operators to your hearts content--just
>> don't bother using the reduced variable outside the loop. (I guess I'm
>> assuming reducing a variable has negligible performance overhead,
>> which it should.) For the rare cases that you want the non-aggregated
>> private, make an assignment to another variable, or use non-inplace
>> operations.
>
> Ahh! Of course! With some control flow analysis we could even eliminate the
> reduction if the variable isn't used after the loop, although I agree the
> cost should be trivial.
>
>
>> Not being able to mix inplace operators might be an annoyance. We
>> could also allow explicit declarations, as per Pauli's suggestion, but
>> not require them. Essentially, as long as we have
>
> I think you should be able to mix them, but if you do a reduction doesn't
> happen. This is slightly uncomfortable, but I believe control flow analysis
> and disabling firstprivate can solve it, see below.
>
> I believe I'm back in the implicit-camp. And the CEP can probably be
> simplified a bit too, I'll try to do that tomorrow.
>
> Two things:
>
> ?* It'd still be nice with something like a parallel block for thread
> setup/teardown rather than "if firstthreaditeration():". So, a prange for
> the 50% simplest cases, followed by a parallel-block for the next 30%.

Definitely, I think it could also make way for things such as sections
etc, but I'll bring that up later :)

> ?* Control flow analysis can help us tight it up a bit: For loops where you
> actually depend on values of thread-private variables computed in the
> previous iteration (beyond reduction), it'd be nice to raise a warning
> unless the variable is explicitly declared thread-local or similar. There
> are uses for such variables but they'd be rather rare, and such a hint could
> be very helpful.
>
> I'm still not sure if we want firstprivate, even if we can do it. It'd be
> good to see a usecase for it. I'd rather have NaN and 0x7FFFFFFF personally,
> as relying on the firstprivate value is likely a bug -- yes, it makes the
> sequential case work, but that is exactly in the case where parallelizing
> the sequential case would be wrong!!

Yeah, I think if we go the implicit route then firstprivate might be
quite a surprise for users.

> Grepping through 30000 lines of heavily OpenMP-ified Fortran code here
> there's no mention of firstprivate or lastprivate (although we certainly
> want lastprivate to align with the sequential case).
>
> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Basically I'm fine with either implicit or explicit, although I think
the explicit case would be easier to understand for people that have
used OpenMP. In either case it would be nice to give prange a 'nogil'
option.

So to be clear, when we assign to a variable it will be lastprivate,
and when we assign to the subscript of a variable we make that
variable shared (unless it is declared inside the parallel with
block), right?

From d.s.seljebotn at astro.uio.no  Tue Apr  5 22:29:29 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 22:29:29 +0200
Subject: [Cython] prange CEP updated
Message-ID: <4D9B7BA9.2060509@astro.uio.no>

I've done a pretty major revision to the prange CEP, bringing in a lot 
of the feedback.

Thread-private variables are now split in two cases:

  i) The safe cases, which really require very little technical 
knowledge -> automatically inferred

  ii) As an advanced feature, unsafe cases that requires some knowledge 
of threading -> must be explicitly declared

I think this split simplifies things a great deal.

I'm rather excited over this now; this could turn out to be a really 
user-friendly and safe feature that would not only allow us to support 
OpenMP-like threading, but be more convenient to use in a range of 
common cases.

http://wiki.cython.org/enhancements/prange 
<http://wiki.cython.org/enhancements/prange#preview>

Dag Sverre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110405/62f6851b/attachment-0001.html>

From d.s.seljebotn at astro.uio.no  Tue Apr  5 22:47:41 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 05 Apr 2011 22:47:41 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4D9B7BA9.2060509@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
Message-ID: <4D9B7FED.4080500@astro.uio.no>

On 04/05/2011 10:29 PM, Dag Sverre Seljebotn wrote:
> I've done a pretty major revision to the prange CEP, bringing in a lot 
> of the feedback.
>
> Thread-private variables are now split in two cases:
>
>  i) The safe cases, which really require very little technical 
> knowledge -> automatically inferred
>
>  ii) As an advanced feature, unsafe cases that requires some knowledge 
> of threading -> must be explicitly declared
>
> I think this split simplifies things a great deal.
>
> I'm rather excited over this now; this could turn out to be a really 
> user-friendly and safe feature that would not only allow us to support 
> OpenMP-like threading, but be more convenient to use in a range of 
> common cases.
>
> http://wiki.cython.org/enhancements/prange 
> <http://wiki.cython.org/enhancements/prange#preview>

As a digression: threadlocal(int)-variables could also be supported 
elsewhere as syntax candy for the pythread.h Thread Local Storage, which 
would work well for fast TLS for any kind of threads (e.g., when using 
threading module).

Dag Sverre

(Sorry about the previous HTML-mail.)

From d.s.seljebotn at astro.uio.no  Wed Apr  6 09:34:06 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 06 Apr 2011 09:34:06 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
Message-ID: <4D9C176E.80406@astro.uio.no>

I just wanted to make everybody aware that there's a paper on Cython in 
this month's CiSE (http://cise.aip.org/).

http://dx.doi.org/10.1109/MCSE.2010.118 (paywall)

Researchers: Please consider citing this paper if Cython helps your 
research in non-trivial ways.

Dag Sverre

From dalcinl at gmail.com  Wed Apr  6 19:14:48 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 6 Apr 2011 14:14:48 -0300
Subject: [Cython] cython broken
Message-ID: <BANLkTi=H2MNcDZE1B9PoGc36+8GKK9BokA@mail.gmail.com>

Since the commit below, Cython fails to compile itself. That fix
requires further work and definitely more tests. If that is impossible
right now, I would ask the guilty parties to revert the change and
continue working on this the bug tracker and repo clones. Please try
to keep cython-dev repo clean.


commit 3069c3e516fc7336b003861881623f30e168849e
Author: Haoyu Bai <baihaoyu at gmail.com>
Date:   Thu Mar 31 04:19:14 2011 +0800

    fix T477 by refactor FuncDefNode, so cython decorators can applied
to cdef function


See yourself:

$ git checkout 3069c3e516fc7336b003861881623f30e168849e
Note: checking out '3069c3e516fc7336b003861881623f30e168849e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 3069c3e... fix T477 by refactor FuncDefNode, so cython
decorators can applied to cdef function
[dalcinl at trantor cython-dev]$ python setup.py --name
Compiling module Cython.Compiler.Scanning ...
Compiling module Cython.Compiler.Parsing ...
Compiling module Cython.Compiler.Code ...

Error compiling Cython file:
------------------------------------------------------------
...
        self.cname = cname
        self.text = text
        self.escaped_value = StringEncoding.escape_byte_string(byte_string)
        self.py_strings = None

    def get_py_string_const(self, encoding, identifier=None, is_str=False):
   ^
------------------------------------------------------------

Cython/Compiler/Code.py:316:4: Signature not compatible with previous
declaration

Error compiling Cython file:
------------------------------------------------------------
...
    cdef public object text
    cdef public object escaped_value
    cdef public dict py_strings

    @cython.locals(intern=bint, is_str=bint, is_unicode=bint)
    cpdef get_py_string_const(self, encoding, identifier=*, is_str=*)
                             ^
------------------------------------------------------------

Cython/Compiler/Code.pxd:62:30: Previous declaration is here
Compilation failed
Cython




-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From thomas.e.keller at gmail.com  Thu Apr  7 00:50:44 2011
From: thomas.e.keller at gmail.com (Thomas Keller)
Date: Wed, 6 Apr 2011 17:50:44 -0500
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9C176E.80406@astro.uio.no>
References: <4D9C176E.80406@astro.uio.no>
Message-ID: <BANLkTikLeR3h_nH=P+MauuWCmnzDoUnVWA@mail.gmail.com>

This article was well written and informative. I thank the six of you for
writing it.

Cheers,
TEK

On Wed, Apr 6, 2011 at 2:34 AM, Dag Sverre Seljebotn <
d.s.seljebotn at astro.uio.no> wrote:

> I just wanted to make everybody aware that there's a paper on Cython in
> this month's CiSE (http://cise.aip.org/).
>
> http://dx.doi.org/10.1109/MCSE.2010.118 (paywall)
>
> Researchers: Please consider citing this paper if Cython helps your
> research in non-trivial ways.
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110406/dc00401b/attachment.html>

From jason-sage at creativetrax.com  Thu Apr  7 00:56:29 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Wed, 06 Apr 2011 17:56:29 -0500
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9C176E.80406@astro.uio.no>
References: <4D9C176E.80406@astro.uio.no>
Message-ID: <4D9CEF9D.401@creativetrax.com>

On 4/6/11 2:34 AM, Dag Sverre Seljebotn wrote:
>
> Researchers: Please consider citing this paper if Cython helps your
> research in non-trivial ways.

Is this the canonical citation reference for Cython now?  If so, can 
this be mentioned on the Cython webpage somewhere that is prominent 
enough to be found?

Thanks,

Jason


From zstone at gmail.com  Thu Apr  7 01:40:19 2011
From: zstone at gmail.com (Zak Stone)
Date: Wed, 6 Apr 2011 19:40:19 -0400
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9CEF9D.401@creativetrax.com>
References: <4D9C176E.80406@astro.uio.no>
	<4D9CEF9D.401@creativetrax.com>
Message-ID: <BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>

>> Researchers: Please consider citing this paper if Cython helps your
>> research in non-trivial ways.
>
> Is this the canonical citation reference for Cython now? ?If so, can this be
> mentioned on the Cython webpage somewhere that is prominent enough to be
> found?

On a related note, would it be possible to post a preprint somewhere
that isn't behind a paywall? If that's allowed, I would be delighted
to share the preprint with friends to introduce them to Cython.

Thanks,
Zak

From robertwb at math.washington.edu  Thu Apr  7 02:12:48 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 6 Apr 2011 17:12:48 -0700
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
Message-ID: <BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>

On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone <zstone at gmail.com> wrote:
>>> Researchers: Please consider citing this paper if Cython helps your
>>> research in non-trivial ways.
>>
>> Is this the canonical citation reference for Cython now? ?If so, can this be
>> mentioned on the Cython webpage somewhere that is prominent enough to be
>> found?
>
> On a related note, would it be possible to post a preprint somewhere
> that isn't behind a paywall? If that's allowed, I would be delighted
> to share the preprint with friends to introduce them to Cython.

Yes, I think we can post the pre-print, though I'm opposed to making
this the "canonical citation" just because of this paywall.

- Robert

From zstone at gmail.com  Thu Apr  7 02:20:37 2011
From: zstone at gmail.com (Zak Stone)
Date: Wed, 6 Apr 2011 20:20:37 -0400
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
Message-ID: <BANLkTik+cNBsovtwSAbuX2Za9=4jZ5CjbA@mail.gmail.com>

>>>> Researchers: Please consider citing this paper if Cython helps your
>>>> research in non-trivial ways.
>>>
>>> Is this the canonical citation reference for Cython now? ?If so, can this be
>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>> found?
>>
>> On a related note, would it be possible to post a preprint somewhere
>> that isn't behind a paywall? If that's allowed, I would be delighted
>> to share the preprint with friends to introduce them to Cython.
>
> Yes, I think we can post the pre-print, though I'm opposed to making
> this the "canonical citation" just because of this paywall.

Agreed. Perhaps you could post the desired BibTeX citation text for
the official version and a link to the official version right next to
the preprint?

Zak

From baihaoyu at gmail.com  Thu Apr  7 07:22:16 2011
From: baihaoyu at gmail.com (Haoyu Bai)
Date: Thu, 7 Apr 2011 13:22:16 +0800
Subject: [Cython] cython broken
In-Reply-To: <BANLkTi=H2MNcDZE1B9PoGc36+8GKK9BokA@mail.gmail.com>
References: <BANLkTi=H2MNcDZE1B9PoGc36+8GKK9BokA@mail.gmail.com>
Message-ID: <BANLkTik95z3xh2_bra7h84AX-c5rUZ4Ynw@mail.gmail.com>

On Thu, Apr 7, 2011 at 1:14 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Since the commit below, Cython fails to compile itself. That fix
> requires further work and definitely more tests. If that is impossible
> right now, I would ask the guilty parties to revert the change and
> continue working on this the bug tracker and repo clones. Please try
> to keep cython-dev repo clean.
>
>

I'm investigating this. For now, please revert this. Meanwhile, I'll
try to get it fixed.

-- 
Haoyu BAI
School of Computing,
National University of Singapore.

From arthurdesribeiro at gmail.com  Thu Apr  7 07:46:31 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Thu, 7 Apr 2011 02:46:31 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
Message-ID: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>

I've wrote a proposal to the project: Reimplement C modules in CPython's
standard library in Cython.

I'd be glad if you could take a look a it and give me your feedback.

the link for the proposal is: http://wiki.cython.org/arthursribeiro

Thank you.

Best Regards

Arthur
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110407/fc0a59f1/attachment.html>

From d.s.seljebotn at astro.uio.no  Thu Apr  7 07:54:31 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 07:54:31 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no>
	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
Message-ID: <4D9D5197.3000206@astro.uio.no>

On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone<zstone at gmail.com>  wrote:
>>>> Researchers: Please consider citing this paper if Cython helps your
>>>> research in non-trivial ways.
>>> Is this the canonical citation reference for Cython now?  If so, can this be
>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>> found?
>> On a related note, would it be possible to post a preprint somewhere
>> that isn't behind a paywall? If that's allowed, I would be delighted
>> to share the preprint with friends to introduce them to Cython.
> Yes, I think we can post the pre-print, though I'm opposed to making
> this the "canonical citation" just because of this paywall.

Is this for ideological or practical reasons?

This is probably the only paper in a "real" journal for some time, and 
citations are going to boost the authors' citation counts. Nobody would 
actually look up the citation anyway simply to learn about Cython, 
they'd just Google it. So unless we're trying to hide the existence of 
the paper, I think we should make it the default citation until there's 
something better.

Next time we've got anything to share in a paper, let's do it here:

http://www.openresearchcomputation.com/

Although that wasn't around when we started writing the paper.

Posting the pre-print is a matter of making the necesarry references 
within it and formatting it.

http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615 
<http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615>

I'll fix it and post a link later today.

Dag Sverre

From r.rex at tu-bs.de  Thu Apr  7 09:12:38 2011
From: r.rex at tu-bs.de (=?ISO-8859-1?Q?Ren=E9_Rex?=)
Date: Thu, 7 Apr 2011 09:12:38 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTik+cNBsovtwSAbuX2Za9=4jZ5CjbA@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
	<BANLkTik+cNBsovtwSAbuX2Za9=4jZ5CjbA@mail.gmail.com>
Message-ID: <BANLkTi=0K8gA2aT=t5AmaU3-ShWWZ7mB4w@mail.gmail.com>

Hello

> Agreed. Perhaps you could post the desired BibTeX citation text for
> the official version and a link to the official version right next to
> the preprint?

BibTeX entry for your convecience:

@article{bradshaw2010cython,
  title={{CYTHON: THE BEST OF BOTH WORLDS}},
  author={Bradshaw, R. and Citro, C. and Seljebotn, D.S.},
  journal={CiSE 2011 Special Python Issue},
  pages={25},
  year={2010}
}

- Ren?

From stefan_ml at behnel.de  Thu Apr  7 09:21:28 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 07 Apr 2011 09:21:28 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTi=0K8gA2aT=t5AmaU3-ShWWZ7mB4w@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no>
	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>	<BANLkTik+cNBsovtwSAbuX2Za9=4jZ5CjbA@mail.gmail.com>
	<BANLkTi=0K8gA2aT=t5AmaU3-ShWWZ7mB4w@mail.gmail.com>
Message-ID: <4D9D65F8.8080503@behnel.de>

Ren? Rex, 07.04.2011 09:12:
>> Agreed. Perhaps you could post the desired BibTeX citation text for
>> the official version and a link to the official version right next to
>> the preprint?
>
> BibTeX entry for your convecience:
>
> @article{bradshaw2010cython,
>    title={{CYTHON: THE BEST OF BOTH WORLDS}},
>    author={Bradshaw, R. and Citro, C. and Seljebotn, D.S.},
>    journal={CiSE 2011 Special Python Issue},
>    pages={25},
>    year={2010}
> }

Looks rather incomplete to me.

Stefan

From r.rex at tu-bs.de  Thu Apr  7 09:34:59 2011
From: r.rex at tu-bs.de (=?ISO-8859-1?Q?Ren=E9_Rex?=)
Date: Thu, 7 Apr 2011 09:34:59 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9D65F8.8080503@behnel.de>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
	<BANLkTik+cNBsovtwSAbuX2Za9=4jZ5CjbA@mail.gmail.com>
	<BANLkTi=0K8gA2aT=t5AmaU3-ShWWZ7mB4w@mail.gmail.com>
	<4D9D65F8.8080503@behnel.de>
Message-ID: <BANLkTimvhX4OWAgy2NcL9cowa837E2NO2A@mail.gmail.com>

> Looks rather incomplete to me.

Your are right. I got the wrong document. It is from the sage website
and has the same title...
http://sage.math.washington.edu/tmp/stein-cise-comments-may22.pdf#page=29

This should be the correct entry:

@article{behnel2010cython,
  title={{Cython: The best of both worlds}},
  author={Behnel, S. and Bradshaw, R. and Citro, C. and Dalcin, L. and
Seljebotn, D.S. and Smith, K.},
  journal={Computing in Science and Engineering},
  issn={1521-9615},
  year={2010},
  publisher={IEEE Computer Society}
}

- Ren?

From stefan_ml at behnel.de  Thu Apr  7 10:00:41 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 07 Apr 2011 10:00:41 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9D5197.3000206@astro.uio.no>
References: <4D9C176E.80406@astro.uio.no>	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
	<4D9D5197.3000206@astro.uio.no>
Message-ID: <4D9D6F29.8080104@behnel.de>

Dag Sverre Seljebotn, 07.04.2011 07:54:
> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote:
>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>> research in non-trivial ways.
>>>> Is this the canonical citation reference for Cython now? If so, can
>>>> this be
>>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>>> found?
>>> On a related note, would it be possible to post a preprint somewhere
>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>> to share the preprint with friends to introduce them to Cython.
>> Yes, I think we can post the pre-print, though I'm opposed to making
>> this the "canonical citation" just because of this paywall.
>
> Is this for ideological or practical reasons?

Both.


> This is probably the only paper in a "real" journal for some time, and
> citations are going to boost the authors' citation counts. Nobody would
> actually look up the citation anyway simply to learn about Cython, they'd
> just Google it.

Depends on the reference. If it's just cited as "you know, Cython", people 
will either look for "Cython" directly and be happy, or they may look up 
the paper, see that it's paid, and keep searching, either for the paper or 
for the project. If it's cited as "in that paper, you can read about doing 
X with Cython", then people will try even harder to get at the paper. In 
either case, chances are that they need to invest more time because of the 
reference, compared to a plain link in a footnote. So citing this article 
is likely to be an inconvenience for interested readers of papers that cite it.


> So unless we're trying to hide the existence of the paper,
> I think we should make it the default citation until there's something better.

We should then at least get a PDF preprint version ready that contains the 
relevant metadata and put the exact same file up on everyone's homepage (so 
that search engines don't get confused and can simply hash-compare them).


> Next time we've got anything to share in a paper, let's do it here:
>
> http://www.openresearchcomputation.com/

Looks good to me (even though PyCon 2011 isn't really an "upcoming 
conference" ;).


> Posting the pre-print is a matter of making the necesarry references within
> it and formatting it.
>
> http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615
> <http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615>
>
> I'll fix it and post a link later today.

Great, thanks!

Stefan

From robertwb at math.washington.edu  Thu Apr  7 10:01:06 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 7 Apr 2011 01:01:06 -0700
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9D5197.3000206@astro.uio.no>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
	<4D9D5197.3000206@astro.uio.no>
Message-ID: <BANLkTimeBkyZ8b2UoDbGtf-kUj+Es7vUSw@mail.gmail.com>

On Wed, Apr 6, 2011 at 10:54 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>
>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone<zstone at gmail.com> ?wrote:
>>>>>
>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>> research in non-trivial ways.
>>>>
>>>> Is this the canonical citation reference for Cython now? ?If so, can
>>>> this be
>>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>>> found?
>>>
>>> On a related note, would it be possible to post a preprint somewhere
>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>> to share the preprint with friends to introduce them to Cython.
>>
>> Yes, I think we can post the pre-print, though I'm opposed to making
>> this the "canonical citation" just because of this paywall.
>
> Is this for ideological or practical reasons?

Both.

Actually, opposed is probably too strong of a word here. I'm
disinclined, but there isn't really a better option. Currently, people
usually just cite the website, for whatever that's worth.
http://scholar.google.com/scholar?q=cython

> This is probably the only paper in a "real" journal for some time, and
> citations are going to boost the authors' citation counts. Nobody would
> actually look up the citation anyway simply to learn about Cython, they'd
> just Google it. So unless we're trying to hide the existence of the paper, I
> think we should make it the default citation until there's something better.
>
> Next time we've got anything to share in a paper, let's do it here:
>
> http://www.openresearchcomputation.com/
>
> Although that wasn't around when we started writing the paper.

Or at least look into this more carefully. Some of CiSE's papers are
open access, I (naively) thought ours wouldn't be hard to get to
either. It is a nice paper though and I think it'll hit a nice
audience (who primarily won't even be aware that they're paying
through it indirectly through university overhead and monolithic
library subscriptions).

> Posting the pre-print is a matter of making the necesarry references within
> it and formatting it.
>
> http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615
> <http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615>
>
> I'll fix it and post a link later today.

Thanks.

- Robert

From d.s.seljebotn at astro.uio.no  Thu Apr  7 10:08:35 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 10:08:35 +0200
Subject: [Cython] CiSE Cython paper: Preprint up
Message-ID: <4D9D7103.7060506@astro.uio.no>

I should have put up this right away, sorry:

http://folk.uio.no/dagss/cython_cise.pdf

It is actually post-review, so it contains most things but some 
stylistic improvements and layout. Not sure about posting this on 
cython.org, but we could perhaps link to my webpage 
(http://folk.uio.no/dagss/) and say it is there...

The repo is here: https://github.com/dagss/cython-cise-postprint

If only the world could move to open access a bit quicker...

Dag Sverre

From d.s.seljebotn at astro.uio.no  Thu Apr  7 10:33:08 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 10:33:08 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9D6F29.8080104@behnel.de>
References: <4D9C176E.80406@astro.uio.no>	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>	<4D9D5197.3000206@astro.uio.no>
	<4D9D6F29.8080104@behnel.de>
Message-ID: <4D9D76C4.5070900@astro.uio.no>

On 04/07/2011 10:00 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 07.04.2011 07:54:
>> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote:
>>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>>> research in non-trivial ways.
>>>>> Is this the canonical citation reference for Cython now? If so, can
>>>>> this be
>>>>> mentioned on the Cython webpage somewhere that is prominent enough 
>>>>> to be
>>>>> found?
>>>> On a related note, would it be possible to post a preprint somewhere
>>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>>> to share the preprint with friends to introduce them to Cython.
>>> Yes, I think we can post the pre-print, though I'm opposed to making
>>> this the "canonical citation" just because of this paywall.
>>
>> Is this for ideological or practical reasons?
>
> Both.
>
>
>> This is probably the only paper in a "real" journal for some time, and
>> citations are going to boost the authors' citation counts. Nobody would
>> actually look up the citation anyway simply to learn about Cython, 
>> they'd
>> just Google it.
>
> Depends on the reference. If it's just cited as "you know, Cython", 
> people will either look for "Cython" directly and be happy, or they 
> may look up the paper, see that it's paid, and keep searching, either 
> for the paper or for the project. If it's cited as "in that paper, you 
> can read about doing X with Cython", then people will try even harder 
> to get at the paper. In either case, chances are that they need to 
> invest more time because of the reference, compared to a plain link in 
> a footnote. So citing this article is likely to be an inconvenience 
> for interested readers of papers that cite it.

I guess this depends on the paper and reader in question then. Myself 
I'd never bother with the paper but go right to the website. Citing is 
just "paying the authors of the software through improving their 
citation stats". Then again my field is unfortunately very much 
pyramid-scheme-inflicted.

I definitely think we should encourage giving a footnote as well.

How about just presenting the situation as it is in a "Citing Cython" 
section, and leave the decision up to who's citing Cython? ("If you 
don't like to cite a paywall paper, a website reference is OK. At any 
rate, please link to the website in a footnote the first time you 
mention Cython.")

Really, I hate the current situation as much as you do. But I see moving 
the world towards open access as the task of those whose already got a 
bit up the food chain; I'm just at the start of my PhD. (And it should 
be obvious I'm arguing with my own interests in mind here.)

Dag Sverre

From stefan_ml at behnel.de  Thu Apr  7 10:40:50 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 07 Apr 2011 10:40:50 +0200
Subject: [Cython] [cython-users] CiSE Cython paper: Preprint up
In-Reply-To: <4D9D7103.7060506@astro.uio.no>
References: <4D9D7103.7060506@astro.uio.no>
Message-ID: <4D9D7892.7030203@behnel.de>

Dag Sverre Seljebotn, 07.04.2011 10:08:
> The repo is here: https://github.com/dagss/cython-cise-postprint

To include metadata, you can change the hyperref setup near the top as follows:

"""
\RequirePackage[colorlinks,breaklinks,pdftex,
     linkcolor=InnerLinkColor,filecolor=OuterLinkColor,
     menucolor=OuterLinkColor,urlcolor=OuterLinkColor,
     citecolor=InnerLinkColor,

     pdfauthor={Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro 
Dalcin, Dag Sverre Seljebotn, Kurt Smith},
     pdftitle={Cython: The best of both worlds},
     pdfkeywords={Cython language, Cython programming, NumPy}

]{hyperref}
"""

Any more keywords to add?

There's also pdfsubject (and pdfcreator and pdfproducer), but that doesn't 
really apply here.

Stefan

From robertwb at math.washington.edu  Thu Apr  7 10:45:14 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 7 Apr 2011 01:45:14 -0700
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <4D9D76C4.5070900@astro.uio.no>
References: <4D9C176E.80406@astro.uio.no> <4D9CEF9D.401@creativetrax.com>
	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>
	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>
	<4D9D5197.3000206@astro.uio.no> <4D9D6F29.8080104@behnel.de>
	<4D9D76C4.5070900@astro.uio.no>
Message-ID: <BANLkTik7zunFEpaqZrd=70E28bxftbjV+Q@mail.gmail.com>

On Thu, Apr 7, 2011 at 1:33 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/07/2011 10:00 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 07.04.2011 07:54:
>>>
>>> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>>>
>>>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote:
>>>>>>>
>>>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>>>> research in non-trivial ways.
>>>>>>
>>>>>> Is this the canonical citation reference for Cython now? If so, can
>>>>>> this be
>>>>>> mentioned on the Cython webpage somewhere that is prominent enough to
>>>>>> be
>>>>>> found?
>>>>>
>>>>> On a related note, would it be possible to post a preprint somewhere
>>>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>>>> to share the preprint with friends to introduce them to Cython.
>>>>
>>>> Yes, I think we can post the pre-print, though I'm opposed to making
>>>> this the "canonical citation" just because of this paywall.
>>>
>>> Is this for ideological or practical reasons?
>>
>> Both.
>>
>>
>>> This is probably the only paper in a "real" journal for some time, and
>>> citations are going to boost the authors' citation counts. Nobody would
>>> actually look up the citation anyway simply to learn about Cython, they'd
>>> just Google it.
>>
>> Depends on the reference. If it's just cited as "you know, Cython", people
>> will either look for "Cython" directly and be happy, or they may look up the
>> paper, see that it's paid, and keep searching, either for the paper or for
>> the project. If it's cited as "in that paper, you can read about doing X
>> with Cython", then people will try even harder to get at the paper. In
>> either case, chances are that they need to invest more time because of the
>> reference, compared to a plain link in a footnote. So citing this article is
>> likely to be an inconvenience for interested readers of papers that cite it.
>
> I guess this depends on the paper and reader in question then. Myself I'd
> never bother with the paper but go right to the website. Citing is just
> "paying the authors of the software through improving their citation stats".
> Then again my field is unfortunately very much pyramid-scheme-inflicted.
>
> I definitely think we should encourage giving a footnote as well.
>
> How about just presenting the situation as it is in a "Citing Cython"
> section, and leave the decision up to who's citing Cython? ("If you don't
> like to cite a paywall paper, a website reference is OK. At any rate, please
> link to the website in a footnote the first time you mention Cython.")

Of course eventually it'd be nice if people just wrote "we coded this
up in Cython" and a reference felt as out of place there as if they
had provided a reference for Fortran or C :-). We're a long way from
there though.

> Really, I hate the current situation as much as you do. But I see moving the
> world towards open access as the task of those whose already got a bit up
> the food chain; I'm just at the start of my PhD. (And it should be obvious
> I'm arguing with my own interests in mind here.)
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Thu Apr  7 10:56:39 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 10:56:39 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTimeBkyZ8b2UoDbGtf-kUj+Es7vUSw@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no>
	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>	<4D9D5197.3000206@astro.uio.no>
	<BANLkTimeBkyZ8b2UoDbGtf-kUj+Es7vUSw@mail.gmail.com>
Message-ID: <4D9D7C47.9050501@astro.uio.no>

On 04/07/2011 10:01 AM, Robert Bradshaw wrote:
> On Wed, Apr 6, 2011 at 10:54 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone<zstone at gmail.com>    wrote:
>>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>>> research in non-trivial ways.
>>>>> Is this the canonical citation reference for Cython now?  If so, can
>>>>> this be
>>>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>>>> found?
>>>> On a related note, would it be possible to post a preprint somewhere
>>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>>> to share the preprint with friends to introduce them to Cython.
>>> Yes, I think we can post the pre-print, though I'm opposed to making
>>> this the "canonical citation" just because of this paywall.
>> Is this for ideological or practical reasons?
> Both.
>
> Actually, opposed is probably too strong of a word here. I'm
> disinclined, but there isn't really a better option. Currently, people
> usually just cite the website, for whatever that's worth.
> http://scholar.google.com/scholar?q=cython

And I don't think that's worth very much. To me it's really looking like 
CiSE citation or no citation at all.

>> Next time we've got anything to share in a paper, let's do it here:
>>
>> http://www.openresearchcomputation.com/
>>
>> Although that wasn't around when we started writing the paper.
> Or at least look into this more carefully. Some of CiSE's papers are
> open access, I (naively) thought ours wouldn't be hard to get to
> either. It is a nice paper though and I think it'll hit a nice
> audience (who primarily won't even be aware that they're paying
> through it indirectly through university overhead and monolithic
> library subscriptions).

I did the same mistake, because I couldn't see the paywall myself. At 
the time I actually had a hard time finding an internet connection that 
wouldn't transparently serve me the PDFs. And once I learned I figured 
it was a bit too late to back out.

I've learned a lot since then.

DS

From r.rex at tu-bs.de  Thu Apr  7 11:37:05 2011
From: r.rex at tu-bs.de (=?ISO-8859-1?Q?Ren=E9_Rex?=)
Date: Thu, 7 Apr 2011 11:37:05 +0200
Subject: [Cython] [cython-users] CiSE Cython paper: Preprint up
In-Reply-To: <4D9D7892.7030203@behnel.de>
References: <4D9D7103.7060506@astro.uio.no>
	<4D9D7892.7030203@behnel.de>
Message-ID: <BANLkTimaxYCR115xUTUQcnGxyVuDtORdYw@mail.gmail.com>

> Any more keywords to add?

What about "Python"? ;)

- Ren?

From d.s.seljebotn at astro.uio.no  Thu Apr  7 13:08:19 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 13:08:19 +0200
Subject: [Cython] [cython-users] CiSE Cython paper: Preprint up
In-Reply-To: <BANLkTimaxYCR115xUTUQcnGxyVuDtORdYw@mail.gmail.com>
References: <4D9D7103.7060506@astro.uio.no>	<4D9D7892.7030203@behnel.de>
	<BANLkTimaxYCR115xUTUQcnGxyVuDtORdYw@mail.gmail.com>
Message-ID: <4D9D9B23.4090900@astro.uio.no>

On 04/07/2011 11:37 AM, Ren? Rex wrote:
>> Any more keywords to add?
> What about "Python"? ;)
>

Done and done. Thanks for the patches.

DS

From d.s.seljebotn at astro.uio.no  Thu Apr  7 13:26:50 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 13:26:50 +0200
Subject: [Cython] Cython paper in Computing in Science & Engineering
In-Reply-To: <BANLkTimeBkyZ8b2UoDbGtf-kUj+Es7vUSw@mail.gmail.com>
References: <4D9C176E.80406@astro.uio.no>
	<4D9CEF9D.401@creativetrax.com>	<BANLkTinzeXzWwcnjY3=f5=rbqqAo8=vnjw@mail.gmail.com>	<BANLkTikfE9ggHWhOkFjEmWwb3j5iaYtZOw@mail.gmail.com>	<4D9D5197.3000206@astro.uio.no>
	<BANLkTimeBkyZ8b2UoDbGtf-kUj+Es7vUSw@mail.gmail.com>
Message-ID: <4D9D9F7A.9040600@astro.uio.no>

On 04/07/2011 10:01 AM, Robert Bradshaw wrote:
> On Wed, Apr 6, 2011 at 10:54 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone<zstone at gmail.com>    wrote:
>>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>>> research in non-trivial ways.
>>>>> Is this the canonical citation reference for Cython now?  If so, can
>>>>> this be
>>>>> mentioned on the Cython webpage somewhere that is prominent enough to be
>>>>> found?
>>>> On a related note, would it be possible to post a preprint somewhere
>>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>>> to share the preprint with friends to introduce them to Cython.
>>> Yes, I think we can post the pre-print, though I'm opposed to making
>>> this the "canonical citation" just because of this paywall.
>> Is this for ideological or practical reasons?
> Both.
>
> Actually, opposed is probably too strong of a word here. I'm
> disinclined, but there isn't really a better option. Currently, people
> usually just cite the website, for whatever that's worth.
> http://scholar.google.com/scholar?q=cython

OK, I wrote this:

http://wiki.cython.org/FAQ#HowdoIciteCythoninanacademicpaper.3F

If any of you can think of something better that that, just do it -- I 
won't start an edit war :-)


Dag Sverre

From stefan_ml at behnel.de  Thu Apr  7 13:46:08 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 07 Apr 2011 13:46:08 +0200
Subject: [Cython] Hudson pyregr testing takes too long
Message-ID: <4D9DA400.4060105@behnel.de>

Hi,

I just noticed that the CPython pyregr tests have jumped up from ~14 
minutes for a run to ~4 hours when we added generator support.

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend

I currently have no idea why that is (well, it's likely because we compile 
more tests now, but Vitja's branch ran the tests in ~30 minutes). It would 
be great if someone could find the time to analyse this problem. The 
current run time makes it basically impossible to keep these tests enabled.

Stefan

From stefan_ml at behnel.de  Thu Apr  7 13:52:00 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 07 Apr 2011 13:52:00 +0200
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <4D9DA400.4060105@behnel.de>
References: <4D9DA400.4060105@behnel.de>
Message-ID: <4D9DA560.8070505@behnel.de>

Stefan Behnel, 07.04.2011 13:46:
> I just noticed that the CPython pyregr tests have jumped up from ~14
> minutes for a run to ~4 hours when we added generator support.
>
> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>
>
> I currently have no idea why that is (well, it's likely because we compile
> more tests now, but Vitja's branch ran the tests in ~30 minutes). It would
> be great if someone could find the time to analyse this problem. The
> current run time makes it basically impossible to keep these tests enabled.

Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7 
tests take 30-45 minutes, which is very long, but not completely out of 
bounds. I've disabled the Py2.6 pyregr tests for now.

Stefan

From baihaoyu at gmail.com  Thu Apr  7 15:04:03 2011
From: baihaoyu at gmail.com (Haoyu Bai)
Date: Thu, 7 Apr 2011 21:04:03 +0800
Subject: [Cython] cython broken
In-Reply-To: <BANLkTik95z3xh2_bra7h84AX-c5rUZ4Ynw@mail.gmail.com>
References: <BANLkTi=H2MNcDZE1B9PoGc36+8GKK9BokA@mail.gmail.com>
	<BANLkTik95z3xh2_bra7h84AX-c5rUZ4Ynw@mail.gmail.com>
Message-ID: <BANLkTi=nass=UxkWJbShasa6tvEDS--5Ug@mail.gmail.com>

On Thu, Apr 7, 2011 at 1:22 PM, Haoyu Bai <baihaoyu at gmail.com> wrote:
> On Thu, Apr 7, 2011 at 1:14 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> Since the commit below, Cython fails to compile itself. That fix
>> requires further work and definitely more tests. If that is impossible
>> right now, I would ask the guilty parties to revert the change and
>> continue working on this the bug tracker and repo clones. Please try
>> to keep cython-dev repo clean.
>>
>>
>
> I'm investigating this. For now, please revert this. Meanwhile, I'll
> try to get it fixed.
>

I just started a pull request that fix the current compiling fail:

https://github.com/cython/cython/pull/21

Thanks!

-- 
Haoyu BAI
School of Computing,
National University of Singapore.

From romain.py at gmail.com  Thu Apr  7 17:01:11 2011
From: romain.py at gmail.com (Romain Guillebert)
Date: Thu, 7 Apr 2011 16:01:11 +0100
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Message-ID: <20110407150110.GA13395@ubuntu>

Hi

I proposed the Summer of Code project regarding the Python backend for
Cython.

As I said in my proposal this would translate Cython code to Python +
FFI code (I don't know yet if it will use ctypes or something specific
to PyPy). PyPy's ctypes is now really fast and this will allow people to
port their Cython code to PyPy.

For the moment I've been mostly in touch with the PyPy people and they
seem happy with my proposal.

Of course I'm available for questions.

Cheers
Romain

From d.s.seljebotn at astro.uio.no  Thu Apr  7 18:06:31 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 07 Apr 2011 18:06:31 +0200
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <20110407150110.GA13395@ubuntu>
References: <20110407150110.GA13395@ubuntu>
Message-ID: <4D9DE107.9040301@astro.uio.no>

On 04/07/2011 05:01 PM, Romain Guillebert wrote:
> Hi
>
> I proposed the Summer of Code project regarding the Python backend for
> Cython.
>
> As I said in my proposal this would translate Cython code to Python +
> FFI code (I don't know yet if it will use ctypes or something specific
> to PyPy). PyPy's ctypes is now really fast and this will allow people to
> port their Cython code to PyPy.
>
> For the moment I've been mostly in touch with the PyPy people and they
> seem happy with my proposal.
>
> Of course I'm available for questions.

Disclaimer: I haven't read the proposal (don't have access yet but will 
soon). So perhaps the below is redundant.

This seems similar to Carl Witty's port of Cython to .NET/IronPython. An 
important insight from that project is that Cython code does NOT specify 
an ABI, only an API which requires a C compiler to make sense. That is; 
many wrapped C libraries have plenty of macros, we only require partial 
definition of struct, we only require approximate typedef's, and so on.

In the .NET port, the consequence was that rather than the original idea 
of generating C# code (with FFI specifications) was dropped, and one 
instead went with C++/CLR (which is a proper C++ compiler that really 
understands the C side on an API level, in addition to giving access to 
the .NET runtime).

There are two ways around this:

  a) In addition to Python code, generate C code that can take (the 
friendlest) APIs and probe for the ABIs (such as, for instance, getting 
the offset of each struct field from the base pointer). Of course, this 
must really be rerun for each platform/build of the wrapped library.

Essentially, you'd use Cython to generate C code that, in a target 
build, would generate Python code...

  b) Create a subset of the Cython language ("RCython" :-)), where you 
require explicit ABIs (essentially this means either disallowing "cdef 
extern from ...", or creating some new form of it). Most Cython 
extensions I know about would not work with this though, so there would 
need to be porting in each case. Ideally one should then have a similar 
mode for Cython+CPython so that one can debug with CPython as well.


Dag Sverre

From carl.witty at gmail.com  Thu Apr  7 18:53:24 2011
From: carl.witty at gmail.com (Carl Witty)
Date: Thu, 7 Apr 2011 09:53:24 -0700
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <4D9DE107.9040301@astro.uio.no>
References: <20110407150110.GA13395@ubuntu> <4D9DE107.9040301@astro.uio.no>
Message-ID: <BANLkTinVdT=GAQTLCDzBx3DSS0byMHixbg@mail.gmail.com>

On Thu, Apr 7, 2011 at 9:06 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/07/2011 05:01 PM, Romain Guillebert wrote:
>>
>> Hi
>>
>> I proposed the Summer of Code project regarding the Python backend for
>> Cython.
>>
>> As I said in my proposal this would translate Cython code to Python +
>> FFI code (I don't know yet if it will use ctypes or something specific
>> to PyPy). PyPy's ctypes is now really fast and this will allow people to
>> port their Cython code to PyPy.
>>
>> For the moment I've been mostly in touch with the PyPy people and they
>> seem happy with my proposal.
>>
>> Of course I'm available for questions.
>
> Disclaimer: I haven't read the proposal (don't have access yet but will
> soon). So perhaps the below is redundant.
>
> This seems similar to Carl Witty's port of Cython to .NET/IronPython. An
> important insight from that project is that Cython code does NOT specify an
> ABI, only an API which requires a C compiler to make sense. That is; many
> wrapped C libraries have plenty of macros, we only require partial
> definition of struct, we only require approximate typedef's, and so on.
>
> In the .NET port, the consequence was that rather than the original idea of
> generating C# code (with FFI specifications) was dropped, and one instead
> went with C++/CLR (which is a proper C++ compiler that really understands
> the C side on an API level, in addition to giving access to the .NET
> runtime).
>
> There are two ways around this:
>
> ?a) In addition to Python code, generate C code that can take (the
> friendlest) APIs and probe for the ABIs (such as, for instance, getting the
> offset of each struct field from the base pointer). Of course, this must
> really be rerun for each platform/build of the wrapped library.
>
> Essentially, you'd use Cython to generate C code that, in a target build,
> would generate Python code...
>
> ?b) Create a subset of the Cython language ("RCython" :-)), where you
> require explicit ABIs (essentially this means either disallowing "cdef
> extern from ...", or creating some new form of it). Most Cython extensions I
> know about would not work with this though, so there would need to be
> porting in each case. Ideally one should then have a similar mode for
> Cython+CPython so that one can debug with CPython as well.

Note that a) is not sufficient in general -- it doesn't handle macros
that expand into code, like errno and putc().  There's another option
I considered,

c) Given the API specification in the Cython file, generate C code
that wraps that API with a known ABI.  So for:

cdef extern from "<errno.h>":
    int errno

you would generate a C file something like:

#include <errno.h>

void _write_errno(int newval) {
  errno = newval;
}

int _read_errno() {
  return errno;
}

and for

cdef extern from "<sys/types.h>":
    ctypedef int ino_t

cdef extern from "<sys/stat.h>":
    cdef struct stat:
        ino_t st_ino

you would generate (in part):

#include <sys/types.h>
#include <sys/stat.h>

long long _read_struct_stat_st_ino(struct stat *ptr) {
    return ptr->st_ino;
}

void _write_struct_stat_st_ino(struct stat *ptr, long long newval) {
    ptr->st_ino = newval;
}

(Of course, you'd want to add more name mangling to these examples.)

Note that I use "long long" for st_ino even though the Cython code
claimed that st_ino was int; that's because Cython generates code that
would work even if st_ino were "long long", and probably some modules
would break if you used the types declared in Cython.

Also, you could combine a), b), and c).  For example, use a) to
determine struct sizes, type sizes, and field offsets; use b) when
you're not worried about macros; and use c) (perhaps triggered by a
new annotation in the Cython source) when you want to handle arbitrary
API's that may be implemented with macros.

Carl

From carl.witty at gmail.com  Thu Apr  7 20:09:37 2011
From: carl.witty at gmail.com (Carl Witty)
Date: Thu, 7 Apr 2011 11:09:37 -0700
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <4D9DE107.9040301@astro.uio.no>
References: <20110407150110.GA13395@ubuntu> <4D9DE107.9040301@astro.uio.no>
Message-ID: <BANLkTi=-3S3ALaqP+08mi4iUM7DQkbjVZA@mail.gmail.com>

On Thu, Apr 7, 2011 at 9:06 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> This seems similar to Carl Witty's port of Cython to .NET/IronPython. An
> important insight from that project is that Cython code does NOT specify an
> ABI, only an API which requires a C compiler to make sense. That is; many
> wrapped C libraries have plenty of macros, we only require partial
> definition of struct, we only require approximate typedef's, and so on.
>
> In the .NET port, the consequence was that rather than the original idea of
> generating C# code (with FFI specifications) was dropped, and one instead
> went with C++/CLR (which is a proper C++ compiler that really understands
> the C side on an API level, in addition to giving access to the .NET
> runtime).

Let me just add that a way to deal with the API vs. ABI issue would be
useful for other potential Cython targets as well, such as IronPython
using C# and Jython.  (A C# port for IronPython would be more valuable
than my C++/CLI port because it would work under Mono -- Mono doesn't
have a C++/CLI compiler and probably never will.)

Carl

From romain.py at gmail.com  Thu Apr  7 22:36:35 2011
From: romain.py at gmail.com (Romain)
Date: Thu, 7 Apr 2011 21:36:35 +0100
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <BANLkTi=-3S3ALaqP+08mi4iUM7DQkbjVZA@mail.gmail.com>
References: <20110407150110.GA13395@ubuntu> <4D9DE107.9040301@astro.uio.no>
	<BANLkTi=-3S3ALaqP+08mi4iUM7DQkbjVZA@mail.gmail.com>
Message-ID: <BANLkTinwFX_AW5s4i66+5TbfQZc_qAjmRg@mail.gmail.com>

Hi again

PyPy has functions to parse C headers to get macros and constants so I could
create C functions to wrap the macros and probably inline constants in the
Python part of the wrapper. This doesn't solve the problem of ifdefs but
this is a start.

Cheers
Romain

2011/4/7 Carl Witty <carl.witty at gmail.com>

> On Thu, Apr 7, 2011 at 9:06 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
> > This seems similar to Carl Witty's port of Cython to .NET/IronPython. An
> > important insight from that project is that Cython code does NOT specify
> an
> > ABI, only an API which requires a C compiler to make sense. That is; many
> > wrapped C libraries have plenty of macros, we only require partial
> > definition of struct, we only require approximate typedef's, and so on.
> >
> > In the .NET port, the consequence was that rather than the original idea
> of
> > generating C# code (with FFI specifications) was dropped, and one instead
> > went with C++/CLR (which is a proper C++ compiler that really understands
> > the C side on an API level, in addition to giving access to the .NET
> > runtime).
>
> Let me just add that a way to deal with the API vs. ABI issue would be
> useful for other potential Cython targets as well, such as IronPython
> using C# and Jython.  (A C# port for IronPython would be more valuable
> than my C++/CLI port because it would work under Mono -- Mono doesn't
> have a C++/CLI compiler and probably never will.)
>
> Carl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110407/ce640a47/attachment.html>

From arthurdesribeiro at gmail.com  Thu Apr  7 23:31:00 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Thu, 7 Apr 2011 18:31:00 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
Message-ID: <BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>

I've submitted to google the link is:
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#

<http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#>It
would be really important if you could give me a feedback to my proposal...

Thank you

Best Regards

Arthur

2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>

> I've wrote a proposal to the project: Reimplement C modules in CPython's
> standard library in Cython.
>
> I'd be glad if you could take a look a it and give me your feedback.
>
> the link for the proposal is: http://wiki.cython.org/arthursribeiro
>
> Thank you.
>
> Best Regards
>
> Arthur
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110407/bb996c6b/attachment.html>

From stefan_ml at behnel.de  Fri Apr  8 01:08:50 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Apr 2011 01:08:50 +0200
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <BANLkTinwFX_AW5s4i66+5TbfQZc_qAjmRg@mail.gmail.com>
References: <20110407150110.GA13395@ubuntu>
	<4D9DE107.9040301@astro.uio.no>	<BANLkTi=-3S3ALaqP+08mi4iUM7DQkbjVZA@mail.gmail.com>
	<BANLkTinwFX_AW5s4i66+5TbfQZc_qAjmRg@mail.gmail.com>
Message-ID: <4D9E4402.7040004@behnel.de>

[fixed up the citation order]

Romain, 07.04.2011 22:36:
> 2011/4/7 Carl Witty
>
>> On Thu, Apr 7, 2011 at 9:06 AM, Dag Sverre Seljebotn wrote:
>>> This seems similar to Carl Witty's port of Cython to .NET/IronPython. An
>>> important insight from that project is that Cython code does NOT specify
>> an
>>> ABI, only an API which requires a C compiler to make sense. That is; many
>>> wrapped C libraries have plenty of macros, we only require partial
>>> definition of struct, we only require approximate typedef's, and so on.
>>>
>>> In the .NET port, the consequence was that rather than the original idea
>> of
>>> generating C# code (with FFI specifications) was dropped, and one instead
>>> went with C++/CLR (which is a proper C++ compiler that really understands
>>> the C side on an API level, in addition to giving access to the .NET
>>> runtime).
>>
>> Let me just add that a way to deal with the API vs. ABI issue would be
>> useful for other potential Cython targets as well, such as IronPython
>> using C# and Jython.  (A C# port for IronPython would be more valuable
>> than my C++/CLI port because it would work under Mono -- Mono doesn't
>> have a C++/CLI compiler and probably never will.)
>
> PyPy has functions to parse C headers to get macros and constants so I could
> create C functions to wrap the macros and probably inline constants in the
> Python part of the wrapper. This doesn't solve the problem of ifdefs but
> this is a start.

Yes, I think this is the only way this can be handled. In the worst case, 
you'd have to additionally fire up a real C preprocessor and let it parse 
the referenced header files in order to get at the platform specific 
declarations, which should then usually be good enough to figure out the ABI.

Stefan

From robertwb at math.washington.edu  Fri Apr  8 01:08:59 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 7 Apr 2011 16:08:59 -0700
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
Message-ID: <BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>

On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
<arthurdesribeiro at gmail.com> wrote:
> I've submitted to google the link
> is:?http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> It would be really important if you could give me a feedback to my
> proposal...
> Thank you
> Best Regards
> Arthur

Some quick points:

- Python ships with extensive regression tests--use (and possibly
augment) those to test your work rather than writing your own.
- Three modules for a whole summer seems a bit weak, especially for
someone who already knows Cython. Target at least one module/week
seems like a good pace; some will be quickies, others might take 40+
hours. And I guarantee you'll get better and faster with practice.
- Now that generators are supported, it could also be interesting to
look at compiling all the non-C modules and fixing exposed bugs if
any, but that might be out of scope.

What I'd like to see is an implementation of a single simple but not
entirely trivial (e.g. not math) module, passing regression tests with
comprable if not better speed than the current C version (though I
think it'd probably make sense to start out with the Python version
and optimize that). E.g. http://docs.python.org/library/json.html
looks like a good candidate. That should only take 8 hours or so,
maybe two days at most, given your background. I'm not expecting
anything before the application deadline, but if you could whip
something like this out in the next week to point to that would help
your application out immensely. In fact, one of the Python
foundation's requirements is that students submit a patch before being
accepted, and this would knock out that requirement and give you a
chance to prove yourself. Create an account on https://github.com and
commit your code into a new repository there.

Hope that helps.

- Robert


> 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>>
>> I've wrote a proposal to the project:?Reimplement C modules in CPython's
>> standard library in Cython.
>> I'd be glad if you could take a look a it and give me your feedback.
>> the link for the proposal is:?http://wiki.cython.org/arthursribeiro
>> Thank you.
>> Best Regards
>> Arthur
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From arthurdesribeiro at gmail.com  Fri Apr  8 02:43:29 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Thu, 7 Apr 2011 21:43:29 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
Message-ID: <BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>

2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>

> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
> <arthurdesribeiro at gmail.com> wrote:
> > I've submitted to google the link
> > is:
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> > It would be really important if you could give me a feedback to my
> > proposal...
> > Thank you
> > Best Regards
> > Arthur
>
> Some quick points:
>
> - Python ships with extensive regression tests--use (and possibly
> augment) those to test your work rather than writing your own.
>

Thank you for that information Robert, I didn't realize this.


> - Three modules for a whole summer seems a bit weak, especially for
> someone who already knows Cython. Target at least one module/week
> seems like a good pace; some will be quickies, others might take 40+
> hours. And I guarantee you'll get better and faster with practice.


I'm going to refactor this Robert, as soon as I  remake my project's roadmap
I'll send to you again.


> - Now that generators are supported, it could also be interesting to
> look at compiling all the non-C modules and fixing exposed bugs if
> any, but that might be out of scope.
>

I will try to take a look at this after implementing some cython code to a
the module you suggested.


>
> What I'd like to see is an implementation of a single simple but not
> entirely trivial (e.g. not math) module, passing regression tests with
> comprable if not better speed than the current C version (though I
> think it'd probably make sense to start out with the Python version
> and optimize that). E.g. http://docs.python.org/library/json.html
> looks like a good candidate. That should only take 8 hours or so,
> maybe two days at most, given your background. I'm not expecting
> anything before the application deadline, but if you could whip
> something like this out in the next week to point to that would help
> your application out immensely. In fact, one of the Python
> foundation's requirements is that students submit a patch before being
> accepted, and this would knock out that requirement and give you a
> chance to prove yourself. Create an account on https://github.com and
> commit your code into a new repository there.
>
>
I will start the implementation of json module right now. I created my
github account and as soon as I have code implemented I will send repository
link.

Thanks for all the points you listed, I will work on all of them and send an
e-mail.

Best Regards.

[]s

Arthur


Hope that helps.
>
> - Robert
>
>
> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
> >>
> >> I've wrote a proposal to the project: Reimplement C modules in CPython's
> >> standard library in Cython.
> >> I'd be glad if you could take a look a it and give me your feedback.
> >> the link for the proposal is: http://wiki.cython.org/arthursribeiro
> >> Thank you.
> >> Best Regards
> >> Arthur
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110407/2a518cf6/attachment.html>

From arthurdesribeiro at gmail.com  Fri Apr  8 08:38:46 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Fri, 8 Apr 2011 03:38:46 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
Message-ID: <BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>

I've made some changes to my proposal, as you said, I changed the number of
modules I'm going to reimplement, I jumped from three to twelve modules,
what modules are these and when I want to implement is described at:

http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#

<http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#>
and

http://wiki.cython.org/arthursribeiro

<http://wiki.cython.org/arthursribeiro>If you could take another look I
would appreciate a lot.

Best Regards.

[]s

Arthur

2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>

>
>
> 2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>
>
>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
>> <arthurdesribeiro at gmail.com> wrote:
>> > I've submitted to google the link
>> > is:
>> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
>> > It would be really important if you could give me a feedback to my
>> > proposal...
>> > Thank you
>> > Best Regards
>> > Arthur
>>
>> Some quick points:
>>
>> - Python ships with extensive regression tests--use (and possibly
>> augment) those to test your work rather than writing your own.
>>
>
> Thank you for that information Robert, I didn't realize this.
>
>
>> - Three modules for a whole summer seems a bit weak, especially for
>> someone who already knows Cython. Target at least one module/week
>> seems like a good pace; some will be quickies, others might take 40+
>> hours. And I guarantee you'll get better and faster with practice.
>
>
> I'm going to refactor this Robert, as soon as I  remake my project's
> roadmap I'll send to you again.
>
>
>> - Now that generators are supported, it could also be interesting to
>> look at compiling all the non-C modules and fixing exposed bugs if
>> any, but that might be out of scope.
>>
>
> I will try to take a look at this after implementing some cython code to a
> the module you suggested.
>
>
>>
>> What I'd like to see is an implementation of a single simple but not
>> entirely trivial (e.g. not math) module, passing regression tests with
>> comprable if not better speed than the current C version (though I
>> think it'd probably make sense to start out with the Python version
>> and optimize that). E.g. http://docs.python.org/library/json.html
>> looks like a good candidate. That should only take 8 hours or so,
>> maybe two days at most, given your background. I'm not expecting
>> anything before the application deadline, but if you could whip
>> something like this out in the next week to point to that would help
>> your application out immensely. In fact, one of the Python
>> foundation's requirements is that students submit a patch before being
>> accepted, and this would knock out that requirement and give you a
>> chance to prove yourself. Create an account on https://github.com and
>> commit your code into a new repository there.
>>
>>
> I will start the implementation of json module right now. I created my
> github account and as soon as I have code implemented I will send repository
> link.
>
> Thanks for all the points you listed, I will work on all of them and send
> an e-mail.
>
> Best Regards.
>
> []s
>
> Arthur
>
>
> Hope that helps.
>>
>> - Robert
>>
>>
>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>> >>
>> >> I've wrote a proposal to the project: Reimplement C modules in
>> CPython's
>> >> standard library in Cython.
>> >> I'd be glad if you could take a look a it and give me your feedback.
>> >> the link for the proposal is: http://wiki.cython.org/arthursribeiro
>> >> Thank you.
>> >> Best Regards
>> >> Arthur
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > http://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110408/4e0e682e/attachment-0001.html>

From robertwb at math.washington.edu  Fri Apr  8 10:50:45 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 8 Apr 2011 01:50:45 -0700
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
Message-ID: <BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>

Looking better. I would add some details about how you plan to
compare/profile your new implementations, and also at least add a note
that compatibility will be ensured with the Python regression tests.

It may make sense to let the exact list of modules be somewhat
flexible, for example, based on feedback from the Python and Cython
community on what would be the most worthwhile to Cythonize. Maybe the
final milestone would be "re-implement several additional modules as
chosen by the Python community to provide maximum value" or something
like that.

- Robert

On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro
<arthurdesribeiro at gmail.com> wrote:
> I've made some changes to my proposal, as you said, I changed the number of
> modules I'm going to reimplement, I jumped from three to twelve modules,
> what modules are these and when I want to implement is described at:
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> and
> http://wiki.cython.org/arthursribeiro
> If you could take another look I would appreciate a lot.
> Best Regards.
> []s
> Arthur
>
> 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>>
>>
>> 2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>
>>>
>>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
>>> <arthurdesribeiro at gmail.com> wrote:
>>> > I've submitted to google the link
>>> >
>>> > is:?http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
>>> > It would be really important if you could give me a feedback to my
>>> > proposal...
>>> > Thank you
>>> > Best Regards
>>> > Arthur
>>>
>>> Some quick points:
>>>
>>> - Python ships with extensive regression tests--use (and possibly
>>> augment) those to test your work rather than writing your own.
>>
>> Thank you for that information Robert, I didn't realize this.
>>
>>>
>>> - Three modules for a whole summer seems a bit weak, especially for
>>> someone who already knows Cython. Target at least one module/week
>>> seems like a good pace; some will be quickies, others might take 40+
>>> hours. And I guarantee you'll get better and faster with practice.
>>
>> I'm going to refactor this Robert, as soon as I ?remake my project's
>> roadmap I'll send to you again.
>>
>>>
>>> - Now that generators are supported, it could also be interesting to
>>> look at compiling all the non-C modules and fixing exposed bugs if
>>> any, but that might be out of scope.
>>
>> I will try to take a look at this after implementing some cython code to a
>> the module you suggested.
>>
>>>
>>> What I'd like to see is an implementation of a single simple but not
>>> entirely trivial (e.g. not math) module, passing regression tests with
>>> comprable if not better speed than the current C version (though I
>>> think it'd probably make sense to start out with the Python version
>>> and optimize that). E.g. http://docs.python.org/library/json.html
>>> looks like a good candidate. That should only take 8 hours or so,
>>> maybe two days at most, given your background. I'm not expecting
>>> anything before the application deadline, but if you could whip
>>> something like this out in the next week to point to that would help
>>> your application out immensely. In fact, one of the Python
>>> foundation's requirements is that students submit a patch before being
>>> accepted, and this would knock out that requirement and give you a
>>> chance to prove yourself. Create an account on https://github.com and
>>> commit your code into a new repository there.
>>>
>>
>> I will start the implementation of json module right now. I created my
>> github account and as soon as I have code implemented I will send repository
>> link.
>> Thanks for all the points you listed, I will work on all of them and send
>> an e-mail.
>> Best Regards.
>> []s
>> Arthur
>>
>>> Hope that helps.
>>>
>>> - Robert
>>>
>>>
>>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>>> >>
>>> >> I've wrote a proposal to the project:?Reimplement C modules in
>>> >> CPython's
>>> >> standard library in Cython.
>>> >> I'd be glad if you could take a look a it and give me your feedback.
>>> >> the link for the proposal is:?http://wiki.cython.org/arthursribeiro
>>> >> Thank you.
>>> >> Best Regards
>>> >> Arthur
>>> >
>>> > _______________________________________________
>>> > cython-devel mailing list
>>> > cython-devel at python.org
>>> > http://mail.python.org/mailman/listinfo/cython-devel
>>> >
>>> >
>>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From stefan_ml at behnel.de  Fri Apr  8 08:50:35 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Apr 2011 08:50:35 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
Message-ID: <4D9EB03B.4020909@behnel.de>

Robert Bradshaw, 08.04.2011 01:08:
> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro wrote:
>> I've submitted to google the link
>> is: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
>> It would be really important if you could give me a feedback to my
>> proposal...
>> Thank you
>> Best Regards
>> Arthur
>
> Some quick points:
>
> - Python ships with extensive regression tests--use (and possibly
> augment) those to test your work rather than writing your own.
> - Three modules for a whole summer seems a bit weak, especially for
> someone who already knows Cython. Target at least one module/week
> seems like a good pace; some will be quickies, others might take 40+
> hours. And I guarantee you'll get better and faster with practice.

Absolutely. There certainly are tricky parts in the C code, and optimising 
the Cython/Python code won't come for free, either, but after a little bit 
of exercise this should run quite fluently.


> - Now that generators are supported, it could also be interesting to
> look at compiling all the non-C modules and fixing exposed bugs if
> any, but that might be out of scope.
>
> What I'd like to see is an implementation of a single simple but not
> entirely trivial (e.g. not math) module, passing regression tests with
> comprable if not better speed than the current C version (though I
> think it'd probably make sense to start out with the Python version
> and optimize that). E.g. http://docs.python.org/library/json.html
> looks like a good candidate.

Right, that's a good one. Clearly more maintenance critical than purely 
time critical, and likely a good candidate for making it both more Python 
compatible (function argument handling?) and maybe even faster than the 
original. And if it can be implemented/optimised in Python syntax, that'd 
drop the maintenance overhead of the binary module by some 99%.


> That should only take 8 hours or so,
> maybe two days at most, given your background. I'm not expecting
> anything before the application deadline, but if you could whip
> something like this out in the next week to point to that would help
> your application out immensely.

+1


> In fact, one of the Python
> foundation's requirements is that students submit a patch before being
> accepted, and this would knock out that requirement and give you a
> chance to prove yourself. Create an account on https://github.com and
> commit your code into a new repository there.

Maybe even clone it from CPython's own stdlib repository in hg.

Stefan

From arthurdesribeiro at gmail.com  Fri Apr  8 19:31:12 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Fri, 8 Apr 2011 14:31:12 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
	<BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
Message-ID: <BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>

2011/4/8 Robert Bradshaw <robertwb at math.washington.edu>

> Looking better. I would add some details about how you plan to
> compare/profile your new implementations, and also at least add a note
> that compatibility will be ensured with the Python regression tests.
>

I'm going to update this on the wiki page, it's just because the application
period in google-melange ends today and it's not recommended to put details
os how you plan to make the project in there. About the comparison, I wonder
talk to the community a little more about it, because I see in cython's web
page and in the tutorials numbers telling how much more efficient cython is
against python, so I planned to use the same strategy that is used to show
the numbers that are there. And about the tests, I'm studying how can I
check compatibility and do these tasks to put that on wiki too.


>
> It may make sense to let the exact list of modules be somewhat
> flexible, for example, based on feedback from the Python and Cython
> community on what would be the most worthwhile to Cythonize. Maybe the
> final milestone would be "re-implement several additional modules as
> chosen by the Python community to provide maximum value" or something
> like that.
>

You mean just to tell how many modules I'm going to re-implement, but not
telling what modules are these? Re-implementing by community demand?

Thank you very much again.

Best Regards.

[]s

Arthur


>
> - Robert
>
> On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro
> <arthurdesribeiro at gmail.com> wrote:
> > I've made some changes to my proposal, as you said, I changed the number
> of
> > modules I'm going to reimplement, I jumped from three to twelve modules,
> > what modules are these and when I want to implement is described at:
> >
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> > and
> > http://wiki.cython.org/arthursribeiro
> > If you could take another look I would appreciate a lot.
> > Best Regards.
> > []s
> > Arthur
> >
> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
> >>
> >>
> >> 2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>
> >>>
> >>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
> >>> <arthurdesribeiro at gmail.com> wrote:
> >>> > I've submitted to google the link
> >>> >
> >>> > is:
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> >>> > It would be really important if you could give me a feedback to my
> >>> > proposal...
> >>> > Thank you
> >>> > Best Regards
> >>> > Arthur
> >>>
> >>> Some quick points:
> >>>
> >>> - Python ships with extensive regression tests--use (and possibly
> >>> augment) those to test your work rather than writing your own.
> >>
> >> Thank you for that information Robert, I didn't realize this.
> >>
> >>>
> >>> - Three modules for a whole summer seems a bit weak, especially for
> >>> someone who already knows Cython. Target at least one module/week
> >>> seems like a good pace; some will be quickies, others might take 40+
> >>> hours. And I guarantee you'll get better and faster with practice.
> >>
> >> I'm going to refactor this Robert, as soon as I  remake my project's
> >> roadmap I'll send to you again.
> >>
> >>>
> >>> - Now that generators are supported, it could also be interesting to
> >>> look at compiling all the non-C modules and fixing exposed bugs if
> >>> any, but that might be out of scope.
> >>
> >> I will try to take a look at this after implementing some cython code to
> a
> >> the module you suggested.
> >>
> >>>
> >>> What I'd like to see is an implementation of a single simple but not
> >>> entirely trivial (e.g. not math) module, passing regression tests with
> >>> comprable if not better speed than the current C version (though I
> >>> think it'd probably make sense to start out with the Python version
> >>> and optimize that). E.g. http://docs.python.org/library/json.html
> >>> looks like a good candidate. That should only take 8 hours or so,
> >>> maybe two days at most, given your background. I'm not expecting
> >>> anything before the application deadline, but if you could whip
> >>> something like this out in the next week to point to that would help
> >>> your application out immensely. In fact, one of the Python
> >>> foundation's requirements is that students submit a patch before being
> >>> accepted, and this would knock out that requirement and give you a
> >>> chance to prove yourself. Create an account on https://github.com and
> >>> commit your code into a new repository there.
> >>>
> >>
> >> I will start the implementation of json module right now. I created my
> >> github account and as soon as I have code implemented I will send
> repository
> >> link.
> >> Thanks for all the points you listed, I will work on all of them and
> send
> >> an e-mail.
> >> Best Regards.
> >> []s
> >> Arthur
> >>
> >>> Hope that helps.
> >>>
> >>> - Robert
> >>>
> >>>
> >>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
> >>> >>
> >>> >> I've wrote a proposal to the project: Reimplement C modules in
> >>> >> CPython's
> >>> >> standard library in Cython.
> >>> >> I'd be glad if you could take a look a it and give me your feedback.
> >>> >> the link for the proposal is: http://wiki.cython.org/arthursribeiro
> >>> >> Thank you.
> >>> >> Best Regards
> >>> >> Arthur
> >>> >
> >>> > _______________________________________________
> >>> > cython-devel mailing list
> >>> > cython-devel at python.org
> >>> > http://mail.python.org/mailman/listinfo/cython-devel
> >>> >
> >>> >
> >>
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110408/b6da7a26/attachment.html>

From robertwb at math.washington.edu  Fri Apr  8 19:40:55 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 8 Apr 2011 10:40:55 -0700
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
	<BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
	<BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>
Message-ID: <BANLkTimH4WKq1u0Oaq38VTnSu0bG7tOZ5A@mail.gmail.com>

On Fri, Apr 8, 2011 at 10:31 AM, Arthur de Souza Ribeiro
<arthurdesribeiro at gmail.com> wrote:
>
>
> 2011/4/8 Robert Bradshaw <robertwb at math.washington.edu>
>>
>> Looking better. I would add some details about how you plan to
>> compare/profile your new implementations, and also at least add a note
>> that compatibility will be ensured with the Python regression tests.
>
> I'm going to update this on the wiki page, it's just because the application
> period in google-melange ends today and it's not recommended to put details
> os how you plan to make the project in there. About the comparison, I wonder
> talk to the community a little more about it, because I see in cython's web
> page and in the tutorials numbers telling how much more efficient cython is
> against python, so I planned to use the same strategy that is used to show
> the numbers that are there. And about the tests, I'm studying how can I
> check compatibility and do these tasks to put that on wiki too.
>
>>
>> It may make sense to let the exact list of modules be somewhat
>> flexible, for example, based on feedback from the Python and Cython
>> community on what would be the most worthwhile to Cythonize. Maybe the
>> final milestone would be "re-implement several additional modules as
>> chosen by the Python community to provide maximum value" or something
>> like that.
>
> You mean just to tell how many modules I'm going to re-implement, but not
> telling what modules are these? Re-implementing by community demand?

Yes, exactly (for the last milestone, I think it's good to have more
direction at the start as well as have something to point to when
soliciting feedback. Maybe say "at least three" as it might be some
big ones or a handful of little ones. You, I, and everyone else will
have a better idea of what'll be most profitable at this point.

> Thank you very much again.

No problem, thanks for your interest.

> Best Regards.
> []s
> Arthur
>
>>
>> - Robert
>>
>> On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro
>> <arthurdesribeiro at gmail.com> wrote:
>> > I've made some changes to my proposal, as you said, I changed the number
>> > of
>> > modules I'm going to reimplement, I jumped from three to twelve modules,
>> > what modules are these and when I want to implement is described at:
>> >
>> > http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
>> > and
>> > http://wiki.cython.org/arthursribeiro
>> > If you could take another look I would appreciate a lot.
>> > Best Regards.
>> > []s
>> > Arthur
>> >
>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>> >>
>> >>
>> >> 2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>
>> >>>
>> >>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
>> >>> <arthurdesribeiro at gmail.com> wrote:
>> >>> > I've submitted to google the link
>> >>> >
>> >>> >
>> >>> > is:?http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
>> >>> > It would be really important if you could give me a feedback to my
>> >>> > proposal...
>> >>> > Thank you
>> >>> > Best Regards
>> >>> > Arthur
>> >>>
>> >>> Some quick points:
>> >>>
>> >>> - Python ships with extensive regression tests--use (and possibly
>> >>> augment) those to test your work rather than writing your own.
>> >>
>> >> Thank you for that information Robert, I didn't realize this.
>> >>
>> >>>
>> >>> - Three modules for a whole summer seems a bit weak, especially for
>> >>> someone who already knows Cython. Target at least one module/week
>> >>> seems like a good pace; some will be quickies, others might take 40+
>> >>> hours. And I guarantee you'll get better and faster with practice.
>> >>
>> >> I'm going to refactor this Robert, as soon as I ?remake my project's
>> >> roadmap I'll send to you again.
>> >>
>> >>>
>> >>> - Now that generators are supported, it could also be interesting to
>> >>> look at compiling all the non-C modules and fixing exposed bugs if
>> >>> any, but that might be out of scope.
>> >>
>> >> I will try to take a look at this after implementing some cython code
>> >> to a
>> >> the module you suggested.
>> >>
>> >>>
>> >>> What I'd like to see is an implementation of a single simple but not
>> >>> entirely trivial (e.g. not math) module, passing regression tests with
>> >>> comprable if not better speed than the current C version (though I
>> >>> think it'd probably make sense to start out with the Python version
>> >>> and optimize that). E.g. http://docs.python.org/library/json.html
>> >>> looks like a good candidate. That should only take 8 hours or so,
>> >>> maybe two days at most, given your background. I'm not expecting
>> >>> anything before the application deadline, but if you could whip
>> >>> something like this out in the next week to point to that would help
>> >>> your application out immensely. In fact, one of the Python
>> >>> foundation's requirements is that students submit a patch before being
>> >>> accepted, and this would knock out that requirement and give you a
>> >>> chance to prove yourself. Create an account on https://github.com and
>> >>> commit your code into a new repository there.
>> >>>
>> >>
>> >> I will start the implementation of json module right now. I created my
>> >> github account and as soon as I have code implemented I will send
>> >> repository
>> >> link.
>> >> Thanks for all the points you listed, I will work on all of them and
>> >> send
>> >> an e-mail.
>> >> Best Regards.
>> >> []s
>> >> Arthur
>> >>
>> >>> Hope that helps.
>> >>>
>> >>> - Robert
>> >>>
>> >>>
>> >>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
>> >>> >>
>> >>> >> I've wrote a proposal to the project:?Reimplement C modules in
>> >>> >> CPython's
>> >>> >> standard library in Cython.
>> >>> >> I'd be glad if you could take a look a it and give me your
>> >>> >> feedback.
>> >>> >> the link for the proposal is:?http://wiki.cython.org/arthursribeiro
>> >>> >> Thank you.
>> >>> >> Best Regards
>> >>> >> Arthur
>> >>> >
>> >>> > _______________________________________________
>> >>> > cython-devel mailing list
>> >>> > cython-devel at python.org
>> >>> > http://mail.python.org/mailman/listinfo/cython-devel
>> >>> >
>> >>> >
>> >>
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > http://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From arthurdesribeiro at gmail.com  Fri Apr  8 19:59:36 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Fri, 8 Apr 2011 14:59:36 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimH4WKq1u0Oaq38VTnSu0bG7tOZ5A@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
	<BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
	<BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>
	<BANLkTimH4WKq1u0Oaq38VTnSu0bG7tOZ5A@mail.gmail.com>
Message-ID: <BANLkTinwpW2hHqH3BC+7LTyp-de8duzcTg@mail.gmail.com>

The moduels suggested for the first two milestones you think are ok?

Best Regards..

[]s

Arthur

2011/4/8 Robert Bradshaw <robertwb at math.washington.edu>

> On Fri, Apr 8, 2011 at 10:31 AM, Arthur de Souza Ribeiro
> <arthurdesribeiro at gmail.com> wrote:
> >
> >
> > 2011/4/8 Robert Bradshaw <robertwb at math.washington.edu>
> >>
> >> Looking better. I would add some details about how you plan to
> >> compare/profile your new implementations, and also at least add a note
> >> that compatibility will be ensured with the Python regression tests.
> >
> > I'm going to update this on the wiki page, it's just because the
> application
> > period in google-melange ends today and it's not recommended to put
> details
> > os how you plan to make the project in there. About the comparison, I
> wonder
> > talk to the community a little more about it, because I see in cython's
> web
> > page and in the tutorials numbers telling how much more efficient cython
> is
> > against python, so I planned to use the same strategy that is used to
> show
> > the numbers that are there. And about the tests, I'm studying how can I
> > check compatibility and do these tasks to put that on wiki too.
> >
> >>
> >> It may make sense to let the exact list of modules be somewhat
> >> flexible, for example, based on feedback from the Python and Cython
> >> community on what would be the most worthwhile to Cythonize. Maybe the
> >> final milestone would be "re-implement several additional modules as
> >> chosen by the Python community to provide maximum value" or something
> >> like that.
> >
> > You mean just to tell how many modules I'm going to re-implement, but not
> > telling what modules are these? Re-implementing by community demand?
>
> Yes, exactly (for the last milestone, I think it's good to have more
> direction at the start as well as have something to point to when
> soliciting feedback. Maybe say "at least three" as it might be some
> big ones or a handful of little ones. You, I, and everyone else will
> have a better idea of what'll be most profitable at this point.
>
> > Thank you very much again.
>
> No problem, thanks for your interest.
>
> > Best Regards.
> > []s
> > Arthur
> >
> >>
> >> - Robert
> >>
> >> On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro
> >> <arthurdesribeiro at gmail.com> wrote:
> >> > I've made some changes to my proposal, as you said, I changed the
> number
> >> > of
> >> > modules I'm going to reimplement, I jumped from three to twelve
> modules,
> >> > what modules are these and when I want to implement is described at:
> >> >
> >> >
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> >> > and
> >> > http://wiki.cython.org/arthursribeiro
> >> > If you could take another look I would appreciate a lot.
> >> > Best Regards.
> >> > []s
> >> > Arthur
> >> >
> >> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
> >> >>
> >> >>
> >> >> 2011/4/7 Robert Bradshaw <robertwb at math.washington.edu>
> >> >>>
> >> >>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro
> >> >>> <arthurdesribeiro at gmail.com> wrote:
> >> >>> > I've submitted to google the link
> >> >>> >
> >> >>> >
> >> >>> > is:
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1#
> >> >>> > It would be really important if you could give me a feedback to my
> >> >>> > proposal...
> >> >>> > Thank you
> >> >>> > Best Regards
> >> >>> > Arthur
> >> >>>
> >> >>> Some quick points:
> >> >>>
> >> >>> - Python ships with extensive regression tests--use (and possibly
> >> >>> augment) those to test your work rather than writing your own.
> >> >>
> >> >> Thank you for that information Robert, I didn't realize this.
> >> >>
> >> >>>
> >> >>> - Three modules for a whole summer seems a bit weak, especially for
> >> >>> someone who already knows Cython. Target at least one module/week
> >> >>> seems like a good pace; some will be quickies, others might take 40+
> >> >>> hours. And I guarantee you'll get better and faster with practice.
> >> >>
> >> >> I'm going to refactor this Robert, as soon as I  remake my project's
> >> >> roadmap I'll send to you again.
> >> >>
> >> >>>
> >> >>> - Now that generators are supported, it could also be interesting to
> >> >>> look at compiling all the non-C modules and fixing exposed bugs if
> >> >>> any, but that might be out of scope.
> >> >>
> >> >> I will try to take a look at this after implementing some cython code
> >> >> to a
> >> >> the module you suggested.
> >> >>
> >> >>>
> >> >>> What I'd like to see is an implementation of a single simple but not
> >> >>> entirely trivial (e.g. not math) module, passing regression tests
> with
> >> >>> comprable if not better speed than the current C version (though I
> >> >>> think it'd probably make sense to start out with the Python version
> >> >>> and optimize that). E.g. http://docs.python.org/library/json.html
> >> >>> looks like a good candidate. That should only take 8 hours or so,
> >> >>> maybe two days at most, given your background. I'm not expecting
> >> >>> anything before the application deadline, but if you could whip
> >> >>> something like this out in the next week to point to that would help
> >> >>> your application out immensely. In fact, one of the Python
> >> >>> foundation's requirements is that students submit a patch before
> being
> >> >>> accepted, and this would knock out that requirement and give you a
> >> >>> chance to prove yourself. Create an account on https://github.comand
> >> >>> commit your code into a new repository there.
> >> >>>
> >> >>
> >> >> I will start the implementation of json module right now. I created
> my
> >> >> github account and as soon as I have code implemented I will send
> >> >> repository
> >> >> link.
> >> >> Thanks for all the points you listed, I will work on all of them and
> >> >> send
> >> >> an e-mail.
> >> >> Best Regards.
> >> >> []s
> >> >> Arthur
> >> >>
> >> >>> Hope that helps.
> >> >>>
> >> >>> - Robert
> >> >>>
> >> >>>
> >> >>> > 2011/4/7 Arthur de Souza Ribeiro <arthurdesribeiro at gmail.com>
> >> >>> >>
> >> >>> >> I've wrote a proposal to the project: Reimplement C modules in
> >> >>> >> CPython's
> >> >>> >> standard library in Cython.
> >> >>> >> I'd be glad if you could take a look a it and give me your
> >> >>> >> feedback.
> >> >>> >> the link for the proposal is:
> http://wiki.cython.org/arthursribeiro
> >> >>> >> Thank you.
> >> >>> >> Best Regards
> >> >>> >> Arthur
> >> >>> >
> >> >>> > _______________________________________________
> >> >>> > cython-devel mailing list
> >> >>> > cython-devel at python.org
> >> >>> > http://mail.python.org/mailman/listinfo/cython-devel
> >> >>> >
> >> >>> >
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > cython-devel mailing list
> >> > cython-devel at python.org
> >> > http://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> >
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> http://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110408/d2c3633c/attachment.html>

From markflorisson88 at gmail.com  Fri Apr  8 20:03:40 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 8 Apr 2011 20:03:40 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTinwpW2hHqH3BC+7LTyp-de8duzcTg@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
	<BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
	<BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>
	<BANLkTimH4WKq1u0Oaq38VTnSu0bG7tOZ5A@mail.gmail.com>
	<BANLkTinwpW2hHqH3BC+7LTyp-de8duzcTg@mail.gmail.com>
Message-ID: <BANLkTim_6qDuEpMexRXQtk_jPgqzwEtDXg@mail.gmail.com>

On 8 April 2011 19:59, Arthur de Souza Ribeiro
<arthurdesribeiro at gmail.com> wrote:
> The moduels suggested for the first two milestones you think are ok?
> Best Regards..
> []s
> Arthur

You mention the 'dis' module, but isn't that one (and 'opcode' too)
entirely written in Python?

From arthurdesribeiro at gmail.com  Fri Apr  8 20:12:00 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Fri, 8 Apr 2011 15:12:00 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTim_6qDuEpMexRXQtk_jPgqzwEtDXg@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<BANLkTimzKO5ReeZneyGBnkK1ODqQAg=-8w@mail.gmail.com>
	<BANLkTinLmLhM-RrkasZZTnx28Vj0pCP3Nw@mail.gmail.com>
	<BANLkTimiez+3hubcJEsRi5d3sLwU2fRzww@mail.gmail.com>
	<BANLkTimH4WKq1u0Oaq38VTnSu0bG7tOZ5A@mail.gmail.com>
	<BANLkTinwpW2hHqH3BC+7LTyp-de8duzcTg@mail.gmail.com>
	<BANLkTim_6qDuEpMexRXQtk_jPgqzwEtDXg@mail.gmail.com>
Message-ID: <BANLkTinN5ZXze-mJ0+mqMbRiivHRG3TEhQ@mail.gmail.com>

My mistake, I just typed wrong, I was talking about the nis one... By the
way, I changed this module to the array one...

Best Regards.

Arthur

2011/4/8 mark florisson <markflorisson88 at gmail.com>

> On 8 April 2011 19:59, Arthur de Souza Ribeiro
> <arthurdesribeiro at gmail.com> wrote:
> > The moduels suggested for the first two milestones you think are ok?
> > Best Regards..
> > []s
> > Arthur
>
> You mention the 'dis' module, but isn't that one (and 'opcode' too)
> entirely written in Python?
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110408/582fb727/attachment-0001.html>

From markflorisson88 at gmail.com  Fri Apr  8 23:38:56 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 8 Apr 2011 23:38:56 +0200
Subject: [Cython] GSoC Proposal for Supporting Parallelism,
 Fused Types and Typed Views on Memory
Message-ID: <BANLkTikzFZc6Vn7MN2ETCHAnZFdqYKkp8g@mail.gmail.com>

Hey,

My GSoC proposal can be found here:
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/markflorisson88/1
. It's about implementing the prange CEP (524), Fused Types (522) and
Typed Memory Views (517).

I really hope to participate this year, but due to time constraints I
may not find the required time to actually complete the GSoC, so next
week I will decide whether I'll chicken out or not.

Cheers,

Mark

From jason-sage at creativetrax.com  Sat Apr  9 13:49:51 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Sat, 09 Apr 2011 06:49:51 -0500
Subject: [Cython] cython-docs repository
Message-ID: <4DA047DF.9040000@creativetrax.com>

What is the relationship between the cython-docs repository and the 
docs/ subdirectory of the cython repository?  I see a recent commit [1] 
that seems to indicate that cython-docs has been merged into the main 
cython repository (+1 from me, for what it's worth).  Is my 
interpretation correct, and is cython-docs now deprecated?

I submitted a pull request for some typos [2], but I'm not sure if I 
should have submitted that pull request to the cython-docs repository.

Thanks,

Jason


[1] 
https://github.com/cython/cython/commit/2bcb14fff262a9a9c7b50bacb360bd983e6a92ee

[2] https://github.com/cython/cython/pull/22

--
Jason Grout

From robertwb at math.washington.edu  Sat Apr  9 19:02:52 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 9 Apr 2011 10:02:52 -0700
Subject: [Cython] cython-docs repository
In-Reply-To: <4DA047DF.9040000@creativetrax.com>
References: <4DA047DF.9040000@creativetrax.com>
Message-ID: <BANLkTikiYQaye8hWyASBCFVmd1chiAMCdg@mail.gmail.com>

On Sat, Apr 9, 2011 at 4:49 AM, Jason Grout <jason-sage at creativetrax.com> wrote:
> What is the relationship between the cython-docs repository and the docs/
> subdirectory of the cython repository? ?I see a recent commit [1] that seems
> to indicate that cython-docs has been merged into the main cython repository
> (+1 from me, for what it's worth). ?Is my interpretation correct, and is
> cython-docs now deprecated?

Yep, we did that during the workshop. I thought I had sent out an
announcement, but I guess not.

> I submitted a pull request for some typos [2], but I'm not sure if I should
> have submitted that pull request to the cython-docs repository.

Thanks, I'll take a look.

- Robert

From jason-sage at creativetrax.com  Sat Apr  9 19:13:35 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Sat, 09 Apr 2011 12:13:35 -0500
Subject: [Cython] cython-docs repository
In-Reply-To: <BANLkTikiYQaye8hWyASBCFVmd1chiAMCdg@mail.gmail.com>
References: <4DA047DF.9040000@creativetrax.com>
	<BANLkTikiYQaye8hWyASBCFVmd1chiAMCdg@mail.gmail.com>
Message-ID: <4DA093BF.7050306@creativetrax.com>

On 4/9/11 12:02 PM, Robert Bradshaw wrote:
> Yep, we did that during the workshop. I thought I had sent out an
> announcement, but I guess not.

Is there a summary anywhere of the exciting things that happened in the 
workshop?  For example, it seems that generators are finally in, if I 
read the commit logs correctly.  Is that true?  If so, fantastic!  Any 
idea of a timeline for that to make it into an official release?

Thanks,

Jason


From markflorisson88 at gmail.com  Mon Apr 11 10:45:34 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 10:45:34 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4D9B7BA9.2060509@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
Message-ID: <BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>

On 5 April 2011 22:29, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> I've done a pretty major revision to the prange CEP, bringing in a lot of
> the feedback.
>
> Thread-private variables are now split in two cases:
>
> ?i) The safe cases, which really require very little technical knowledge ->
> automatically inferred
>
> ?ii) As an advanced feature, unsafe cases that requires some knowledge of
> threading -> must be explicitly declared
>
> I think this split simplifies things a great deal.

Can't we obsolete the declaration entirely by assigning to variables
that need to have firstprivate behaviour inside the with parallel
block? Basically in the same way the scratch space is used. The only
problem with that is that it won't be lastprivate, so the value will
be undefined after the parallel block (but not after the worksharing
loop).

cdef int myvariable

with nogil, parallel:
    myvariable = 2
    for i in prange(...):
        use myvariable
        maybe assign to myvariable

    # myvariable is well-defined here

# myvariable is not well-defined here

If you still desperately want lastprivate behaviour you can simply
assign myvariable to another variable in the loop body.

> I'm rather excited over this now; this could turn out to be a really
> user-friendly and safe feature that would not only allow us to support
> OpenMP-like threading, but be more convenient to use in a range of common
> cases.
>
> http://wiki.cython.org/enhancements/prange
>
> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From d.s.seljebotn at astro.uio.no  Mon Apr 11 11:10:52 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 11 Apr 2011 11:10:52 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>
Message-ID: <4DA2C59C.8080703@astro.uio.no>

On 04/11/2011 10:45 AM, mark florisson wrote:
> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>> the feedback.
>>
>> Thread-private variables are now split in two cases:
>>
>>   i) The safe cases, which really require very little technical knowledge ->
>> automatically inferred
>>
>>   ii) As an advanced feature, unsafe cases that requires some knowledge of
>> threading ->  must be explicitly declared
>>
>> I think this split simplifies things a great deal.
>
> Can't we obsolete the declaration entirely by assigning to variables
> that need to have firstprivate behaviour inside the with parallel
> block? Basically in the same way the scratch space is used. The only
> problem with that is that it won't be lastprivate, so the value will
> be undefined after the parallel block (but not after the worksharing
> loop).
>
> cdef int myvariable
>
> with nogil, parallel:
>      myvariable = 2
>      for i in prange(...):
>          use myvariable
>          maybe assign to myvariable
>
>      # myvariable is well-defined here
>
> # myvariable is not well-defined here
>
> If you still desperately want lastprivate behaviour you can simply
> assign myvariable to another variable in the loop body.

I don't care about lastprivate, I don't think that is an issue, as you say.

My problem with this is that it means going into an area where possibly 
tricky things are implicit rather than explicit. I also see this as a 
rather special case that will be seldomly used, and implicit behaviour 
is more difficult to justify because of that.

(The other instance of thread-local variables I feel is still explicit: 
You use prange instead of range, which means that you declare that 
values created in the iteration does not leak to the next iteration. The 
rest is just optimization from there.)

As Robert said in his recent talk: A lot of languages are easy to write. 
The advantage of Python is that it is easy to *read*. That's what I feel 
is wrong with the proposal above: An assignment to a variable changes 
the semantics of it. Granted, it happens in a way so that it will almost 
always be correct, but I feel that reading the code, I'd spend some 
extra cycles to go "ah, so this variable is thread-local and therefore 
its values survive across a loop iteration".

If I even knew about the feature in the first place. In seeing 
"threadprivate" spelled out, it is either obvious what it means, or 
obvious that I should look up the docs.

There's *a lot* of things that can be made implicit in a programming 
language; Python/Cython simply usually leans towards the explicit side.

Oh, and we may want to support writable shared variables (and flush) 
eventually too, and the above doesn't easily differentiate there?

That's just my opinion, I'm happy to be overruled here.

Dag Sverre

From markflorisson88 at gmail.com  Mon Apr 11 11:41:14 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 11:41:14 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA2C59C.8080703@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>
	<4DA2C59C.8080703@astro.uio.no>
Message-ID: <BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>

On 11 April 2011 11:10, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/11/2011 10:45 AM, mark florisson wrote:
>>
>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>> the feedback.
>>>
>>> Thread-private variables are now split in two cases:
>>>
>>> ?i) The safe cases, which really require very little technical knowledge
>>> ->
>>> automatically inferred
>>>
>>> ?ii) As an advanced feature, unsafe cases that requires some knowledge of
>>> threading -> ?must be explicitly declared
>>>
>>> I think this split simplifies things a great deal.
>>
>> Can't we obsolete the declaration entirely by assigning to variables
>> that need to have firstprivate behaviour inside the with parallel
>> block? Basically in the same way the scratch space is used. The only
>> problem with that is that it won't be lastprivate, so the value will
>> be undefined after the parallel block (but not after the worksharing
>> loop).
>>
>> cdef int myvariable
>>
>> with nogil, parallel:
>> ? ? myvariable = 2
>> ? ? for i in prange(...):
>> ? ? ? ? use myvariable
>> ? ? ? ? maybe assign to myvariable
>>
>> ? ? # myvariable is well-defined here
>>
>> # myvariable is not well-defined here
>>
>> If you still desperately want lastprivate behaviour you can simply
>> assign myvariable to another variable in the loop body.
>
> I don't care about lastprivate, I don't think that is an issue, as you say.
>
> My problem with this is that it means going into an area where possibly
> tricky things are implicit rather than explicit. I also see this as a rather
> special case that will be seldomly used, and implicit behaviour is more
> difficult to justify because of that.

Indeed, I actually considered if we should support firstprivate at
all, as it's really about "being firstprivate and lastprivate".
Without any declaration, you can have firstprivate or lastprivate, but
not both :) So I agree that supporting such a (probably) uncommon case
is better left explicit. On the other hand it seems silly to have
support for such a weird case.

> (The other instance of thread-local variables I feel is still explicit: You
> use prange instead of range, which means that you declare that values
> created in the iteration does not leak to the next iteration. The rest is
> just optimization from there.)
>
> As Robert said in his recent talk: A lot of languages are easy to write. The
> advantage of Python is that it is easy to *read*. That's what I feel is
> wrong with the proposal above: An assignment to a variable changes the
> semantics of it. Granted, it happens in a way so that it will almost always
> be correct, but I feel that reading the code, I'd spend some extra cycles to
> go "ah, so this variable is thread-local and therefore its values survive
> across a loop iteration".
>
> If I even knew about the feature in the first place. In seeing
> "threadprivate" spelled out, it is either obvious what it means, or obvious
> that I should look up the docs.
>
> There's *a lot* of things that can be made implicit in a programming
> language; Python/Cython simply usually leans towards the explicit side.
>
> Oh, and we may want to support writable shared variables (and flush)
> eventually too, and the above doesn't easily differentiate there?

Right, everything is implicit. So I guess it'll be good to introduce
it anyway as you say, so we can later declare stuff shared with
similar syntax. I suppose that's the point where I'm convinced.

> That's just my opinion, I'm happy to be overruled here.
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Mon Apr 11 12:08:11 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 11 Apr 2011 12:08:11 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>	<4DA2C59C.8080703@astro.uio.no>
	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>
Message-ID: <4DA2D30B.8040209@astro.uio.no>

On 04/11/2011 11:41 AM, mark florisson wrote:
> On 11 April 2011 11:10, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>
>>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>   wrote:
>>>>
>>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>>> the feedback.
>>>>
>>>> Thread-private variables are now split in two cases:
>>>>
>>>>   i) The safe cases, which really require very little technical knowledge
>>>> ->
>>>> automatically inferred
>>>>
>>>>   ii) As an advanced feature, unsafe cases that requires some knowledge of
>>>> threading ->    must be explicitly declared
>>>>
>>>> I think this split simplifies things a great deal.
>>>
>>> Can't we obsolete the declaration entirely by assigning to variables
>>> that need to have firstprivate behaviour inside the with parallel
>>> block? Basically in the same way the scratch space is used. The only
>>> problem with that is that it won't be lastprivate, so the value will
>>> be undefined after the parallel block (but not after the worksharing
>>> loop).
>>>
>>> cdef int myvariable
>>>
>>> with nogil, parallel:
>>>      myvariable = 2
>>>      for i in prange(...):
>>>          use myvariable
>>>          maybe assign to myvariable
>>>
>>>      # myvariable is well-defined here
>>>
>>> # myvariable is not well-defined here
>>>
>>> If you still desperately want lastprivate behaviour you can simply
>>> assign myvariable to another variable in the loop body.
>>
>> I don't care about lastprivate, I don't think that is an issue, as you say.
>>
>> My problem with this is that it means going into an area where possibly
>> tricky things are implicit rather than explicit. I also see this as a rather
>> special case that will be seldomly used, and implicit behaviour is more
>> difficult to justify because of that.
>
> Indeed, I actually considered if we should support firstprivate at
> all, as it's really about "being firstprivate and lastprivate".
> Without any declaration, you can have firstprivate or lastprivate, but
> not both :) So I agree that supporting such a (probably) uncommon case
> is better left explicit. On the other hand it seems silly to have
> support for such a weird case.

Well, I actually need to do the per-thread cache thing I described in 
the CEP in my own codes, so it's not *that* special; it'd be nice to 
support it.

OTOH I *could* work around it by having an array of scalars

cdef int[:] old_ell = int[:numthreads]()

...
     if old_ell[threadid()] != ell: ...


So I guess, it's at least on the bottom of list of priorities in that CEP.

Dag Sverre

From markflorisson88 at gmail.com  Mon Apr 11 12:14:36 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 12:14:36 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA2D30B.8040209@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>
	<4DA2C59C.8080703@astro.uio.no>
	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>
	<4DA2D30B.8040209@astro.uio.no>
Message-ID: <BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>

On 11 April 2011 12:08, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/11/2011 11:41 AM, mark florisson wrote:
>>
>> On 11 April 2011 11:10, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>
>>>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>> ?wrote:
>>>>>
>>>>> I've done a pretty major revision to the prange CEP, bringing in a lot
>>>>> of
>>>>> the feedback.
>>>>>
>>>>> Thread-private variables are now split in two cases:
>>>>>
>>>>> ?i) The safe cases, which really require very little technical
>>>>> knowledge
>>>>> ->
>>>>> automatically inferred
>>>>>
>>>>> ?ii) As an advanced feature, unsafe cases that requires some knowledge
>>>>> of
>>>>> threading -> ? ?must be explicitly declared
>>>>>
>>>>> I think this split simplifies things a great deal.
>>>>
>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>> that need to have firstprivate behaviour inside the with parallel
>>>> block? Basically in the same way the scratch space is used. The only
>>>> problem with that is that it won't be lastprivate, so the value will
>>>> be undefined after the parallel block (but not after the worksharing
>>>> loop).
>>>>
>>>> cdef int myvariable
>>>>
>>>> with nogil, parallel:
>>>> ? ? myvariable = 2
>>>> ? ? for i in prange(...):
>>>> ? ? ? ? use myvariable
>>>> ? ? ? ? maybe assign to myvariable
>>>>
>>>> ? ? # myvariable is well-defined here
>>>>
>>>> # myvariable is not well-defined here
>>>>
>>>> If you still desperately want lastprivate behaviour you can simply
>>>> assign myvariable to another variable in the loop body.
>>>
>>> I don't care about lastprivate, I don't think that is an issue, as you
>>> say.
>>>
>>> My problem with this is that it means going into an area where possibly
>>> tricky things are implicit rather than explicit. I also see this as a
>>> rather
>>> special case that will be seldomly used, and implicit behaviour is more
>>> difficult to justify because of that.
>>
>> Indeed, I actually considered if we should support firstprivate at
>> all, as it's really about "being firstprivate and lastprivate".
>> Without any declaration, you can have firstprivate or lastprivate, but
>> not both :) So I agree that supporting such a (probably) uncommon case
>> is better left explicit. On the other hand it seems silly to have
>> support for such a weird case.
>
> Well, I actually need to do the per-thread cache thing I described in the
> CEP in my own codes, so it's not *that* special; it'd be nice to support it.

You need 'old_ell' and 'alpha' after the loop?

> OTOH I *could* work around it by having an array of scalars
>
> cdef int[:] old_ell = int[:numthreads]()
>
> ...
> ? ?if old_ell[threadid()] != ell: ...
>
>
> So I guess, it's at least on the bottom of list of priorities in that CEP.
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Mon Apr 11 12:26:03 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 12:26:03 +0200
Subject: [Cython] Test runner
Message-ID: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>

Can we select tests in the tests directory selectively? I see the -T
or --ticket option, but it doens't seem to find the test tagged with #
ticket: <number here>.

I can select unit tests using python runtests.py
Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
thing for tests in the tests directory. Running the entire suite takes
rather long.

From stefan_ml at behnel.de  Mon Apr 11 12:45:28 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 11 Apr 2011 12:45:28 +0200
Subject: [Cython] Test runner
In-Reply-To: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
Message-ID: <4DA2DBC8.1010007@behnel.de>

mark florisson, 11.04.2011 12:26:
> Can we select tests in the tests directory selectively? I see the -T
> or --ticket option, but it doens't seem to find the test tagged with #
> ticket:<number here>.
>
> I can select unit tests using python runtests.py
> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
> thing for tests in the tests directory. Running the entire suite takes
> rather long.

You can still select them by name using a regex, e.g.

    runtests.py 'run\.empty_builtin_constructors'

Stefan

From markflorisson88 at gmail.com  Mon Apr 11 12:53:49 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 12:53:49 +0200
Subject: [Cython] Test runner
In-Reply-To: <4DA2DBC8.1010007@behnel.de>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
	<4DA2DBC8.1010007@behnel.de>
Message-ID: <BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>

On 11 April 2011 12:45, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 11.04.2011 12:26:
>>
>> Can we select tests in the tests directory selectively? I see the -T
>> or --ticket option, but it doens't seem to find the test tagged with #
>> ticket:<number here>.
>>
>> I can select unit tests using python runtests.py
>> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
>> thing for tests in the tests directory. Running the entire suite takes
>> rather long.
>
> You can still select them by name using a regex, e.g.
>
> ? runtests.py 'run\.empty_builtin_constructors'
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Great, thanks! I'll update the hackerguide wiki.

From markflorisson88 at gmail.com  Mon Apr 11 12:56:13 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 12:56:13 +0200
Subject: [Cython] Test runner
In-Reply-To: <BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
	<4DA2DBC8.1010007@behnel.de>
	<BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>
Message-ID: <BANLkTinf_g5sknckTT+M_ZBrW4-_rXT9Jw@mail.gmail.com>

On 11 April 2011 12:53, mark florisson <markflorisson88 at gmail.com> wrote:
> On 11 April 2011 12:45, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 11.04.2011 12:26:
>>>
>>> Can we select tests in the tests directory selectively? I see the -T
>>> or --ticket option, but it doens't seem to find the test tagged with #
>>> ticket:<number here>.
>>>
>>> I can select unit tests using python runtests.py
>>> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
>>> thing for tests in the tests directory. Running the entire suite takes
>>> rather long.
>>
>> You can still select them by name using a regex, e.g.
>>
>> ? runtests.py 'run\.empty_builtin_constructors'
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> Great, thanks! I'll update the hackerguide wiki.
>
I see now that it is briefly mentioned there, apologies.

From d.s.seljebotn at astro.uio.no  Mon Apr 11 13:02:10 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 11 Apr 2011 13:02:10 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>	<4DA2C59C.8080703@astro.uio.no>	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>	<4DA2D30B.8040209@astro.uio.no>
	<BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>
Message-ID: <4DA2DFB2.1030302@astro.uio.no>

On 04/11/2011 12:14 PM, mark florisson wrote:
> On 11 April 2011 12:08, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/11/2011 11:41 AM, mark florisson wrote:
>>>
>>> On 11 April 2011 11:10, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>   wrote:
>>>>
>>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>>
>>>>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> I've done a pretty major revision to the prange CEP, bringing in a lot
>>>>>> of
>>>>>> the feedback.
>>>>>>
>>>>>> Thread-private variables are now split in two cases:
>>>>>>
>>>>>>   i) The safe cases, which really require very little technical
>>>>>> knowledge
>>>>>> ->
>>>>>> automatically inferred
>>>>>>
>>>>>>   ii) As an advanced feature, unsafe cases that requires some knowledge
>>>>>> of
>>>>>> threading ->      must be explicitly declared
>>>>>>
>>>>>> I think this split simplifies things a great deal.
>>>>>
>>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>>> that need to have firstprivate behaviour inside the with parallel
>>>>> block? Basically in the same way the scratch space is used. The only
>>>>> problem with that is that it won't be lastprivate, so the value will
>>>>> be undefined after the parallel block (but not after the worksharing
>>>>> loop).
>>>>>
>>>>> cdef int myvariable
>>>>>
>>>>> with nogil, parallel:
>>>>>      myvariable = 2
>>>>>      for i in prange(...):
>>>>>          use myvariable
>>>>>          maybe assign to myvariable
>>>>>
>>>>>      # myvariable is well-defined here
>>>>>
>>>>> # myvariable is not well-defined here
>>>>>
>>>>> If you still desperately want lastprivate behaviour you can simply
>>>>> assign myvariable to another variable in the loop body.
>>>>
>>>> I don't care about lastprivate, I don't think that is an issue, as you
>>>> say.
>>>>
>>>> My problem with this is that it means going into an area where possibly
>>>> tricky things are implicit rather than explicit. I also see this as a
>>>> rather
>>>> special case that will be seldomly used, and implicit behaviour is more
>>>> difficult to justify because of that.
>>>
>>> Indeed, I actually considered if we should support firstprivate at
>>> all, as it's really about "being firstprivate and lastprivate".
>>> Without any declaration, you can have firstprivate or lastprivate, but
>>> not both :) So I agree that supporting such a (probably) uncommon case
>>> is better left explicit. On the other hand it seems silly to have
>>> support for such a weird case.
>>
>> Well, I actually need to do the per-thread cache thing I described in the
>> CEP in my own codes, so it's not *that* special; it'd be nice to support it.
>
> You need 'old_ell' and 'alpha' after the loop?


No...but I need the values to not be blanked out at the beginning of 
each loop iteration!

Note that in the CEP, the implicitly thread-local variables are *not 
available* before the first assignment in the loop. That is, code such 
as this is NOT allowed:

cdef double x
...
for i in prange(10):
     print x
     x = f(x)

We raise a compiler error in such cases if we can: The code above is 
violating the contract that the order of execution of loop bodies should 
not matter.

In cases where we can't raise an error (because we didn't bother or 
because it is not possible with a proof), we still initialize the 
variables to invalid values (NaN for double) at the beginning of the 
for-loop just to be sure the contract is satisfied.

This was added to answer Stefan's objection to new types of implicit 
scopes (and I agree with his concern).

Dag Sverre

From d.s.seljebotn at astro.uio.no  Mon Apr 11 13:03:37 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 11 Apr 2011 13:03:37 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA2DFB2.1030302@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>	<4DA2C59C.8080703@astro.uio.no>	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>	<4DA2D30B.8040209@astro.uio.no>
	<BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>
	<4DA2DFB2.1030302@astro.uio.no>
Message-ID: <4DA2E009.2080704@astro.uio.no>

On 04/11/2011 01:02 PM, Dag Sverre Seljebotn wrote:
> On 04/11/2011 12:14 PM, mark florisson wrote:
>> On 11 April 2011 12:08, Dag Sverre
>> Seljebotn<d.s.seljebotn at astro.uio.no> wrote:
>>> On 04/11/2011 11:41 AM, mark florisson wrote:
>>>>
>>>> On 11 April 2011 11:10, Dag Sverre
>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>> wrote:
>>>>>
>>>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>>>
>>>>>> On 5 April 2011 22:29, Dag Sverre
>>>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>>> wrote:
>>>>>>>
>>>>>>> I've done a pretty major revision to the prange CEP, bringing in
>>>>>>> a lot
>>>>>>> of
>>>>>>> the feedback.
>>>>>>>
>>>>>>> Thread-private variables are now split in two cases:
>>>>>>>
>>>>>>> i) The safe cases, which really require very little technical
>>>>>>> knowledge
>>>>>>> ->
>>>>>>> automatically inferred
>>>>>>>
>>>>>>> ii) As an advanced feature, unsafe cases that requires some
>>>>>>> knowledge
>>>>>>> of
>>>>>>> threading -> must be explicitly declared
>>>>>>>
>>>>>>> I think this split simplifies things a great deal.
>>>>>>
>>>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>>>> that need to have firstprivate behaviour inside the with parallel
>>>>>> block? Basically in the same way the scratch space is used. The only
>>>>>> problem with that is that it won't be lastprivate, so the value will
>>>>>> be undefined after the parallel block (but not after the worksharing
>>>>>> loop).
>>>>>>
>>>>>> cdef int myvariable
>>>>>>
>>>>>> with nogil, parallel:
>>>>>> myvariable = 2
>>>>>> for i in prange(...):
>>>>>> use myvariable
>>>>>> maybe assign to myvariable
>>>>>>
>>>>>> # myvariable is well-defined here
>>>>>>
>>>>>> # myvariable is not well-defined here
>>>>>>
>>>>>> If you still desperately want lastprivate behaviour you can simply
>>>>>> assign myvariable to another variable in the loop body.
>>>>>
>>>>> I don't care about lastprivate, I don't think that is an issue, as you
>>>>> say.
>>>>>
>>>>> My problem with this is that it means going into an area where
>>>>> possibly
>>>>> tricky things are implicit rather than explicit. I also see this as a
>>>>> rather
>>>>> special case that will be seldomly used, and implicit behaviour is
>>>>> more
>>>>> difficult to justify because of that.
>>>>
>>>> Indeed, I actually considered if we should support firstprivate at
>>>> all, as it's really about "being firstprivate and lastprivate".
>>>> Without any declaration, you can have firstprivate or lastprivate, but
>>>> not both :) So I agree that supporting such a (probably) uncommon case
>>>> is better left explicit. On the other hand it seems silly to have
>>>> support for such a weird case.
>>>
>>> Well, I actually need to do the per-thread cache thing I described in
>>> the
>>> CEP in my own codes, so it's not *that* special; it'd be nice to
>>> support it.
>>
>> You need 'old_ell' and 'alpha' after the loop?
>
>
> No...but I need the values to not be blanked out at the beginning of
> each loop iteration!

Sorry, I now realize that re-reading your email I may have misunderstood 
you. Anyway, no, I don't need lastprivate at all anywhere.

Dag Sverre

>
> Note that in the CEP, the implicitly thread-local variables are *not
> available* before the first assignment in the loop. That is, code such
> as this is NOT allowed:
>
> cdef double x
> ...
> for i in prange(10):
> print x
> x = f(x)
>
> We raise a compiler error in such cases if we can: The code above is
> violating the contract that the order of execution of loop bodies should
> not matter.
>
> In cases where we can't raise an error (because we didn't bother or
> because it is not possible with a proof), we still initialize the
> variables to invalid values (NaN for double) at the beginning of the
> for-loop just to be sure the contract is satisfied.
>
> This was added to answer Stefan's objection to new types of implicit
> scopes (and I agree with his concern).
>
> Dag Sverre


From markflorisson88 at gmail.com  Mon Apr 11 13:12:27 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 11 Apr 2011 13:12:27 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA2E009.2080704@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>
	<4DA2C59C.8080703@astro.uio.no>
	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>
	<4DA2D30B.8040209@astro.uio.no>
	<BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>
	<4DA2DFB2.1030302@astro.uio.no> <4DA2E009.2080704@astro.uio.no>
Message-ID: <BANLkTinY-Lk7-53Zhi_89zKEo4sUaF3Ucg@mail.gmail.com>

On 11 April 2011 13:03, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/11/2011 01:02 PM, Dag Sverre Seljebotn wrote:
>>
>> On 04/11/2011 12:14 PM, mark florisson wrote:
>>>
>>> On 11 April 2011 12:08, Dag Sverre
>>> Seljebotn<d.s.seljebotn at astro.uio.no> wrote:
>>>>
>>>> On 04/11/2011 11:41 AM, mark florisson wrote:
>>>>>
>>>>> On 11 April 2011 11:10, Dag Sverre
>>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>> wrote:
>>>>>>
>>>>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>>>>
>>>>>>> On 5 April 2011 22:29, Dag Sverre
>>>>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I've done a pretty major revision to the prange CEP, bringing in
>>>>>>>> a lot
>>>>>>>> of
>>>>>>>> the feedback.
>>>>>>>>
>>>>>>>> Thread-private variables are now split in two cases:
>>>>>>>>
>>>>>>>> i) The safe cases, which really require very little technical
>>>>>>>> knowledge
>>>>>>>> ->
>>>>>>>> automatically inferred
>>>>>>>>
>>>>>>>> ii) As an advanced feature, unsafe cases that requires some
>>>>>>>> knowledge
>>>>>>>> of
>>>>>>>> threading -> must be explicitly declared
>>>>>>>>
>>>>>>>> I think this split simplifies things a great deal.
>>>>>>>
>>>>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>>>>> that need to have firstprivate behaviour inside the with parallel
>>>>>>> block? Basically in the same way the scratch space is used. The only
>>>>>>> problem with that is that it won't be lastprivate, so the value will
>>>>>>> be undefined after the parallel block (but not after the worksharing
>>>>>>> loop).
>>>>>>>
>>>>>>> cdef int myvariable
>>>>>>>
>>>>>>> with nogil, parallel:
>>>>>>> myvariable = 2
>>>>>>> for i in prange(...):
>>>>>>> use myvariable
>>>>>>> maybe assign to myvariable
>>>>>>>
>>>>>>> # myvariable is well-defined here
>>>>>>>
>>>>>>> # myvariable is not well-defined here
>>>>>>>
>>>>>>> If you still desperately want lastprivate behaviour you can simply
>>>>>>> assign myvariable to another variable in the loop body.
>>>>>>
>>>>>> I don't care about lastprivate, I don't think that is an issue, as you
>>>>>> say.
>>>>>>
>>>>>> My problem with this is that it means going into an area where
>>>>>> possibly
>>>>>> tricky things are implicit rather than explicit. I also see this as a
>>>>>> rather
>>>>>> special case that will be seldomly used, and implicit behaviour is
>>>>>> more
>>>>>> difficult to justify because of that.
>>>>>
>>>>> Indeed, I actually considered if we should support firstprivate at
>>>>> all, as it's really about "being firstprivate and lastprivate".
>>>>> Without any declaration, you can have firstprivate or lastprivate, but
>>>>> not both :) So I agree that supporting such a (probably) uncommon case
>>>>> is better left explicit. On the other hand it seems silly to have
>>>>> support for such a weird case.
>>>>
>>>> Well, I actually need to do the per-thread cache thing I described in
>>>> the
>>>> CEP in my own codes, so it's not *that* special; it'd be nice to
>>>> support it.
>>>
>>> You need 'old_ell' and 'alpha' after the loop?
>>
>>
>> No...but I need the values to not be blanked out at the beginning of
>> each loop iteration!
>
> Sorry, I now realize that re-reading your email I may have misunderstood
> you. Anyway, no, I don't need lastprivate at all anywhere.

Right, so basically you can rewrite your example by introducing the
parallel block (which doesn't add an indentation level as you're
already using nogil) and assigning to your variables that need to be
firstprivate there. The only thing you miss out on is lastprivate
behaviour. So basically, the question is, do we want explicit syntax
for such a rare case (firstprivate + lastprivate)?

I must say, I found your previous argument of future shared
declarations persuasive enough to introduce explicit syntax.

> Dag Sverre
>
>>
>> Note that in the CEP, the implicitly thread-local variables are *not
>> available* before the first assignment in the loop. That is, code such
>> as this is NOT allowed:
>>
>> cdef double x
>> ...
>> for i in prange(10):
>> print x
>> x = f(x)
>>
>> We raise a compiler error in such cases if we can: The code above is
>> violating the contract that the order of execution of loop bodies should
>> not matter.
>>
>> In cases where we can't raise an error (because we didn't bother or
>> because it is not possible with a proof), we still initialize the
>> variables to invalid values (NaN for double) at the beginning of the
>> for-loop just to be sure the contract is satisfied.
>>
>> This was added to answer Stefan's objection to new types of implicit
>> scopes (and I agree with his concern).
>>
>> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Mon Apr 11 13:26:19 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 11 Apr 2011 13:26:19 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTinY-Lk7-53Zhi_89zKEo4sUaF3Ucg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTi=StJQth0VxvZGcxndpGOVczfhOXA@mail.gmail.com>	<4DA2C59C.8080703@astro.uio.no>	<BANLkTinzN1bm8EmZHpiOsdsyY9dNN3hW2A@mail.gmail.com>	<4DA2D30B.8040209@astro.uio.no>	<BANLkTini2UYLBm_tu_YeFQR9hxSw3aOuhg@mail.gmail.com>	<4DA2DFB2.1030302@astro.uio.no>
	<4DA2E009.2080704@astro.uio.no>
	<BANLkTinY-Lk7-53Zhi_89zKEo4sUaF3Ucg@mail.gmail.com>
Message-ID: <4DA2E55B.5060702@astro.uio.no>

On 04/11/2011 01:12 PM, mark florisson wrote:
> On 11 April 2011 13:03, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/11/2011 01:02 PM, Dag Sverre Seljebotn wrote:
>>>
>>> On 04/11/2011 12:14 PM, mark florisson wrote:
>>>>
>>>> On 11 April 2011 12:08, Dag Sverre
>>>> Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>>>>>
>>>>> On 04/11/2011 11:41 AM, mark florisson wrote:
>>>>>>
>>>>>> On 11 April 2011 11:10, Dag Sverre
>>>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>>>>>
>>>>>>>> On 5 April 2011 22:29, Dag Sverre
>>>>>>>> Seljebotn<d.s.seljebotn at astro.uio.no>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I've done a pretty major revision to the prange CEP, bringing in
>>>>>>>>> a lot
>>>>>>>>> of
>>>>>>>>> the feedback.
>>>>>>>>>
>>>>>>>>> Thread-private variables are now split in two cases:
>>>>>>>>>
>>>>>>>>> i) The safe cases, which really require very little technical
>>>>>>>>> knowledge
>>>>>>>>> ->
>>>>>>>>> automatically inferred
>>>>>>>>>
>>>>>>>>> ii) As an advanced feature, unsafe cases that requires some
>>>>>>>>> knowledge
>>>>>>>>> of
>>>>>>>>> threading ->  must be explicitly declared
>>>>>>>>>
>>>>>>>>> I think this split simplifies things a great deal.
>>>>>>>>
>>>>>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>>>>>> that need to have firstprivate behaviour inside the with parallel
>>>>>>>> block? Basically in the same way the scratch space is used. The only
>>>>>>>> problem with that is that it won't be lastprivate, so the value will
>>>>>>>> be undefined after the parallel block (but not after the worksharing
>>>>>>>> loop).
>>>>>>>>
>>>>>>>> cdef int myvariable
>>>>>>>>
>>>>>>>> with nogil, parallel:
>>>>>>>> myvariable = 2
>>>>>>>> for i in prange(...):
>>>>>>>> use myvariable
>>>>>>>> maybe assign to myvariable
>>>>>>>>
>>>>>>>> # myvariable is well-defined here
>>>>>>>>
>>>>>>>> # myvariable is not well-defined here
>>>>>>>>
>>>>>>>> If you still desperately want lastprivate behaviour you can simply
>>>>>>>> assign myvariable to another variable in the loop body.
>>>>>>>
>>>>>>> I don't care about lastprivate, I don't think that is an issue, as you
>>>>>>> say.
>>>>>>>
>>>>>>> My problem with this is that it means going into an area where
>>>>>>> possibly
>>>>>>> tricky things are implicit rather than explicit. I also see this as a
>>>>>>> rather
>>>>>>> special case that will be seldomly used, and implicit behaviour is
>>>>>>> more
>>>>>>> difficult to justify because of that.
>>>>>>
>>>>>> Indeed, I actually considered if we should support firstprivate at
>>>>>> all, as it's really about "being firstprivate and lastprivate".
>>>>>> Without any declaration, you can have firstprivate or lastprivate, but
>>>>>> not both :) So I agree that supporting such a (probably) uncommon case
>>>>>> is better left explicit. On the other hand it seems silly to have
>>>>>> support for such a weird case.
>>>>>
>>>>> Well, I actually need to do the per-thread cache thing I described in
>>>>> the
>>>>> CEP in my own codes, so it's not *that* special; it'd be nice to
>>>>> support it.
>>>>
>>>> You need 'old_ell' and 'alpha' after the loop?
>>>
>>>
>>> No...but I need the values to not be blanked out at the beginning of
>>> each loop iteration!
>>
>> Sorry, I now realize that re-reading your email I may have misunderstood
>> you. Anyway, no, I don't need lastprivate at all anywhere.
>
> Right, so basically you can rewrite your example by introducing the
> parallel block (which doesn't add an indentation level as you're
> already using nogil) and assigning to your variables that need to be
> firstprivate there. The only thing you miss out on is lastprivate
> behaviour. So basically, the question is, do we want explicit syntax
> for such a rare case (firstprivate + lastprivate)?

OK, we're on the same page here.

> I must say, I found your previous argument of future shared
> declarations persuasive enough to introduce explicit syntax.

OK, lets leave it at this then, we don't have to agree for the same 
reasons :-)

Dag Sverre

From stefan_ml at behnel.de  Mon Apr 11 15:08:46 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 11 Apr 2011 15:08:46 +0200
Subject: [Cython] speed.pypy.org
Message-ID: <4DA2FD5E.5090704@behnel.de>

Hi,

I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython 
running on speed.pypy.org (that's what I wrote "cythonrun" for). If it 
works out well, we may have it up in a couple of days.

I would expect that Cython won't be a big winner in this game, given that 
it will only compile plain untyped Python code. It's also going to fail 
entirely in some of the benchmarks. But I think it's worth having it up 
there, simply as a way for us to see where we are performance-wise and to 
get quick (nightly) feed-back about optimisations we try. The benchmark 
suite is also a nice set of real-world Python code that will allow us to 
find compliance issues.

Stefan

From vitja.makarov at gmail.com  Mon Apr 11 15:14:17 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 11 Apr 2011 17:14:17 +0400
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DA2FD5E.5090704@behnel.de>
References: <4DA2FD5E.5090704@behnel.de>
Message-ID: <BANLkTin15upJ-Hx4xpLRDFkpMdbjLFKd6A@mail.gmail.com>

2011/4/11 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it works
> out well, we may have it up in a couple of days.
>
> I would expect that Cython won't be a big winner in this game, given that it
> will only compile plain untyped Python code. It's also going to fail
> entirely in some of the benchmarks. But I think it's worth having it up
> there, simply as a way for us to see where we are performance-wise and to
> get quick (nightly) feed-back about optimisations we try. The benchmark
> suite is also a nice set of real-world Python code that will allow us to
> find compliance issues.
>
> Stefan


Cool, that would be nice!

-- 
vitja.

From faltet at pytables.org  Mon Apr 11 23:35:55 2011
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 11 Apr 2011 23:35:55 +0200
Subject: [Cython] "Cython's Users Guide"
In-Reply-To: <BANLkTin6JRCYWCgZehKGuDxaFzfzKwNUaQ@mail.gmail.com>
References: <BANLkTin6JRCYWCgZehKGuDxaFzfzKwNUaQ@mail.gmail.com>
Message-ID: <BANLkTimkaphwtkSodLzqBWfzemN=bq=TFg@mail.gmail.com>

2011/4/11 William Stein <wstein at gmail.com>

> Hi,
>
> I'm teaching Cython in my Sage course yet again, and noticed that
> again there are some very confusing aspects of the Cython
> documentation organization, which could probably be improved by a few
> simple changes.
>
>  1. At http://cython.org/ there is a big link in the middle of the
> page labeled "Cython Users Guide" which goes to
> http://docs.cython.org/.   However, http://docs.cython.org/ is *not*
> the users guide -- it is "Cython?s Documentation".     In fact, the
> Users Guide is Chapter 3 of the documentation.
>
>  2. Looking at http://docs.cython.org, we see that Chapter 2 is
> "Tutorials".  But then looking down to Chapter 3 we see that it is
> "Cython Users Guide".  Of course, that's what one is after having just
> clicked a link called "Cython Users Guide".  So we click on "Cython
> Users Guide" again.
>
>  3. We arrive at a page that again has "Tutorial" as Chapter 2.   For
> some reason this makes me feel even more confused.
>
> Recommend changes:
>
>  1. Change the link on the main page from "Cython Users Guide" to
> "Documentation"  or put a direct link into the Users Guide, or have
> two links.
>
>  2. At http://docs.cython.org/ rename the "Cython Users Guide" to
> "Users Guide", since it is obviously the Cython Users Guide at this
> point and "Cython documentation" is in the upper left of the page
> everywhere.
>
>  3. Possibly rename the tutorial in chapter 2 of the users guide to
> something like "First Steps" or "Basic Tutorial" or something.
>

Yeah, that's something that we discussed in the past workshop in Munich
(BTW, many thanks for providing the means for making this happen!).  The
basic idea is to completely remove the Chapter 3 (Cython Users Guide) by
moving its parts to either Chapter 2 (Tutorials), or either to Chapter 4
(Reference Guide).  During the meeting we agreed that the doc repository
should be moved (and has been moved indeed) into the source repo, so that
modifications that affect to code and docs can be put in the same
commit/branch. Also, the wiki has a lot of information that can be better
consolidated and integrated into the User's Guide.

In fact, I already started some job in this direction and created a couple
of pull requests during the workshop (that they have been already
integrated).  I plan to continue this job, but unfortunately I'm pretty busy
lately, so I don't think I can progress a lot in the next weeks, so if
anybody is interested in joining the effort for improving Cython's
documentation, she will be very welcome indeed!

-- 
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110411/a13bbb7c/attachment.html>

From stefan_ml at behnel.de  Tue Apr 12 08:42:29 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Apr 2011 08:42:29 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
Message-ID: <4DA3F455.8070006@behnel.de>

Arthur de Souza Ribeiro, 08.04.2011 02:43:
> 2011/4/7 Robert Bradshaw
>> What I'd like to see is an implementation of a single simple but not
>> entirely trivial (e.g. not math) module, passing regression tests with
>> comprable if not better speed than the current C version (though I
>> think it'd probably make sense to start out with the Python version
>> and optimize that). E.g. http://docs.python.org/library/json.html
>> looks like a good candidate. That should only take 8 hours or so,
>> maybe two days at most, given your background. I'm not expecting
>> anything before the application deadline, but if you could whip
>> something like this out in the next week to point to that would help
>> your application out immensely. In fact, one of the Python
>> foundation's requirements is that students submit a patch before being
>> accepted, and this would knock out that requirement and give you a
>> chance to prove yourself. Create an account on https://github.com and
>> commit your code into a new repository there.
>
> I will start the implementation of json module right now. I created my
> github account and as soon as I have code implemented I will send repository
> link.

Any news on this? We're currently discussing which of the Cython-related 
projects to mentor. It's likely that not all of our projects will get 
accepted, so if you could get us a good initial idea about your work, we'd 
have a stronger incentive to value yours over the others.

Stefan

From arthurdesribeiro at gmail.com  Tue Apr 12 14:59:11 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Tue, 12 Apr 2011 09:59:11 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DA3F455.8070006@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
Message-ID: <BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>

Hi Stefan, yes, I'm working on this, in fact I'm trying to recompile json
module (http://docs.python.org/library/json.html) adding some type
definitions and cython things o get the code faster.

I'm getting in trouble with some things too, I'm going to enumerate here so
that, you could give me some tips about how to solve them.

1 - Compile package modules - json module is inside a package (files:
__init__.py, decoder.py, encoder.py, decoder.py) is there a way to generate
the cython modules just like its get generated by cython?

2 - Because I'm getting in trouble with issue #1, I'm running the tests
manually, I go to %Python-dir%/Lib/tests/json_tests, get the files
corresponding to the tests python make and run manually.

3 - To get the performance of the module, I'm thinking about to use the
timeit function in  the unit tests for the project. I think a good number of
executions would be made and it would be possible to compare each time.

4 - I didn't create the .pxd files, some problems are happening, it tells
methods are not defined, but, they are defined, I will try to investigate
this better

The code is in this repository:
https://github.com/arthursribeiro/JSON-module your feedback would be very
important, so that I could improve my skills to get more and more able to
work sooner in the project.

I think some things implemented in this rewriting process are going to be
useful when doing this with C modules...

Thank you very much.

Best Regards.

[]s

Arthur

2011/4/12 Stefan Behnel <stefan_ml at behnel.de>

> Arthur de Souza Ribeiro, 08.04.2011 02:43:
>
>> 2011/4/7 Robert Bradshaw
>>
>>> What I'd like to see is an implementation of a single simple but not
>>>
>>> entirely trivial (e.g. not math) module, passing regression tests with
>>> comprable if not better speed than the current C version (though I
>>> think it'd probably make sense to start out with the Python version
>>> and optimize that). E.g. http://docs.python.org/library/json.html
>>> looks like a good candidate. That should only take 8 hours or so,
>>> maybe two days at most, given your background. I'm not expecting
>>> anything before the application deadline, but if you could whip
>>> something like this out in the next week to point to that would help
>>> your application out immensely. In fact, one of the Python
>>> foundation's requirements is that students submit a patch before being
>>> accepted, and this would knock out that requirement and give you a
>>> chance to prove yourself. Create an account on https://github.com and
>>> commit your code into a new repository there.
>>>
>>
>> I will start the implementation of json module right now. I created my
>> github account and as soon as I have code implemented I will send
>> repository
>> link.
>>
>
> Any news on this? We're currently discussing which of the Cython-related
> projects to mentor. It's likely that not all of our projects will get
> accepted, so if you could get us a good initial idea about your work, we'd
> have a stronger incentive to value yours over the others.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110412/b4814f1a/attachment.html>

From robertwb at math.washington.edu  Tue Apr 12 15:10:05 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 Apr 2011 06:10:05 -0700
Subject: [Cython] Test runner
In-Reply-To: <BANLkTinf_g5sknckTT+M_ZBrW4-_rXT9Jw@mail.gmail.com>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
	<4DA2DBC8.1010007@behnel.de>
	<BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>
	<BANLkTinf_g5sknckTT+M_ZBrW4-_rXT9Jw@mail.gmail.com>
Message-ID: <BANLkTinCD_HJWgJpS1vi-Vq7X_+g64Xyug@mail.gmail.com>

On Mon, Apr 11, 2011 at 3:56 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 11 April 2011 12:53, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 11 April 2011 12:45, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 11.04.2011 12:26:
>>>>
>>>> Can we select tests in the tests directory selectively? I see the -T
>>>> or --ticket option, but it doens't seem to find the test tagged with #
>>>> ticket:<number here>.
>>>>
>>>> I can select unit tests using python runtests.py
>>>> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
>>>> thing for tests in the tests directory. Running the entire suite takes
>>>> rather long.
>>>
>>> You can still select them by name using a regex, e.g.
>>>
>>> ? runtests.py 'run\.empty_builtin_constructors'
>>>
>>> Stefan
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> Great, thanks! I'll update the hackerguide wiki.
>>
> I see now that it is briefly mentioned there, apologies.

I've added a note there about tags as well, and fixed the -T to look
at the ticket tag. Note that "mode:run" is the default, so you don't
need to explicitly tag mode except for compile/error tests.

- Robert

From robertwb at math.washington.edu  Tue Apr 12 15:25:14 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 Apr 2011 06:25:14 -0700
Subject: [Cython] cython-docs repository
In-Reply-To: <4DA093BF.7050306@creativetrax.com>
References: <4DA047DF.9040000@creativetrax.com>
	<BANLkTikiYQaye8hWyASBCFVmd1chiAMCdg@mail.gmail.com>
	<4DA093BF.7050306@creativetrax.com>
Message-ID: <BANLkTi=CN_QhaAb_RKG3GDB+Jmy9KGaByw@mail.gmail.com>

On Sat, Apr 9, 2011 at 10:13 AM, Jason Grout
<jason-sage at creativetrax.com> wrote:
> On 4/9/11 12:02 PM, Robert Bradshaw wrote:
>>
>> Yep, we did that during the workshop. I thought I had sent out an
>> announcement, but I guess not.
>
> Is there a summary anywhere of the exciting things that happened in the
> workshop?

Not yet, but I'll post as soon as I have a writeup.

> For example, it seems that generators are finally in, if I read
> the commit logs correctly. ?Is that true? ?If so, fantastic!

Yep!

> Any idea of a timeline for that to make it into an official release?

There's still a some fallout and more testing to do, but hopefully it
won't be too long before a release. The biggest step back seems to be
the disabling of inline generators, now they're full-fledged
generators, which is an optimization regression but not a feature
regression.

- Robert

From robertwb at math.washington.edu  Tue Apr 12 15:42:39 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 Apr 2011 06:42:39 -0700
Subject: [Cython] "Cython's Users Guide"
In-Reply-To: <BANLkTimkaphwtkSodLzqBWfzemN=bq=TFg@mail.gmail.com>
References: <BANLkTin6JRCYWCgZehKGuDxaFzfzKwNUaQ@mail.gmail.com>
	<BANLkTimkaphwtkSodLzqBWfzemN=bq=TFg@mail.gmail.com>
Message-ID: <BANLkTimrvFUxdC9FGKQRF3=P6YvFSVMWxQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 2:35 PM, Francesc Alted <faltet at pytables.org> wrote:
> 2011/4/11 William Stein <wstein at gmail.com>
>>
>> Hi,
>>
>> I'm teaching Cython in my Sage course yet again, and noticed that
>> again there are some very confusing aspects of the Cython
>> documentation organization, which could probably be improved by a few
>> simple changes.
>>
>> ?1. At http://cython.org/ there is a big link in the middle of the
>> page labeled "Cython Users Guide" which goes to
>> http://docs.cython.org/. ? However, http://docs.cython.org/ is *not*
>> the users guide -- it is "Cython?s Documentation". ? ? In fact, the
>> Users Guide is Chapter 3 of the documentation.
>>
>> ?2. Looking at http://docs.cython.org, we see that Chapter 2 is
>> "Tutorials". ?But then looking down to Chapter 3 we see that it is
>> "Cython Users Guide". ?Of course, that's what one is after having just
>> clicked a link called "Cython Users Guide". ?So we click on "Cython
>> Users Guide" again.
>>
>> ?3. We arrive at a page that again has "Tutorial" as Chapter 2. ? For
>> some reason this makes me feel even more confused.
>>
>> Recommend changes:
>>
>> ?1. Change the link on the main page from "Cython Users Guide" to
>> "Documentation" ?or put a direct link into the Users Guide, or have
>> two links.
>>
>> ?2. At http://docs.cython.org/ rename the "Cython Users Guide" to
>> "Users Guide", since it is obviously the Cython Users Guide at this
>> point and "Cython documentation" is in the upper left of the page
>> everywhere.
>>
>> ?3. Possibly rename the tutorial in chapter 2 of the users guide to
>> something like "First Steps" or "Basic Tutorial" or something.

Thanks for the suggestions. Done.

> Yeah, that's something that we discussed in the past workshop in Munich
> (BTW, many thanks for providing the means for making this happen!). ?The
> basic idea is to completely remove the Chapter 3 (Cython Users Guide) by
> moving its parts to either Chapter 2 (Tutorials), or either to?Chapter 4
> (Reference Guide). ?During the meeting we agreed that the doc repository
> should be moved (and has been moved indeed) into the source repo, so that
> modifications that affect to code and docs can be put in the same
> commit/branch. Also, the wiki has a lot of information that can be better
> consolidated and integrated into the User's Guide.
> In fact, I already started some job in this direction and created a couple
> of pull requests during the workshop (that they have been already
> integrated). ?I plan to continue this job, but unfortunately I'm pretty busy
> lately, so I don't think I can progress a lot in the next weeks, so if
> anybody is interested in joining the effort for improving Cython's
> documentation, she will be very welcome indeed!

+1, and thanks, Fransesc, for the work you've started in moving us in
this direction.

- Robert

From chris.lasher at gmail.com  Tue Apr 12 16:53:40 2011
From: chris.lasher at gmail.com (Chris Lasher)
Date: Tue, 12 Apr 2011 10:53:40 -0400
Subject: [Cython] Code examples missing in Cython User's Guide
Message-ID: <BANLkTikJEq6L73VtgZofci1ojbBh=Eu+DA@mail.gmail.com>

My apologies for cross-posting this from cython-users, but I realized I
should have sent this bug report to cython-devel.

Several code examples are missing from the User's Guide due to the source
code files being moved or deleted. See for example the Tutorial page on the
User's Guide http://docs.cython.org/src/userguide/tutorial.html The code for
both fib.pyx and primes.pyx (and their setup.py files) is absent from the
document.

I looked at the ReST files and they are trying to source the files from an
"examples" directory. Looking through the git repository, I wasn't able to
locate this directory. Has it been moved or deleted?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110412/26875db2/attachment.html>

From sturla at molden.no  Tue Apr 12 17:33:49 2011
From: sturla at molden.no (Sturla Molden)
Date: Tue, 12 Apr 2011 17:33:49 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
Message-ID: <4DA470DD.3050601@molden.no>

Den 12.04.2011 14:59, skrev Arthur de Souza Ribeiro:
>
> 1 - Compile package modules - json module is inside a package (files: 
> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to 
> generate the cython modules just like its get generated by cython?
>


I'll propose these 10 guidelines:

1. The major concern is to replace the manual use of Python C API with 
Cython.  We aim to improve correctness and readability, not speed.

2. Replacing plain C with Cython for readability is less important, 
sometimes even discourged. If you do, it's ok to leverage on Python 
container types if it makes the code concise and readable, even if it 
will sacrifice some speed.

3. Use exceptions instead of C style error checks: It's better to ask 
forgiveness than permission.

4. Use exceptions correctly. All resourse C allocation belongs in 
__cinit__. All C resource deallocation belongs in __dealloc__. Remember 
that exceptions can cause resource leaks if you don't. Wrap all resource 
allocation in an extension type. Never use functions like malloc or 
fopen directly in your Cython code, except in a __cinit__ method.

5. We should keep as much of the code in Python as we can. Replacing 
Python with Cython for speed is less important. Only the parts that will 
really benefit from static typing should be changed to Cython.

6. Leave the __init__.py file as it is. A Python package is allowed 
contain a mix of Python source files and Cython extension libraries.

7. Be careful to release the GIL whenever appropriate, and never release 
it otherwise. Don't yield the GIL just because you can, it does not come 
for free, even with a single thread.

8. Use the Python and C standard libraries whenever you can.  Don't 
re-invent the wheel. Don't use system dependent APIs when the standard 
libraries declare a common interface. Callbacks to Python are ok.

9. Write code that will work correctly on 32 and 64 bit systems, big- or 
little-endian. Know your C: Py_intptr_t can contain a pointer. 
Py_ssize_t can represent the largest array size allowed. Py_intptr_t and 
Py_ssize_t can have different size. The native array offset can be 
different from Py_ssize_t, for which a common example is AMD64.

10. Don't clutter the namespace, use pxd includes. Short source files 
are preferred to long. Simple is better than complex. Keep the source 
nice and tidy.


Sturla

From arthurdesribeiro at gmail.com  Tue Apr 12 19:39:51 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Tue, 12 Apr 2011 14:39:51 -0300
Subject: [Cython] Code examples missing in Cython User's Guide
In-Reply-To: <BANLkTikJEq6L73VtgZofci1ojbBh=Eu+DA@mail.gmail.com>
References: <BANLkTikJEq6L73VtgZofci1ojbBh=Eu+DA@mail.gmail.com>
Message-ID: <BANLkTinB329Y4=5itnwmpc4+0OeHODYLzg@mail.gmail.com>

Hey Chris, the code for primes and fib examples, are in the directory
'Demos' of the repository...

Best Regards.

[]s

Arthur

2011/4/12 Chris Lasher <chris.lasher at gmail.com>

> My apologies for cross-posting this from cython-users, but I realized I
> should have sent this bug report to cython-devel.
>
> Several code examples are missing from the User's Guide due to the source
> code files being moved or deleted. See for example the Tutorial page on the
> User's Guide http://docs.cython.org/src/userguide/tutorial.html The code
> for both fib.pyx and primes.pyx (and their setup.py files) is absent from
> the document.
>
> I looked at the ReST files and they are trying to source the files from an
> "examples" directory. Looking through the git repository, I wasn't able to
> locate this directory. Has it been moved or deleted?
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110412/e89e6a6d/attachment.html>

From stefan_ml at behnel.de  Tue Apr 12 20:22:05 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Apr 2011 20:22:05 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
Message-ID: <4DA4984D.3020007@behnel.de>

Arthur de Souza Ribeiro, 12.04.2011 14:59:
> Hi Stefan, yes, I'm working on this, in fact I'm trying to recompile json
> module (http://docs.python.org/library/json.html) adding some type
> definitions and cython things o get the code faster.

Cool.


> I'm getting in trouble with some things too, I'm going to enumerate here so
> that, you could give me some tips about how to solve them.
>
> 1 - Compile package modules - json module is inside a package (files:
> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to generate
> the cython modules just like its get generated by cython?

The __init__.py doesn't really look performance critical. It's better to 
leave that modules in plain Python, that improves readability by reducing 
surprises and simplifies reuse by other implementations.

That being said, you can compile each module separately, just use the 
"cython" command line tool for that, or write a little distutils script as in

http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils

Don't worry too much about a build integration for now.


> 2 - Because I'm getting in trouble with issue #1, I'm running the tests
> manually, I go to %Python-dir%/Lib/tests/json_tests, get the files
> corresponding to the tests python make and run manually.

That's fine.


> 3 - To get the performance of the module, I'm thinking about to use the
> timeit function in  the unit tests for the project. I think a good number of
> executions would be made and it would be possible to compare each time.

That's ok for a start, artificial benchmarks are good to test specific 
functionality. However, unit tests tend to be short running with a lot of 
overhead, so later on, you will need to use real code to benchmark the 
modules. I would expect that there are benchmarks for JSON implementations 
around, and you can just generate a large JSON file and run loads and dumps 
on it.


> 4 - I didn't create the .pxd files, some problems are happening, it tells
> methods are not defined, but, they are defined, I will try to investigate
> this better

When reporting usage related problems (preferably on the cython-users 
mailing list), it's best to present the exact error messages and the 
relevant code snippets, so that others can quickly understand what's going 
on and/or reproduce the problem.


> The code is in this repository:
> https://github.com/arthursribeiro/JSON-module your feedback would be very
> important, so that I could improve my skills to get more and more able to
> work sooner in the project.

I'd strongly suggest implementing this in pure Python (.py files instead of 
.pyx files), with externally provided static types for performance. A 
single code base is very advantageous for a large project like CPython, 
much more than the ultimate 5% better performance.


> I think some things implemented in this rewriting process are going to be
> useful when doing this with C modules...

Well, if you can get the existing Python implementation up to mostly 
comparable speed as the C implementation, then there is no need to care 
about the C module anymore. Even if you can get only 90% of a module to run 
at comparable speed, and need to keep 10% in plain C, that's already a huge 
improvement in terms of maintainability.

Stefan

From robertwb at math.washington.edu  Tue Apr 12 22:42:02 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 Apr 2011 13:42:02 -0700
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DA4984D.3020007@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
Message-ID: <BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:22 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Arthur de Souza Ribeiro, 12.04.2011 14:59:
>>
>> Hi Stefan, yes, I'm working on this, in fact I'm trying to recompile json
>> module (http://docs.python.org/library/json.html) adding some type
>> definitions and cython things o get the code faster.
>
> Cool.
>
>
>> I'm getting in trouble with some things too, I'm going to enumerate here
>> so
>> that, you could give me some tips about how to solve them.
>>
>> 1 - Compile package modules - json module is inside a package (files:
>> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to
>> generate
>> the cython modules just like its get generated by cython?
>
> The __init__.py doesn't really look performance critical. It's better to
> leave that modules in plain Python, that improves readability by reducing
> surprises and simplifies reuse by other implementations.
>
> That being said, you can compile each module separately, just use the
> "cython" command line tool for that, or write a little distutils script as
> in
>
> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>
> Don't worry too much about a build integration for now.
>
>
>> 2 - Because I'm getting in trouble with issue #1, I'm running the tests
>> manually, I go to %Python-dir%/Lib/tests/json_tests, get the files
>> corresponding to the tests python make and run manually.
>
> That's fine.
>
>
>> 3 - To get the performance of the module, I'm thinking about to use the
>> timeit function in ?the unit tests for the project. I think a good number
>> of
>> executions would be made and it would be possible to compare each time.
>
> That's ok for a start, artificial benchmarks are good to test specific
> functionality. However, unit tests tend to be short running with a lot of
> overhead, so later on, you will need to use real code to benchmark the
> modules. I would expect that there are benchmarks for JSON implementations
> around, and you can just generate a large JSON file and run loads and dumps
> on it.
>
>
>> 4 - I didn't create the .pxd files, some problems are happening, it tells
>> methods are not defined, but, they are defined, I will try to investigate
>> this better
>
> When reporting usage related problems (preferably on the cython-users
> mailing list), it's best to present the exact error messages and the
> relevant code snippets, so that others can quickly understand what's going
> on and/or reproduce the problem.
>
>
>> The code is in this repository:
>> https://github.com/arthursribeiro/JSON-module your feedback would be very
>> important, so that I could improve my skills to get more and more able to
>> work sooner in the project.
>
> I'd strongly suggest implementing this in pure Python (.py files instead of
> .pyx files), with externally provided static types for performance. A single
> code base is very advantageous for a large project like CPython, much more
> than the ultimate 5% better performance.

While this is advantageous for the final product, it may not be the
easiest to get up and running with.

>> I think some things implemented in this rewriting process are going to be
>> useful when doing this with C modules...
>
> Well, if you can get the existing Python implementation up to mostly
> comparable speed as the C implementation, then there is no need to care
> about the C module anymore. Even if you can get only 90% of a module to run
> at comparable speed, and need to keep 10% in plain C, that's already a huge
> improvement in terms of maintainability.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Tue Apr 12 22:55:01 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 Apr 2011 13:55:01 -0700
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DA470DD.3050601@molden.no>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA470DD.3050601@molden.no>
Message-ID: <BANLkTineCt=ZKDsLxEBWooA1_D3dD+05Ew@mail.gmail.com>

On Tue, Apr 12, 2011 at 8:33 AM, Sturla Molden <sturla at molden.no> wrote:
> Den 12.04.2011 14:59, skrev Arthur de Souza Ribeiro:
>>
>> 1 - Compile package modules - json module is inside a package (files:
>> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to generate
>> the cython modules just like its get generated by cython?
>>
>
>
> I'll propose these 10 guidelines:
>
> 1. The major concern is to replace the manual use of Python C API with
> Cython. ?We aim to improve correctness and readability, not speed.

Speed is a concern, otherwise many of these modules wouldn't have been
written in C in the first place (at least, not the ones with a pure
Python counterpart). Of course some of them are just wrapping C
libraries where speed doesn't matter as much.

> 2. Replacing plain C with Cython for readability is less important,
> sometimes even discourged.

Huh? I'd say this is a big point of the project. Maybe less so than
manual dependance on the C API, but certainly not discouraged.

> If you do, it's ok to leverage on Python
> container types if it makes the code concise and readable, even if it will
> sacrifice some speed.

That's true.

> 3. Use exceptions instead of C style error checks: It's better to ask
> forgiveness than permission.

Yep, this is natural in Cython.

> 4. Use exceptions correctly. All resourse C allocation belongs in __cinit__.
> All C resource deallocation belongs in __dealloc__. Remember that exceptions
> can cause resource leaks if you don't. Wrap all resource allocation in an
> extension type. Never use functions like malloc or fopen directly in your
> Cython code, except in a __cinit__ method.

This can be useful advice, but is not strictly necessary. Try..finally
can fill this need as well.

> 5. We should keep as much of the code in Python as we can. Replacing Python
> with Cython for speed is less important. Only the parts that will really
> benefit from static typing should be changed to Cython.

True. Of course, compiling the (unchanged) pure Python files with
Cython could also yield interesting results, but that's not part of
the project.

> 6. Leave the __init__.py file as it is. A Python package is allowed contain
> a mix of Python source files and Cython extension libraries.
>
> 7. Be careful to release the GIL whenever appropriate, and never release it
> otherwise. Don't yield the GIL just because you can, it does not come for
> free, even with a single thread.
>
> 8. Use the Python and C standard libraries whenever you can. ?Don't
> re-invent the wheel. Don't use system dependent APIs when the standard
> libraries declare a common interface. Callbacks to Python are ok.
>
> 9. Write code that will work correctly on 32 and 64 bit systems, big- or
> little-endian. Know your C: Py_intptr_t can contain a pointer. Py_ssize_t
> can represent the largest array size allowed. Py_intptr_t and Py_ssize_t can
> have different size. The native array offset can be different from
> Py_ssize_t, for which a common example is AMD64.

It's rare to have to do pointer arithmetic in Cython, and rarer still
to have to store the pointer as an integer.

> 10. Don't clutter the namespace, use pxd includes. Short source files are
> preferred to long. Simple is better than complex. Keep the source nice and
> tidy.

Not sure what you mean by "pxd includes," but yes, you should use pxd
files and cimport just as you would in Python to keep things
manageable and modular.

- Robert

From markflorisson88 at gmail.com  Wed Apr 13 13:07:54 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 13 Apr 2011 13:07:54 +0200
Subject: [Cython] Test runner
In-Reply-To: <BANLkTinCD_HJWgJpS1vi-Vq7X_+g64Xyug@mail.gmail.com>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
	<4DA2DBC8.1010007@behnel.de>
	<BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>
	<BANLkTinf_g5sknckTT+M_ZBrW4-_rXT9Jw@mail.gmail.com>
	<BANLkTinCD_HJWgJpS1vi-Vq7X_+g64Xyug@mail.gmail.com>
Message-ID: <BANLkTi=Qr1TiGg4mthf=B0MDHhkY0VqwrQ@mail.gmail.com>

Another, different but related issue: how can we get useful output
from the test runner? e.g. I'm running my test with a
'@cython.test_assert_path_exists("...")' and I get this error output:

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 555, in run
    self.runCompileTest()
  File "runtests.py", line 386, in runCompileTest
    self.test_directory, self.expect_errors, self.annotate)
  File "runtests.py", line 532, in compile
    self.assertEquals(None, unexpected_error)
AssertionError: None != u'9:0: Compiler crash in TreeAssertVisitor'

So I'm seeing a traceback from the test runner (which I'm not really
interested in :), but the actual traceback is not displayed.

Can I also specify special link and compiler flags for a certain test,
like in http://wiki.cython.org/enhancements/distutils_preprocessing ?
Or do I have to export LDFLAGS and CFLAGS in my environment?

From robertwb at math.washington.edu  Wed Apr 13 15:17:32 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 Apr 2011 06:17:32 -0700
Subject: [Cython] Test runner
In-Reply-To: <BANLkTi=Qr1TiGg4mthf=B0MDHhkY0VqwrQ@mail.gmail.com>
References: <BANLkTimp8rHStV6JNvA0+1YCQuvYhrgcUg@mail.gmail.com>
	<4DA2DBC8.1010007@behnel.de>
	<BANLkTincXn3fMW6Mi2eg0jT=gR3w2jOUuw@mail.gmail.com>
	<BANLkTinf_g5sknckTT+M_ZBrW4-_rXT9Jw@mail.gmail.com>
	<BANLkTinCD_HJWgJpS1vi-Vq7X_+g64Xyug@mail.gmail.com>
	<BANLkTi=Qr1TiGg4mthf=B0MDHhkY0VqwrQ@mail.gmail.com>
Message-ID: <BANLkTinZNg4KAPr14uZA0dhPc1w-MXdeow@mail.gmail.com>

On Wed, Apr 13, 2011 at 4:07 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Another, different but related issue: how can we get useful output
> from the test runner? e.g. I'm running my test with a
> '@cython.test_assert_path_exists("...")' and I get this error output:
>
> ======================================================================
> ERROR: runTest (__main__.CythonRunTestCase)
> compiling (c) and running parallel
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 555, in run
> ? ?self.runCompileTest()
> ?File "runtests.py", line 386, in runCompileTest
> ? ?self.test_directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 532, in compile
> ? ?self.assertEquals(None, unexpected_error)
> AssertionError: None != u'9:0: Compiler crash in TreeAssertVisitor'
>
> So I'm seeing a traceback from the test runner (which I'm not really
> interested in :), but the actual traceback is not displayed.

I agree this could be improved, but I'm not sure the best way to do it.

> Can I also specify special link and compiler flags for a certain test,
> like in http://wiki.cython.org/enhancements/distutils_preprocessing ?
> Or do I have to export LDFLAGS and CFLAGS in my environment?

You can't right now, but it would probably be worth adding. I'm not
sure how we would handle missing dependancies in that case though.

- Robert

From stefan_ml at behnel.de  Wed Apr 13 18:58:12 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 13 Apr 2011 18:58:12 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>	<4DA3F455.8070006@behnel.de>	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
Message-ID: <4DA5D624.50609@behnel.de>

Robert Bradshaw, 12.04.2011 22:42:
> On Tue, Apr 12, 2011 at 11:22 AM, Stefan Behnel wrote:
>> Arthur de Souza Ribeiro, 12.04.2011 14:59:
>>> The code is in this repository:
>>> https://github.com/arthursribeiro/JSON-module your feedback would be very
>>> important, so that I could improve my skills to get more and more able to
>>> work sooner in the project.
>>
>> I'd strongly suggest implementing this in pure Python (.py files instead of
>> .pyx files), with externally provided static types for performance. A single
>> code base is very advantageous for a large project like CPython, much more
>> than the ultimate 5% better performance.
>
> While this is advantageous for the final product, it may not be the
> easiest to get up and running with.

Agreed. Arthur, it's fine if you write Cython code in a .pyx file to get 
started. You can just extract the declarations later.

Stefan

From markflorisson88 at gmail.com  Wed Apr 13 21:31:46 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 13 Apr 2011 21:31:46 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4D9B7BA9.2060509@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
Message-ID: <BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>

On 5 April 2011 22:29, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> I've done a pretty major revision to the prange CEP, bringing in a lot of
> the feedback.
>
> Thread-private variables are now split in two cases:
>
> ?i) The safe cases, which really require very little technical knowledge ->
> automatically inferred
>
> ?ii) As an advanced feature, unsafe cases that requires some knowledge of
> threading -> must be explicitly declared
>
> I think this split simplifies things a great deal.
>
> I'm rather excited over this now; this could turn out to be a really
> user-friendly and safe feature that would not only allow us to support
> OpenMP-like threading, but be more convenient to use in a range of common
> cases.
>
> http://wiki.cython.org/enhancements/prange
>
> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

If we want to support cython.parallel.threadsavailable outside of
parallel regions (which does not depend on the schedule used for
worksharing constructs!), then we have to disable dynamic scheduling.
For instance, if OpenMP sees some OpenMP threads are already busy,
then with dynamic scheduling it dynamically establishes how many
threads to use for any parallel region.
So basically, if you put omp_get_num_threads() in a parallel region,
you have a race when you depend on that result in a subsequent
parallel region, because the number of busy OpenMP threads may have
changed.

So basically, to make threadsavailable() work outside parallel
regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
Of course, when OpenMP cannot request the amount of threads desired
(because they are bounded by a configurable thread limit (and the OS
of course)), the behaviour will be implementation defined. So then we
could just put a warning in the docs for that, and users can check for
this in the parallel region using threadsavailable() if it's really
important.

Does that sound like a good idea? And should I update the CEP?

From d.s.seljebotn at astro.uio.no  Wed Apr 13 21:57:07 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 Apr 2011 21:57:07 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
Message-ID: <4DA60013.6060901@astro.uio.no>

On 04/13/2011 09:31 PM, mark florisson wrote:
> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>> the feedback.
>>
>> Thread-private variables are now split in two cases:
>>
>>   i) The safe cases, which really require very little technical knowledge ->
>> automatically inferred
>>
>>   ii) As an advanced feature, unsafe cases that requires some knowledge of
>> threading ->  must be explicitly declared
>>
>> I think this split simplifies things a great deal.
>>
>> I'm rather excited over this now; this could turn out to be a really
>> user-friendly and safe feature that would not only allow us to support
>> OpenMP-like threading, but be more convenient to use in a range of common
>> cases.
>>
>> http://wiki.cython.org/enhancements/prange
>>
>> Dag Sverre
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
> If we want to support cython.parallel.threadsavailable outside of
> parallel regions (which does not depend on the schedule used for
> worksharing constructs!), then we have to disable dynamic scheduling.
> For instance, if OpenMP sees some OpenMP threads are already busy,
> then with dynamic scheduling it dynamically establishes how many
> threads to use for any parallel region.
> So basically, if you put omp_get_num_threads() in a parallel region,
> you have a race when you depend on that result in a subsequent
> parallel region, because the number of busy OpenMP threads may have
> changed.

Ah, I don't know why I thought there wouldn't be a race condition. I 
wonder if the whole threadsavailable() idea should just be ditched and 
that we should think of something else. It's not a very common usecase. 
Starting to disable some forms of scheduling just to, essentially, 
shoehorn in one particular syntax, doesn't seem like the way to go.

Perhaps this calls for support for the critical(?) block then, after 
all. I'm at least +1 on dropping threadsavailable() and instead require 
that you call numthreads() in a critical block:

with parallel:
     with critical:
         # call numthreads() and allocate global buffer
         # calling threadid() not allowed, if we can manage that
     # get buffer slice for each thread

> So basically, to make threadsavailable() work outside parallel
> regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
> Of course, when OpenMP cannot request the amount of threads desired
> (because they are bounded by a configurable thread limit (and the OS
> of course)), the behaviour will be implementation defined. So then we
> could just put a warning in the docs for that, and users can check for
> this in the parallel region using threadsavailable() if it's really
> important.

Do you have any experience with what actually happen with, say, GNU 
OpenMP? I blindly assumed from the specs that it was an error condition 
("flag an error any way you like"), but I guess that may be wrong.

Just curious, I think we can just fall back to OpenMP behaviour; unless 
it terminates the interpreter in an error condition, in which case we 
should look into how expensive it is to check for the condition up front...


Dag Sverre


From markflorisson88 at gmail.com  Wed Apr 13 22:53:03 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 13 Apr 2011 22:53:03 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA60013.6060901@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
Message-ID: <BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>

On 13 April 2011 21:57, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/13/2011 09:31 PM, mark florisson wrote:
>>
>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>> the feedback.
>>>
>>> Thread-private variables are now split in two cases:
>>>
>>> ?i) The safe cases, which really require very little technical knowledge
>>> ->
>>> automatically inferred
>>>
>>> ?ii) As an advanced feature, unsafe cases that requires some knowledge of
>>> threading -> ?must be explicitly declared
>>>
>>> I think this split simplifies things a great deal.
>>>
>>> I'm rather excited over this now; this could turn out to be a really
>>> user-friendly and safe feature that would not only allow us to support
>>> OpenMP-like threading, but be more convenient to use in a range of common
>>> cases.
>>>
>>> http://wiki.cython.org/enhancements/prange
>>>
>>> Dag Sverre
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>> If we want to support cython.parallel.threadsavailable outside of
>> parallel regions (which does not depend on the schedule used for
>> worksharing constructs!), then we have to disable dynamic scheduling.
>> For instance, if OpenMP sees some OpenMP threads are already busy,
>> then with dynamic scheduling it dynamically establishes how many
>> threads to use for any parallel region.
>> So basically, if you put omp_get_num_threads() in a parallel region,
>> you have a race when you depend on that result in a subsequent
>> parallel region, because the number of busy OpenMP threads may have
>> changed.
>
> Ah, I don't know why I thought there wouldn't be a race condition. I wonder
> if the whole threadsavailable() idea should just be ditched and that we
> should think of something else. It's not a very common usecase. Starting to
> disable some forms of scheduling just to, essentially, shoehorn in one
> particular syntax, doesn't seem like the way to go.
>
> Perhaps this calls for support for the critical(?) block then, after all.
> I'm at least +1 on dropping threadsavailable() and instead require that you
> call numthreads() in a critical block:
>
> with parallel:
> ? ?with critical:
> ? ? ? ?# call numthreads() and allocate global buffer
> ? ? ? ?# calling threadid() not allowed, if we can manage that
> ? ?# get buffer slice for each thread

In that case I think you'd want single + a barrier. 'critical' means
that all threads execute the section, but exclusively. I think you
usually want to allocate either a shared worksharing buffer, or a
private thread-local buffer. In the former case you can allocate your
buffer outside any parallel section, in the latter case within the
parallel section. It the latter case the buffer will just not be
available outside of the parallel section.

We can still support any write-back to shared variables that are
explicitly declared later on (supposing we'd also support single and
barriers. Then the code would read as follows

cdef shared(void *) buf
cdef void *localbuf

with nogil, parallel:
    with single:
        buf = malloc(n * numthreads())

    barrier()

    localbuf = buf + n * threadid()
    <actual code here that uses localbuf (or buf if you don't assign to it)>

# localbuf undefined here
# buf is well-defined here

However, I don't believe it's very common to want to use private
buffers after the loop. If you have a buffer in terms of your loop
size, you want it shared, but I can't imagine a case where you want to
examine buffers that were allocated specifically for each thread after
the parallel section. So I'm +1 on dropping threadsavailable outside
parallel sections, but currently -1 on supporting this case, because
we can solve it later on with support for explicitly declared
variables + single + barriers.

>> So basically, to make threadsavailable() work outside parallel
>> regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
>> Of course, when OpenMP cannot request the amount of threads desired
>> (because they are bounded by a configurable thread limit (and the OS
>> of course)), the behaviour will be implementation defined. So then we
>> could just put a warning in the docs for that, and users can check for
>> this in the parallel region using threadsavailable() if it's really
>> important.
>
> Do you have any experience with what actually happen with, say, GNU OpenMP?
> I blindly assumed from the specs that it was an error condition ("flag an
> error any way you like"), but I guess that may be wrong.
>
> Just curious, I think we can just fall back to OpenMP behaviour; unless it
> terminates the interpreter in an error condition, in which case we should
> look into how expensive it is to check for the condition up front...

With libgomp you just get the maximum amount of available threads, up
to the number requested. So this code

  1 #include <stdio.h>
  2 #include <omp.h>
  3
  4 int main(void) {
  5     printf("The thread limit is: %d\n", omp_get_thread_limit());
  6     #pragma omp parallel num_threads(4)
  7     {
  8         #pragma omp single
  9         printf("We have %d threads in the thread team\n",
omp_get_num_threads());
 10     }
 11     return 0;
 12 }

requests 4 threads, but it gets only 2:

[0] [22:28] ~/code/openmp  ? OMP_THREAD_LIMIT=2 ./testomp
The thread limit is: 2
We have 2 threads in the thread team

>
> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Wed Apr 13 23:13:56 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 13 Apr 2011 23:13:56 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
Message-ID: <BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>

On 13 April 2011 22:53, mark florisson <markflorisson88 at gmail.com> wrote:
> On 13 April 2011 21:57, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> On 04/13/2011 09:31 PM, mark florisson wrote:
>>>
>>> On 5 April 2011 22:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>> ?wrote:
>>>>
>>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>>> the feedback.
>>>>
>>>> Thread-private variables are now split in two cases:
>>>>
>>>> ?i) The safe cases, which really require very little technical knowledge
>>>> ->
>>>> automatically inferred
>>>>
>>>> ?ii) As an advanced feature, unsafe cases that requires some knowledge of
>>>> threading -> ?must be explicitly declared
>>>>
>>>> I think this split simplifies things a great deal.
>>>>
>>>> I'm rather excited over this now; this could turn out to be a really
>>>> user-friendly and safe feature that would not only allow us to support
>>>> OpenMP-like threading, but be more convenient to use in a range of common
>>>> cases.
>>>>
>>>> http://wiki.cython.org/enhancements/prange
>>>>
>>>> Dag Sverre
>>>>
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>>
>>>
>>> If we want to support cython.parallel.threadsavailable outside of
>>> parallel regions (which does not depend on the schedule used for
>>> worksharing constructs!), then we have to disable dynamic scheduling.
>>> For instance, if OpenMP sees some OpenMP threads are already busy,
>>> then with dynamic scheduling it dynamically establishes how many
>>> threads to use for any parallel region.
>>> So basically, if you put omp_get_num_threads() in a parallel region,
>>> you have a race when you depend on that result in a subsequent
>>> parallel region, because the number of busy OpenMP threads may have
>>> changed.
>>
>> Ah, I don't know why I thought there wouldn't be a race condition. I wonder
>> if the whole threadsavailable() idea should just be ditched and that we
>> should think of something else. It's not a very common usecase. Starting to
>> disable some forms of scheduling just to, essentially, shoehorn in one
>> particular syntax, doesn't seem like the way to go.
>>
>> Perhaps this calls for support for the critical(?) block then, after all.
>> I'm at least +1 on dropping threadsavailable() and instead require that you
>> call numthreads() in a critical block:
>>
>> with parallel:
>> ? ?with critical:
>> ? ? ? ?# call numthreads() and allocate global buffer
>> ? ? ? ?# calling threadid() not allowed, if we can manage that
>> ? ?# get buffer slice for each thread
>
> In that case I think you'd want single + a barrier. 'critical' means
> that all threads execute the section, but exclusively. I think you
> usually want to allocate either a shared worksharing buffer, or a
> private thread-local buffer. In the former case you can allocate your
> buffer outside any parallel section, in the latter case within the
> parallel section. It the latter case the buffer will just not be
> available outside of the parallel section.
>
> We can still support any write-back to shared variables that are
> explicitly declared later on (supposing we'd also support single and
> barriers. Then the code would read as follows
>
> cdef shared(void *) buf
> cdef void *localbuf
>
> with nogil, parallel:
> ? ?with single:
> ? ? ? ?buf = malloc(n * numthreads())
>
> ? ?barrier()
>
> ? ?localbuf = buf + n * threadid()
> ? ?<actual code here that uses localbuf (or buf if you don't assign to it)>
>
> # localbuf undefined here
> # buf is well-defined here
>
> However, I don't believe it's very common to want to use private
> buffers after the loop. If you have a buffer in terms of your loop
> size, you want it shared, but I can't imagine a case where you want to
> examine buffers that were allocated specifically for each thread after
> the parallel section. So I'm +1 on dropping threadsavailable outside
> parallel sections, but currently -1 on supporting this case, because
> we can solve it later on with support for explicitly declared
> variables + single + barriers.
>
>>> So basically, to make threadsavailable() work outside parallel
>>> regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
>>> Of course, when OpenMP cannot request the amount of threads desired
>>> (because they are bounded by a configurable thread limit (and the OS
>>> of course)), the behaviour will be implementation defined. So then we
>>> could just put a warning in the docs for that, and users can check for
>>> this in the parallel region using threadsavailable() if it's really
>>> important.
>>
>> Do you have any experience with what actually happen with, say, GNU OpenMP?
>> I blindly assumed from the specs that it was an error condition ("flag an
>> error any way you like"), but I guess that may be wrong.
>>
>> Just curious, I think we can just fall back to OpenMP behaviour; unless it
>> terminates the interpreter in an error condition, in which case we should
>> look into how expensive it is to check for the condition up front...
>
> With libgomp you just get the maximum amount of available threads, up
> to the number requested. So this code
>
> ?1 #include <stdio.h>
> ?2 #include <omp.h>
> ?3
> ?4 int main(void) {
> ?5 ? ? printf("The thread limit is: %d\n", omp_get_thread_limit());
> ?6 ? ? #pragma omp parallel num_threads(4)
> ?7 ? ? {
> ?8 ? ? ? ? #pragma omp single
> ?9 ? ? ? ? printf("We have %d threads in the thread team\n",
> omp_get_num_threads());
> ?10 ? ? }
> ?11 ? ? return 0;
> ?12 }
>
> requests 4 threads, but it gets only 2:
>
> [0] [22:28] ~/code/openmp ?? OMP_THREAD_LIMIT=2 ./testomp
> The thread limit is: 2
> We have 2 threads in the thread team
>
>>
>> Dag Sverre
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

Although there is omp_get_max_threads():

"The omp_get_max_threads routine returns an upper bound on the number
of threads that could be used to form a new team if a parallel region
without a num_threads clause were encountered after execution returns
from this routine."

So we could have threadsvailable() evaluate to that if encountered
outside a parallel region. Inside, it would evaluate to
omp_get_num_threads(). At worst, people would over-allocate a bit.

From vitja.makarov at gmail.com  Thu Apr 14 09:27:56 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 14 Apr 2011 11:27:56 +0400
Subject: [Cython] Control flow graph
In-Reply-To: <AANLkTikk0Rnz6oV73HUreY0oC_Ypj-ek+vbi1Vc830vq@mail.gmail.com>
References: <AANLkTin+1cDFwSpvCyROGyByL9vcaALeRfJLYzHqoK5U@mail.gmail.com>
	<AANLkTikw4H6PnucBWRV3=_mQD81+dCqxH+aM+NwBK_Ci@mail.gmail.com>
	<AANLkTimX+_dBNh7q0U1ym7nzohfkn2+rBbpm+gpnNp7h@mail.gmail.com>
	<AANLkTimX_hnOEAspvTghCRRGLyvEOYstRWFodxpyYKwa@mail.gmail.com>
	<4D5A2D6E.9060406@behnel.de>
	<AANLkTin7AC6Pi3d==1TrB458n6AQD51Uyh3DeNTrjEON@mail.gmail.com>
	<AANLkTikM4tO+i1aydG86ev0NA8CETWKaa-GEQgd+==MX@mail.gmail.com>
	<4D641C26.9070705@behnel.de>
	<AANLkTimEi6NBqHiHStnRU5AdwfRWEOT=71Whbqh_MSW7@mail.gmail.com>
	<AANLkTi=tZT15uz+NpLyS+-oMK_2T1t=nzvnJNYPvRgeW@mail.gmail.com>
	<AANLkTik0NcQ7S4=Aqqc_CcTTijmQ6eN+5BVhtf1ZhCWv@mail.gmail.com>
	<AANLkTikk0Rnz6oV73HUreY0oC_Ypj-ek+vbi1Vc830vq@mail.gmail.com>
Message-ID: <BANLkTi=wJo+tpBNRUpaRFhzrCp02YDAtHw@mail.gmail.com>

Can I use cython-generators project on hudson for control-flow tests?
So I'll move cf branch development to my master branch.

-- 
vitja.

From stefan_ml at behnel.de  Thu Apr 14 09:34:18 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 Apr 2011 09:34:18 +0200
Subject: [Cython] Control flow graph
In-Reply-To: <BANLkTi=wJo+tpBNRUpaRFhzrCp02YDAtHw@mail.gmail.com>
References: <AANLkTin+1cDFwSpvCyROGyByL9vcaALeRfJLYzHqoK5U@mail.gmail.com>	<AANLkTikw4H6PnucBWRV3=_mQD81+dCqxH+aM+NwBK_Ci@mail.gmail.com>	<AANLkTimX+_dBNh7q0U1ym7nzohfkn2+rBbpm+gpnNp7h@mail.gmail.com>	<AANLkTimX_hnOEAspvTghCRRGLyvEOYstRWFodxpyYKwa@mail.gmail.com>	<4D5A2D6E.9060406@behnel.de>	<AANLkTin7AC6Pi3d==1TrB458n6AQD51Uyh3DeNTrjEON@mail.gmail.com>	<AANLkTikM4tO+i1aydG86ev0NA8CETWKaa-GEQgd+==MX@mail.gmail.com>	<4D641C26.9070705@behnel.de>	<AANLkTimEi6NBqHiHStnRU5AdwfRWEOT=71Whbqh_MSW7@mail.gmail.com>	<AANLkTi=tZT15uz+NpLyS+-oMK_2T1t=nzvnJNYPvRgeW@mail.gmail.com>	<AANLkTik0NcQ7S4=Aqqc_CcTTijmQ6eN+5BVhtf1ZhCWv@mail.gmail.com>	<AANLkTikk0Rnz6oV73HUreY0oC_Ypj-ek+vbi1Vc830vq@mail.gmail.com>
	<BANLkTi=wJo+tpBNRUpaRFhzrCp02YDAtHw@mail.gmail.com>
Message-ID: <4DA6A37A.8020209@behnel.de>

Vitja Makarov, 14.04.2011 09:27:
> Can I use cython-generators project on hudson for control-flow tests?
> So I'll move cf branch development to my master branch.

Sure. I renamed the tab to "cython-vitek". It's your repo, use it as you 
see fit.

Stefan

From d.s.seljebotn at astro.uio.no  Thu Apr 14 20:29:54 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 Apr 2011 20:29:54 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
Message-ID: <4DA73D22.9050605@astro.uio.no>

On 04/13/2011 11:13 PM, mark florisson wrote:
>
> Although there is omp_get_max_threads():
>
> "The omp_get_max_threads routine returns an upper bound on the number
> of threads that could be used to form a new team if a parallel region
> without a num_threads clause were encountered after execution returns
> from this routine."
>
> So we could have threadsvailable() evaluate to that if encountered
> outside a parallel region. Inside, it would evaluate to
> omp_get_num_threads(). At worst, people would over-allocate a bit.

Well, over-allocating could well mean 1 GB, which could well mean 
getting an unecesarry MemoryError (or, like in my case, if I'm not 
careful to set ulimit, getting a SIGKILL sent to you 2 minutes after the 
fact by the cluster patrol process...)

But even ignoring this, we also have to plan for people misusing the 
feature. If we put it in there, somebody somewhere *will* write code 
like this:

nthreads = threadsavailable()
with parallel:
     for i in prange(nthreads):
         for j in range(100*i, 100*(i+1)): [...]

(Yes, they shouldn't. Yes, they will.)

Combined with a race condition that will only very seldomly trigger, 
this starts to sound like a very bad idea indeed.

So I agree with you that we should just leave it for now, and do 
single/barrier later.

DS

From markflorisson88 at gmail.com  Thu Apr 14 20:39:56 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 14 Apr 2011 20:39:56 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA73D22.9050605@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
Message-ID: <BANLkTi=Pd2mj3NFiM3DUXtd8B4Zgqg1Hyw@mail.gmail.com>

On 14 April 2011 20:29, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/13/2011 11:13 PM, mark florisson wrote:
>>
>> Although there is omp_get_max_threads():
>>
>> "The omp_get_max_threads routine returns an upper bound on the number
>> of threads that could be used to form a new team if a parallel region
>> without a num_threads clause were encountered after execution returns
>> from this routine."
>>
>> So we could have threadsvailable() evaluate to that if encountered
>> outside a parallel region. Inside, it would evaluate to
>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>
> Well, over-allocating could well mean 1 GB, which could well mean getting an
> unecesarry MemoryError (or, like in my case, if I'm not careful to set
> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
> cluster patrol process...)

The upper bound is not "however many threads you think you can start",
but rather "how many threads are considered useful for your machine".
So if you use omp_set_num_threads(), it will return the value you set
there. Otherwise, if you have e.g. a quadcore, it will return 4. The
spec says:

"Note ? The return value of the omp_get_max_threads routine can be
used to dynamically allocate sufficient storage for all threads in the
team formed at the subsequent active parallel region."

So this sounds like a viable option.

> But even ignoring this, we also have to plan for people misusing the
> feature. If we put it in there, somebody somewhere *will* write code like
> this:
>
> nthreads = threadsavailable()
> with parallel:
> ? ?for i in prange(nthreads):
> ? ? ? ?for j in range(100*i, 100*(i+1)): [...]
>
> (Yes, they shouldn't. Yes, they will.)
>
> Combined with a race condition that will only very seldomly trigger, this
> starts to sound like a very bad idea indeed.
>
> So I agree with you that we should just leave it for now, and do
> single/barrier later.
>
> DS
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Thu Apr 14 20:42:13 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 14 Apr 2011 20:42:13 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA73D22.9050605@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
Message-ID: <BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>

On 14 April 2011 20:29, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/13/2011 11:13 PM, mark florisson wrote:
>>
>> Although there is omp_get_max_threads():
>>
>> "The omp_get_max_threads routine returns an upper bound on the number
>> of threads that could be used to form a new team if a parallel region
>> without a num_threads clause were encountered after execution returns
>> from this routine."
>>
>> So we could have threadsvailable() evaluate to that if encountered
>> outside a parallel region. Inside, it would evaluate to
>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>
> Well, over-allocating could well mean 1 GB, which could well mean getting an
> unecesarry MemoryError (or, like in my case, if I'm not careful to set
> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
> cluster patrol process...)
>
> But even ignoring this, we also have to plan for people misusing the
> feature. If we put it in there, somebody somewhere *will* write code like
> this:
>
> nthreads = threadsavailable()
> with parallel:
> ? ?for i in prange(nthreads):
> ? ? ? ?for j in range(100*i, 100*(i+1)): [...]
>
> (Yes, they shouldn't. Yes, they will.)
>
> Combined with a race condition that will only very seldomly trigger, this
> starts to sound like a very bad idea indeed.
>
> So I agree with you that we should just leave it for now, and do
> single/barrier later.

omp_get_max_threads() doesn't have a race, as it returns the upper
bound. So e.g. if between your call and your parallel section less
OpenMP threads become available, then you might get less threads, but
never more.

> DS
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Thu Apr 14 20:55:41 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 Apr 2011 20:55:41 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=Pd2mj3NFiM3DUXtd8B4Zgqg1Hyw@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>	<4DA73D22.9050605@astro.uio.no>
	<BANLkTi=Pd2mj3NFiM3DUXtd8B4Zgqg1Hyw@mail.gmail.com>
Message-ID: <4DA7432D.7080207@astro.uio.no>

On 04/14/2011 08:39 PM, mark florisson wrote:
> On 14 April 2011 20:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>
>>> Although there is omp_get_max_threads():
>>>
>>> "The omp_get_max_threads routine returns an upper bound on the number
>>> of threads that could be used to form a new team if a parallel region
>>> without a num_threads clause were encountered after execution returns
>>> from this routine."
>>>
>>> So we could have threadsvailable() evaluate to that if encountered
>>> outside a parallel region. Inside, it would evaluate to
>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>
>> Well, over-allocating could well mean 1 GB, which could well mean getting an
>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>> cluster patrol process...)
>
> The upper bound is not "however many threads you think you can start",
> but rather "how many threads are considered useful for your machine".
> So if you use omp_set_num_threads(), it will return the value you set
> there. Otherwise, if you have e.g. a quadcore, it will return 4. The
> spec says:
>
> "Note ? The return value of the omp_get_max_threads routine can be
> used to dynamically allocate sufficient storage for all threads in the
> team formed at the subsequent active parallel region."
>
> So this sounds like a viable option.

What would happen here: We have 8 cores. Some code has an OpenMP 
parallel section with maxthreads=2, and inside the section another 
function is called.

That called function uses threadsavailable(), and has a parallel block 
that wants as many threads as it can get.

I don't know the details as well as you do, but my uninformed guess is 
that in this case it'd be quite possible with a race where 
omp_get_max_threads would return 7 in each case, then the first one to 
the parallel would get the 7 threads. The remaining thread then has 
allocated storage for 7 threads but only has 1 thread running.

BTW, I'm not sure what the difference is between the original idea and 
omp_get_max_threads -- in the absence of such races as above, my 
original idea with entering a parallel section (with the same scheduling 
parameters) just to see how many threads we got, would work as well?

DS

From d.s.seljebotn at astro.uio.no  Thu Apr 14 20:58:55 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 Apr 2011 20:58:55 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
Message-ID: <4DA743EF.7080200@astro.uio.no>

On 04/14/2011 08:42 PM, mark florisson wrote:
> On 14 April 2011 20:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>
>>> Although there is omp_get_max_threads():
>>>
>>> "The omp_get_max_threads routine returns an upper bound on the number
>>> of threads that could be used to form a new team if a parallel region
>>> without a num_threads clause were encountered after execution returns
>>> from this routine."
>>>
>>> So we could have threadsvailable() evaluate to that if encountered
>>> outside a parallel region. Inside, it would evaluate to
>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>
>> Well, over-allocating could well mean 1 GB, which could well mean getting an
>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>> cluster patrol process...)
>>
>> But even ignoring this, we also have to plan for people misusing the
>> feature. If we put it in there, somebody somewhere *will* write code like
>> this:
>>
>> nthreads = threadsavailable()
>> with parallel:
>>     for i in prange(nthreads):
>>         for j in range(100*i, 100*(i+1)): [...]
>>
>> (Yes, they shouldn't. Yes, they will.)
>>
>> Combined with a race condition that will only very seldomly trigger, this
>> starts to sound like a very bad idea indeed.
>>
>> So I agree with you that we should just leave it for now, and do
>> single/barrier later.
>
> omp_get_max_threads() doesn't have a race, as it returns the upper
> bound. So e.g. if between your call and your parallel section less
> OpenMP threads become available, then you might get less threads, but
> never more.

Oh, now I'm following you.

Well, my argument was that I think erroring in that direction is pretty 
bad as well.

Also, even if we're not making it available in cython.parallel, we're 
not stopping people from calling omp_get_max_threads directly 
themselves, which should be OK for the people who know enough to do this 
safely...

Dag Sverre

From chris.lasher at gmail.com  Thu Apr 14 21:05:07 2011
From: chris.lasher at gmail.com (Chris Lasher)
Date: Thu, 14 Apr 2011 15:05:07 -0400
Subject: [Cython] Code examples missing in Cython User's Guide
In-Reply-To: <BANLkTinB329Y4=5itnwmpc4+0OeHODYLzg@mail.gmail.com>
References: <BANLkTikJEq6L73VtgZofci1ojbBh=Eu+DA@mail.gmail.com>
	<BANLkTinB329Y4=5itnwmpc4+0OeHODYLzg@mail.gmail.com>
Message-ID: <BANLkTim1nXMnPWoj8RqX-T30uWj1mbFG0g@mail.gmail.com>

Thanks Arthur. I actually found the code examples in a grandparent directory
of the User's Guide documentation source directory. I have submitted a pull
request on GitHub which corrects the User's Guide Tutorial documentation so
that it now includes the code.

https://github.com/cython/cython/pull/24

Could a Cython dev please review and approve this change and then build and
update the docs on the Cython website for the sake of other Cython newbies?

Thanks,
Chris L.

On Tue, Apr 12, 2011 at 1:39 PM, Arthur de Souza Ribeiro <
arthurdesribeiro at gmail.com> wrote:

> Hey Chris, the code for primes and fib examples, are in the directory
> 'Demos' of the repository...
>
> Best Regards.
>
> []s
>
> Arthur
>
> 2011/4/12 Chris Lasher <chris.lasher at gmail.com>
>
>>  My apologies for cross-posting this from cython-users, but I realized I
>> should have sent this bug report to cython-devel.
>>
>> Several code examples are missing from the User's Guide due to the source
>> code files being moved or deleted. See for example the Tutorial page on the
>> User's Guide http://docs.cython.org/src/userguide/tutorial.html The code
>> for both fib.pyx and primes.pyx (and their setup.py files) is absent from
>> the document.
>>
>> I looked at the ReST files and they are trying to source the files from an
>> "examples" directory. Looking through the git repository, I wasn't able to
>> locate this directory. Has it been moved or deleted?
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110414/36f68ac2/attachment.html>

From markflorisson88 at gmail.com  Thu Apr 14 21:08:39 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 14 Apr 2011 21:08:39 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA743EF.7080200@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
Message-ID: <BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>

On 14 April 2011 20:58, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/14/2011 08:42 PM, mark florisson wrote:
>>
>> On 14 April 2011 20:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>>
>>>> Although there is omp_get_max_threads():
>>>>
>>>> "The omp_get_max_threads routine returns an upper bound on the number
>>>> of threads that could be used to form a new team if a parallel region
>>>> without a num_threads clause were encountered after execution returns
>>>> from this routine."
>>>>
>>>> So we could have threadsvailable() evaluate to that if encountered
>>>> outside a parallel region. Inside, it would evaluate to
>>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>>
>>> Well, over-allocating could well mean 1 GB, which could well mean getting
>>> an
>>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>>> cluster patrol process...)
>>>
>>> But even ignoring this, we also have to plan for people misusing the
>>> feature. If we put it in there, somebody somewhere *will* write code like
>>> this:
>>>
>>> nthreads = threadsavailable()
>>> with parallel:
>>> ? ?for i in prange(nthreads):
>>> ? ? ? ?for j in range(100*i, 100*(i+1)): [...]
>>>
>>> (Yes, they shouldn't. Yes, they will.)
>>>
>>> Combined with a race condition that will only very seldomly trigger, this
>>> starts to sound like a very bad idea indeed.
>>>
>>> So I agree with you that we should just leave it for now, and do
>>> single/barrier later.
>>
>> omp_get_max_threads() doesn't have a race, as it returns the upper
>> bound. So e.g. if between your call and your parallel section less
>> OpenMP threads become available, then you might get less threads, but
>> never more.
>
> Oh, now I'm following you.
>
> Well, my argument was that I think erroring in that direction is pretty bad
> as well.
>
> Also, even if we're not making it available in cython.parallel, we're not
> stopping people from calling omp_get_max_threads directly themselves, which
> should be OK for the people who know enough to do this safely...

True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any
event, we could just put a warning in the docs stating that using
threadsavailable outside parallel sections returns an upper bound on
the actual number of threads in a subsequent parallel section.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Thu Apr 14 21:37:16 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 Apr 2011 21:37:16 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>	<4DA73D22.9050605@astro.uio.no>	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
Message-ID: <4DA74CEC.2000609@astro.uio.no>

On 04/14/2011 09:08 PM, mark florisson wrote:
> On 14 April 2011 20:58, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>> On 04/14/2011 08:42 PM, mark florisson wrote:
>>>
>>> On 14 April 2011 20:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>   wrote:
>>>>
>>>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>>>
>>>>> Although there is omp_get_max_threads():
>>>>>
>>>>> "The omp_get_max_threads routine returns an upper bound on the number
>>>>> of threads that could be used to form a new team if a parallel region
>>>>> without a num_threads clause were encountered after execution returns
>>>>> from this routine."
>>>>>
>>>>> So we could have threadsvailable() evaluate to that if encountered
>>>>> outside a parallel region. Inside, it would evaluate to
>>>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>>>
>>>> Well, over-allocating could well mean 1 GB, which could well mean getting
>>>> an
>>>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>>>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>>>> cluster patrol process...)
>>>>
>>>> But even ignoring this, we also have to plan for people misusing the
>>>> feature. If we put it in there, somebody somewhere *will* write code like
>>>> this:
>>>>
>>>> nthreads = threadsavailable()
>>>> with parallel:
>>>>     for i in prange(nthreads):
>>>>         for j in range(100*i, 100*(i+1)): [...]
>>>>
>>>> (Yes, they shouldn't. Yes, they will.)
>>>>
>>>> Combined with a race condition that will only very seldomly trigger, this
>>>> starts to sound like a very bad idea indeed.
>>>>
>>>> So I agree with you that we should just leave it for now, and do
>>>> single/barrier later.
>>>
>>> omp_get_max_threads() doesn't have a race, as it returns the upper
>>> bound. So e.g. if between your call and your parallel section less
>>> OpenMP threads become available, then you might get less threads, but
>>> never more.
>>
>> Oh, now I'm following you.
>>
>> Well, my argument was that I think erroring in that direction is pretty bad
>> as well.
>>
>> Also, even if we're not making it available in cython.parallel, we're not
>> stopping people from calling omp_get_max_threads directly themselves, which
>> should be OK for the people who know enough to do this safely...
>
> True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any
> event, we could just put a warning in the docs stating that using
> threadsavailable outside parallel sections returns an upper bound on
> the actual number of threads in a subsequent parallel section.

I don't think outside or within makes a difference -- what about nested 
parallel sections? At least my intention in the CEP was that 
threadsavailable was always for the next section (so often it would be 1 
after entering the section).

Perhaps just calling it "maxthreads" instead solves the issue.

(Still, I favour just dropping threadsavailable/maxthreads for the time 
being. It is much simpler to add something later, when we've had some 
time to use it and reflect about it, than to remove something that 
shouldn't have been added.)

Dag Sverre

From markflorisson88 at gmail.com  Thu Apr 14 21:58:52 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 14 Apr 2011 21:58:52 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA74CEC.2000609@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
Message-ID: <BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>

On 14 April 2011 21:37, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/14/2011 09:08 PM, mark florisson wrote:
>>
>> On 14 April 2011 20:58, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>> ?wrote:
>>>
>>> On 04/14/2011 08:42 PM, mark florisson wrote:
>>>>
>>>> On 14 April 2011 20:29, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>> ?wrote:
>>>>>
>>>>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>>>>
>>>>>> Although there is omp_get_max_threads():
>>>>>>
>>>>>> "The omp_get_max_threads routine returns an upper bound on the number
>>>>>> of threads that could be used to form a new team if a parallel region
>>>>>> without a num_threads clause were encountered after execution returns
>>>>>> from this routine."
>>>>>>
>>>>>> So we could have threadsvailable() evaluate to that if encountered
>>>>>> outside a parallel region. Inside, it would evaluate to
>>>>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>>>>
>>>>> Well, over-allocating could well mean 1 GB, which could well mean
>>>>> getting
>>>>> an
>>>>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>>>>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>>>>> cluster patrol process...)
>>>>>
>>>>> But even ignoring this, we also have to plan for people misusing the
>>>>> feature. If we put it in there, somebody somewhere *will* write code
>>>>> like
>>>>> this:
>>>>>
>>>>> nthreads = threadsavailable()
>>>>> with parallel:
>>>>> ? ?for i in prange(nthreads):
>>>>> ? ? ? ?for j in range(100*i, 100*(i+1)): [...]
>>>>>
>>>>> (Yes, they shouldn't. Yes, they will.)
>>>>>
>>>>> Combined with a race condition that will only very seldomly trigger,
>>>>> this
>>>>> starts to sound like a very bad idea indeed.
>>>>>
>>>>> So I agree with you that we should just leave it for now, and do
>>>>> single/barrier later.
>>>>
>>>> omp_get_max_threads() doesn't have a race, as it returns the upper
>>>> bound. So e.g. if between your call and your parallel section less
>>>> OpenMP threads become available, then you might get less threads, but
>>>> never more.
>>>
>>> Oh, now I'm following you.
>>>
>>> Well, my argument was that I think erroring in that direction is pretty
>>> bad
>>> as well.
>>>
>>> Also, even if we're not making it available in cython.parallel, we're not
>>> stopping people from calling omp_get_max_threads directly themselves,
>>> which
>>> should be OK for the people who know enough to do this safely...
>>
>> True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any
>> event, we could just put a warning in the docs stating that using
>> threadsavailable outside parallel sections returns an upper bound on
>> the actual number of threads in a subsequent parallel section.
>
> I don't think outside or within makes a difference -- what about nested
> parallel sections? At least my intention in the CEP was that
> threadsavailable was always for the next section (so often it would be 1
> after entering the section).
>
> Perhaps just calling it "maxthreads" instead solves the issue.
>
> (Still, I favour just dropping threadsavailable/maxthreads for the time
> being. It is much simpler to add something later, when we've had some time
> to use it and reflect about it, than to remove something that shouldn't have
> been added.)
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Definitely true, I'll disable it for now.

From arthurdesribeiro at gmail.com  Fri Apr 15 04:31:09 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Thu, 14 Apr 2011 23:31:09 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DA5D624.50609@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
Message-ID: <BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>

I've created the .pyx files and it passed in all python tests.

To test them, as I said, I copied the .py test files to my project
directory, generated the .so files, import them instead of python modules
and run. I run every test file and it passed in all of them. To run the
tests, run the file 'run-tests.sh'

I used just .pyx in this module, should I reimplement it using pxd with the
normal .py?

I'll still see the performance, and tell here... The code is updated in
repository...

Best Regards

[]s

Arthur

2011/4/13 Stefan Behnel <stefan_ml at behnel.de>

> Robert Bradshaw, 12.04.2011 22:42:
>
>> On Tue, Apr 12, 2011 at 11:22 AM, Stefan Behnel wrote:
>>
>>> Arthur de Souza Ribeiro, 12.04.2011 14:59:
>>>
>>>> The code is in this repository:
>>>>
>>>> https://github.com/arthursribeiro/JSON-module your feedback would be
>>>> very
>>>> important, so that I could improve my skills to get more and more able
>>>> to
>>>> work sooner in the project.
>>>>
>>>
>>> I'd strongly suggest implementing this in pure Python (.py files instead
>>> of
>>> .pyx files), with externally provided static types for performance. A
>>> single
>>> code base is very advantageous for a large project like CPython, much
>>> more
>>> than the ultimate 5% better performance.
>>>
>>
>> While this is advantageous for the final product, it may not be the
>> easiest to get up and running with.
>>
>
> Agreed. Arthur, it's fine if you write Cython code in a .pyx file to get
> started. You can just extract the declarations later.
>
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110414/94dd8962/attachment.html>

From stefan_ml at behnel.de  Fri Apr 15 08:31:47 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Apr 2011 08:31:47 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
Message-ID: <4DA7E653.4060801@behnel.de>

[please avoid top-posting]

Arthur de Souza Ribeiro, 15.04.2011 04:31:
> I've created the .pyx files and it passed in all python tests.

Fine.

As far as I can see, you only added static types in some places. Did you 
test if they are actually required (maybe using "cython -a")? Some of them 
look rather counterproductive and should lead to a major slow-down. I added 
comments to your initial commit.

Note that it's not obvious from your initial commit what you actually 
changed. It would have been better to import the original file first, 
rename it to .pyx, and then commit your changes.

It appears that you accidentally added your .c and .so files to your repo.

https://github.com/arthursribeiro/JSON-module


> To test them, as I said, I copied the .py test files to my project
> directory, generated the .so files, import them instead of python modules
> and run. I run every test file and it passed in all of them. To run the
> tests, run the file 'run-tests.sh'
>
> I used just .pyx in this module, should I reimplement it using pxd with the
> normal .py?

Not at this point. I think it's more important to get some performance 
numbers to see how your module behaves compared to the C accelerator module 
(_json.c). I think the best approach to this project would actually be to 
start with profiling the Python implementation to see where performance 
problems occur (or to look through _json.c to see what the CPython 
developers considered performance critical), and then put the focus on 
trying to speed up only those parts of the Python implementation, by adding 
static types and potentially even rewriting them in a way that Cython can 
optimise them better.

Stefan

From yury at shurup.com  Fri Apr 15 16:20:09 2011
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 15 Apr 2011 16:20:09 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
Message-ID: <1302877209.6733.61.camel@mypride>

Hi folks!

I have just ran into buffer protocol incompatibility problem with
numpy-1.5.1, which lead me to discover the following ticket (discussed
back in December 2010 on this list):

http://trac.cython.org/cython_trac/ticket/630

In despair, I was about to try to see if there is anything I can do to
fix it myself. To this end I cloned your git repository and set up my
Python environment to use the latest bleeding edge version from there.

To my surprise I discovered that my code started working and I don't
have the buffer interface problem that I was facing before anymore.

So my suggestion would be to maybe test it more thoroughly and if it is
indeed the case, close the ticket. I tried to subscribe to it or leave a
comment, but I need an account which I can't register on my own.

Still, I have the following question: which risks am I running by using
the bleeding edge version and is there any chance this could inflict
cancer on me?

Do you have plans to make a new point release which includes this very
important improvement anytime soon?

Thanks!

P.S. I've met Stefan Behnel at FACETS CodeJam the year before last. Back
then I didn't realize how much you rock guys and how I will need Cython
in the future for my research...
 
-- 
Sincerely yours,
Yury V. Zaytsev


From yury at shurup.com  Fri Apr 15 16:48:47 2011
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 15 Apr 2011 16:48:47 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <1302877209.6733.61.camel@mypride>
References: <1302877209.6733.61.camel@mypride>
Message-ID: <1302878927.6733.65.camel@mypride>

On Fri, 2011-04-15 at 16:20 +0200, Yury V. Zaytsev wrote:

> To my surprise I discovered that my code started working and I don't
> have the buffer interface problem that I was facing before anymore.

I am under impression that the culprit was this commit:

Commit SHA1: dd3da6c283a0750c9c6514991be719ac064e79b4
Commit message: PEP 3118: fix for NULL Py_buffer arg
Committer: Lisandro Dalcin <dalcinl at gmail.com>  2011-04-08 00:25:45
 
-- 
Sincerely yours,
Yury V. Zaytsev


From dalcinl at gmail.com  Fri Apr 15 18:45:31 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 Apr 2011 13:45:31 -0300
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <1302877209.6733.61.camel@mypride>
References: <1302877209.6733.61.camel@mypride>
Message-ID: <BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>

On 15 April 2011 11:20, Yury V. Zaytsev <yury at shurup.com> wrote:
> Hi folks!
>
> I have just ran into buffer protocol incompatibility problem with
> numpy-1.5.1, which lead me to discover the following ticket (discussed
> back in December 2010 on this list):
>
> http://trac.cython.org/cython_trac/ticket/630
>
> In despair, I was about to try to see if there is anything I can do to
> fix it myself. To this end I cloned your git repository and set up my
> Python environment to use the latest bleeding edge version from there.
>
> To my surprise I discovered that my code started working and I don't
> have the buffer interface problem that I was facing before anymore.
>
> So my suggestion would be to maybe test it more thoroughly and if it is
> indeed the case, close the ticket. I tried to subscribe to it or leave a
> comment, but I need an account which I can't register on my own.
>

I'm opposed to close the ticket. Cython cannot currently parse format
strings according to the full spec. It worked for you just because of
some quick fixes I've pushed for simple cases.

> Still, I have the following question: which risks am I running by using
> the bleeding edge version and is there any chance this could inflict
> cancer on me?
>

Well, usually the bleeding edge is more featured and bug-free than
latest release. However, right now, generator support was merged and
you could have some performance regressions. But I'm using the
bleeding edge for every day development work of mpi4py/petsc4py.

>
> Do you have plans to make a new point release which includes this very
> important improvement anytime soon?
>

Not sure, we still have some (performance) regressions to fix, all of
them related to the new generators support.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Fri Apr 15 18:52:29 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 Apr 2011 13:52:29 -0300
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <1302878927.6733.65.camel@mypride>
References: <1302877209.6733.61.camel@mypride>
	<1302878927.6733.65.camel@mypride>
Message-ID: <BANLkTinfOvz_=jXZR9yd-w2dS_zJVnHuHA@mail.gmail.com>

On 15 April 2011 11:48, Yury V. Zaytsev <yury at shurup.com> wrote:
> On Fri, 2011-04-15 at 16:20 +0200, Yury V. Zaytsev wrote:
>
>> To my surprise I discovered that my code started working and I don't
>> have the buffer interface problem that I was facing before anymore.
>
> I am under impression that the culprit was this commit:
>
> Commit SHA1: dd3da6c283a0750c9c6514991be719ac064e79b4
> Commit message: PEP 3118: fix for NULL Py_buffer arg
> Committer: Lisandro Dalcin <dalcinl at gmail.com> ?2011-04-08 00:25:45
>

This commit should be easy to cherry-pick and backport to latest
release. However, there is a potential risk that user code
implementing __getbuffer__ in extension classes do not make the check
"if info == NULL: return", and then you get a SEGFAULT. AFAIK, all
this business with Py_buffer pointer being NULL was added because of
some obscure bug in Python==3.0.x, but I would expect people out there
to be using Python >= 3.1.x, so not a big deal.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From d.s.seljebotn at astro.uio.no  Fri Apr 15 19:57:21 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 Apr 2011 19:57:21 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
References: <1302877209.6733.61.camel@mypride>
	<BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
Message-ID: <4DA88701.5050506@astro.uio.no>

On 04/15/2011 06:45 PM, Lisandro Dalcin wrote:
> On 15 April 2011 11:20, Yury V. Zaytsev<yury at shurup.com>  wrote:
>> Hi folks!
>>
>> I have just ran into buffer protocol incompatibility problem with
>> numpy-1.5.1, which lead me to discover the following ticket (discussed
>> back in December 2010 on this list):
>>
>> http://trac.cython.org/cython_trac/ticket/630
>>
>> In despair, I was about to try to see if there is anything I can do to
>> fix it myself. To this end I cloned your git repository and set up my
>> Python environment to use the latest bleeding edge version from there.
>>
>> To my surprise I discovered that my code started working and I don't
>> have the buffer interface problem that I was facing before anymore.
>>
>> So my suggestion would be to maybe test it more thoroughly and if it is
>> indeed the case, close the ticket. I tried to subscribe to it or leave a
>> comment, but I need an account which I can't register on my own.
>>
>
> I'm opposed to close the ticket. Cython cannot currently parse format
> strings according to the full spec. It worked for you just because of
> some quick fixes I've pushed for simple cases.

Pauli Virtanen fixed this and there's a pull request here:

https://github.com/cython/cython/pull/17

I'll get to it in a couple of days if nobody beats me to it.

BTW, did anyone figure out how to get emailed on pull requests? (Yes, I 
checked off all the "send me email" boxes etc., we were talking about 
this during the workshop -- it seems that org admins can't subscribe...)

DS

From d.s.seljebotn at astro.uio.no  Fri Apr 15 20:00:32 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 Apr 2011 20:00:32 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <4DA88701.5050506@astro.uio.no>
References: <1302877209.6733.61.camel@mypride>	<BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
	<4DA88701.5050506@astro.uio.no>
Message-ID: <4DA887C0.3000505@astro.uio.no>

On 04/15/2011 07:57 PM, Dag Sverre Seljebotn wrote:
> On 04/15/2011 06:45 PM, Lisandro Dalcin wrote:
>> On 15 April 2011 11:20, Yury V. Zaytsev<yury at shurup.com> wrote:
>>> Hi folks!
>>>
>>> I have just ran into buffer protocol incompatibility problem with
>>> numpy-1.5.1, which lead me to discover the following ticket (discussed
>>> back in December 2010 on this list):
>>>
>>> http://trac.cython.org/cython_trac/ticket/630
>>>
>>> In despair, I was about to try to see if there is anything I can do to
>>> fix it myself. To this end I cloned your git repository and set up my
>>> Python environment to use the latest bleeding edge version from there.
>>>
>>> To my surprise I discovered that my code started working and I don't
>>> have the buffer interface problem that I was facing before anymore.
>>>
>>> So my suggestion would be to maybe test it more thoroughly and if it is
>>> indeed the case, close the ticket. I tried to subscribe to it or leave a
>>> comment, but I need an account which I can't register on my own.
>>>
>>
>> I'm opposed to close the ticket. Cython cannot currently parse format
>> strings according to the full spec. It worked for you just because of
>> some quick fixes I've pushed for simple cases.
>
> Pauli Virtanen fixed this and there's a pull request here:
>
> https://github.com/cython/cython/pull/17
>
> I'll get to it in a couple of days if nobody beats me to it.
>
> BTW, did anyone figure out how to get emailed on pull requests? (Yes, I
> checked off all the "send me email" boxes etc., we were talking about
> this during the workshop -- it seems that org admins can't subscribe...)

And BTW, the Cython test suite exposed a bug in NumPy as well -- you can 
see it in the test case comments. So there may be NumPy builds (and 
releases?) out there that fail the Cython test suite because of a bug in 
NumPy.

It only affect unpacked structs though; I believe you're good with 
packed structs (and few use NumPy with unpacked structs).

DS

From yury at shurup.com  Fri Apr 15 20:21:38 2011
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 15 Apr 2011 20:21:38 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <4DA88701.5050506@astro.uio.no>
References: <1302877209.6733.61.camel@mypride>
	<BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
	<4DA88701.5050506@astro.uio.no>
Message-ID: <1302891698.6733.72.camel@mypride>

On Fri, 2011-04-15 at 19:57 +0200, Dag Sverre Seljebotn wrote:

> BTW, did anyone figure out how to get emailed on pull requests? (Yes, I 
> checked off all the "send me email" boxes etc., we were talking about 
> this during the workshop -- it seems that org admins can't subscribe...)

That's strange, I'm getting notifications, when I get pull requests for
the projects I have in my organization. I don't remember ticking any
extra boxes other than in the Account Settings -> Notification Center
though.

Maybe you are not actually watching the main repository? I would try to
set the watch flag and see if it works...

Thanks for you your hard work on Cython by the way!
 
-- 
Sincerely yours,
Yury V. Zaytsev


From yury at shurup.com  Fri Apr 15 20:23:15 2011
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 15 Apr 2011 20:23:15 +0200
Subject: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
In-Reply-To: <BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
References: <1302877209.6733.61.camel@mypride>
	<BANLkTinuPFzuL=rK+ZV9fuLkcd_NF7o98w@mail.gmail.com>
Message-ID: <1302891795.6733.75.camel@mypride>

On Fri, 2011-04-15 at 13:45 -0300, Lisandro Dalcin wrote:

> I'm opposed to close the ticket. Cython cannot currently parse format
> strings according to the full spec. It worked for you just because of
> some quick fixes I've pushed for simple cases.

I don't insist on that, especially now that I didn't have the full
picture. Maybe if you have access to the track just add a note that most
of the problems have been solved in the latest git master and it will be
fixed for good before the next release?

Thanks, Lisandro!
 
-- 
Sincerely yours,
Yury V. Zaytsev


From stefan_ml at behnel.de  Fri Apr 15 22:20:51 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Apr 2011 22:20:51 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DA2FD5E.5090704@behnel.de>
References: <4DA2FD5E.5090704@behnel.de>
Message-ID: <4DA8A8A3.6080600@behnel.de>

Stefan Behnel, 11.04.2011 15:08:
> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
> works out well, we may have it up in a couple of days.

... or maybe not. It may take a little longer due to lack of time on his side.


> I would expect that Cython won't be a big winner in this game, given that
> it will only compile plain untyped Python code. It's also going to fail
> entirely in some of the benchmarks. But I think it's worth having it up
> there, simply as a way for us to see where we are performance-wise and to
> get quick (nightly) feed-back about optimisations we try. The benchmark
> suite is also a nice set of real-world Python code that will allow us to
> find compliance issues.

Ok, here's what I have so far. I fixed a couple of bugs in Cython and got 
at least some of the benchmarks running. Note that they are actually simple 
ones, only a single module. Basically all complex benchmarks fail due to 
known bugs, such as Cython def functions not accepting attribute 
assignments (e.g. on wrapping). There's also a problem with code that uses 
platform specific names conditionally, such as WindowsError when running on 
Windows. Cython complains about non-builtin names here. I'm considering to 
turn that into a visible warning instead of an error, so that the name 
would instead be looked up dynamically to let the code fail at runtime 
*iff* it reaches the name lookup.

Anyway, here are the numbers. I got them with "auto_cpdef" enabled, 
although that doesn't even seem to make that a big difference. The baseline 
is a self-compiled Python 2.7.1+ (about a month old).

Stefan


### ai ###
Min: 0.402407 -> 0.362190: 1.1110x faster
Avg: 0.408784 -> 0.366898: 1.1142x faster
Significant (t=10.017195, a=0.95)
Stddev: 0.00824 -> 0.00442: 1.8668x smaller


### chaos ###  (with a bug fixed in the benchmark itself)
Min: 0.393362 -> 0.231932: 1.6960x faster
Avg: 0.401941 -> 0.234089: 1.7170x faster
Significant (t=36.128709, a=0.95)
Stddev: 0.01004 -> 0.00267: 3.7538x smaller


### crypto_pyaes ###
Min: 2.629560 -> 1.276433: 2.0601x faster
Avg: 2.639409 -> 1.277742: 2.0657x faster
Significant (t=368.652396, a=0.95)
Stddev: 0.00812 -> 0.00153: 5.3215x smaller


### fannkuch ###
Min: 1.512630 -> 0.853309: 1.7727x faster
Avg: 1.522860 -> 0.860237: 1.7703x faster
Significant (t=118.573908, a=0.95)
Stddev: 0.00880 -> 0.00887: 1.0073x larger


### float ###
Min: 0.452620 -> 0.343341: 1.3183x faster
Avg: 0.475137 -> 0.349356: 1.3600x faster
Significant (t=9.575876, a=0.95)
Stddev: 0.02838 -> 0.00757: 3.7489x smaller


### go ###
Min: 0.758998 -> 0.491929: 1.5429x faster
Avg: 0.764110 -> 0.496518: 1.5389x faster
Significant (t=90.848407, a=0.95)
Stddev: 0.00400 -> 0.00523: 1.3096x larger


### nbody_modified ###
Min: 0.399168 -> 0.197931: 2.0167x faster
Avg: 0.401379 -> 0.203112: 1.9762x faster
Significant (t=42.377829, a=0.95)
Stddev: 0.00293 -> 0.01004: 3.4337x larger


### raytracesimple ###   (module renamed from "raytrace-simple")
Min: 2.016425 -> 1.182970: 1.7045x faster
Avg: 2.030030 -> 1.192164: 1.7028x faster
Significant (t=78.219481, a=0.95)
Stddev: 0.02184 -> 0.00983: 2.2211x smaller


### richards ###
Min: 0.286723 -> 0.162430: 1.7652x faster
Avg: 0.289933 -> 0.165193: 1.7551x faster
Significant (t=52.898468, a=0.95)
Stddev: 0.00392 -> 0.00352: 1.1127x smaller

From robertwb at math.washington.edu  Sat Apr 16 08:53:46 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 Apr 2011 23:53:46 -0700
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DA8A8A3.6080600@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
Message-ID: <BANLkTikqutv+0WOw7U=uyO9ndM_aqs_z6w@mail.gmail.com>

On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 11.04.2011 15:08:
>>
>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>> works out well, we may have it up in a couple of days.
>
> ... or maybe not. It may take a little longer due to lack of time on his
> side.
>
>
>> I would expect that Cython won't be a big winner in this game, given that
>> it will only compile plain untyped Python code. It's also going to fail
>> entirely in some of the benchmarks. But I think it's worth having it up
>> there, simply as a way for us to see where we are performance-wise and to
>> get quick (nightly) feed-back about optimisations we try. The benchmark
>> suite is also a nice set of real-world Python code that will allow us to
>> find compliance issues.
>
> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got at
> least some of the benchmarks running. Note that they are actually simple
> ones, only a single module. Basically all complex benchmarks fail due to
> known bugs, such as Cython def functions not accepting attribute assignments
> (e.g. on wrapping). There's also a problem with code that uses platform
> specific names conditionally, such as WindowsError when running on Windows.
> Cython complains about non-builtin names here. I'm considering to turn that
> into a visible warning instead of an error, so that the name would instead
> be looked up dynamically to let the code fail at runtime *iff* it reaches
> the name lookup.

Given the usefulness of the error, and the (relative) lack of issues
with it so far, I'd rather not turn it into only a warning by default
(though an option might be nice). Another option would be to whitelist
the presumably small, finite set of names that are platform-dependent.

> Anyway, here are the numbers. I got them with "auto_cpdef" enabled, although
> that doesn't even seem to make that a big difference. The baseline is a
> self-compiled Python 2.7.1+ (about a month old).

Cool.  So basically everything is faster, usually somewhere between a
50-100% improvement. There's lots of room for improvement, though a
JIT has a significant advantage that we don't get for untyped code.

- Robert


> ### ai ###
> Min: 0.402407 -> 0.362190: 1.1110x faster
> Avg: 0.408784 -> 0.366898: 1.1142x faster
> Significant (t=10.017195, a=0.95)
> Stddev: 0.00824 -> 0.00442: 1.8668x smaller
>
>
> ### chaos ### ?(with a bug fixed in the benchmark itself)
> Min: 0.393362 -> 0.231932: 1.6960x faster
> Avg: 0.401941 -> 0.234089: 1.7170x faster
> Significant (t=36.128709, a=0.95)
> Stddev: 0.01004 -> 0.00267: 3.7538x smaller
>
>
> ### crypto_pyaes ###
> Min: 2.629560 -> 1.276433: 2.0601x faster
> Avg: 2.639409 -> 1.277742: 2.0657x faster
> Significant (t=368.652396, a=0.95)
> Stddev: 0.00812 -> 0.00153: 5.3215x smaller
>
>
> ### fannkuch ###
> Min: 1.512630 -> 0.853309: 1.7727x faster
> Avg: 1.522860 -> 0.860237: 1.7703x faster
> Significant (t=118.573908, a=0.95)
> Stddev: 0.00880 -> 0.00887: 1.0073x larger
>
>
> ### float ###
> Min: 0.452620 -> 0.343341: 1.3183x faster
> Avg: 0.475137 -> 0.349356: 1.3600x faster
> Significant (t=9.575876, a=0.95)
> Stddev: 0.02838 -> 0.00757: 3.7489x smaller
>
>
> ### go ###
> Min: 0.758998 -> 0.491929: 1.5429x faster
> Avg: 0.764110 -> 0.496518: 1.5389x faster
> Significant (t=90.848407, a=0.95)
> Stddev: 0.00400 -> 0.00523: 1.3096x larger
>
>
> ### nbody_modified ###
> Min: 0.399168 -> 0.197931: 2.0167x faster
> Avg: 0.401379 -> 0.203112: 1.9762x faster
> Significant (t=42.377829, a=0.95)
> Stddev: 0.00293 -> 0.01004: 3.4337x larger
>
>
> ### raytracesimple ### ? (module renamed from "raytrace-simple")
> Min: 2.016425 -> 1.182970: 1.7045x faster
> Avg: 2.030030 -> 1.192164: 1.7028x faster
> Significant (t=78.219481, a=0.95)
> Stddev: 0.02184 -> 0.00983: 2.2211x smaller
>
>
> ### richards ###
> Min: 0.286723 -> 0.162430: 1.7652x faster
> Avg: 0.289933 -> 0.165193: 1.7551x faster
> Significant (t=52.898468, a=0.95)
> Stddev: 0.00392 -> 0.00352: 1.1127x smaller
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Sat Apr 16 10:20:12 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Apr 2011 10:20:12 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <BANLkTikqutv+0WOw7U=uyO9ndM_aqs_z6w@mail.gmail.com>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
	<BANLkTikqutv+0WOw7U=uyO9ndM_aqs_z6w@mail.gmail.com>
Message-ID: <4DA9513C.1000503@behnel.de>

Robert Bradshaw, 16.04.2011 08:53:
> On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel wrote:
>> Stefan Behnel, 11.04.2011 15:08:
>>>
>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>>> works out well, we may have it up in a couple of days.
>>>
>>> I would expect that Cython won't be a big winner in this game, given that
>>> it will only compile plain untyped Python code. It's also going to fail
>>> entirely in some of the benchmarks. But I think it's worth having it up
>>> there, simply as a way for us to see where we are performance-wise and to
>>> get quick (nightly) feed-back about optimisations we try. The benchmark
>>> suite is also a nice set of real-world Python code that will allow us to
>>> find compliance issues.
>>
>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got at
>> least some of the benchmarks running. Note that they are actually simple
>> ones, only a single module. Basically all complex benchmarks fail due to
>> known bugs, such as Cython def functions not accepting attribute assignments
>> (e.g. on wrapping). There's also a problem with code that uses platform
>> specific names conditionally, such as WindowsError when running on Windows.
>> Cython complains about non-builtin names here. I'm considering to turn that
>> into a visible warning instead of an error, so that the name would instead
>> be looked up dynamically to let the code fail at runtime *iff* it reaches
>> the name lookup.
>
> Given the usefulness of the error, and the (relative) lack of issues
> with it so far, I'd rather not turn it into only a warning by default
> (though an option might be nice). Another option would be to whitelist
> the presumably small, finite set of names that are platform-dependent.

I agree, this has caught countless bugs in the past. I think a whitelist 
makes sense, but note that this does not obey Python semantics, either. In 
Python, any unknown name is just fine as long as it's not being looked up. 
Even though the use cases for this are clearly less common than the cases 
where it bites users.

I'm currently changing the builtins caching support to simply not cache 
unknown names, so that they will be looked up at runtime at the point where 
they are used (even though, of cause, they are compile time errors by 
default). In combination with a whitelist and with an option to make 
unknown builtins a warning instead of an error, this will give us a pretty 
good default trade-off between Python semantics, safety and performance, 
with an easy option for better Python compatibility.


>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled, although
>> that doesn't even seem to make that a big difference. The baseline is a
>> self-compiled Python 2.7.1+ (about a month old).
>
> Cool.  So basically everything is faster, usually somewhere between a
> 50-100% improvement. There's lots of room for improvement, though a
> JIT has a significant advantage that we don't get for untyped code.

Sure, we won't be as fast as PyPy for plain untyped Python code. But the 
benchmark suite gives us a clear target, both in terms of performance and 
compatibility.

Stefan

From robertwb at math.washington.edu  Sat Apr 16 10:30:31 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 Apr 2011 01:30:31 -0700
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DA9513C.1000503@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
	<BANLkTikqutv+0WOw7U=uyO9ndM_aqs_z6w@mail.gmail.com>
	<4DA9513C.1000503@behnel.de>
Message-ID: <BANLkTi=jcu3YrMUOuV51JTNjk1+gL=9M8Q@mail.gmail.com>

On Sat, Apr 16, 2011 at 1:20 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 16.04.2011 08:53:
>>
>> On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel wrote:
>>>
>>> Stefan Behnel, 11.04.2011 15:08:
>>>>
>>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get
>>>> Cython
>>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>>>> works out well, we may have it up in a couple of days.
>>>>
>>>> I would expect that Cython won't be a big winner in this game, given
>>>> that
>>>> it will only compile plain untyped Python code. It's also going to fail
>>>> entirely in some of the benchmarks. But I think it's worth having it up
>>>> there, simply as a way for us to see where we are performance-wise and
>>>> to
>>>> get quick (nightly) feed-back about optimisations we try. The benchmark
>>>> suite is also a nice set of real-world Python code that will allow us to
>>>> find compliance issues.
>>>
>>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got
>>> at
>>> least some of the benchmarks running. Note that they are actually simple
>>> ones, only a single module. Basically all complex benchmarks fail due to
>>> known bugs, such as Cython def functions not accepting attribute
>>> assignments
>>> (e.g. on wrapping). There's also a problem with code that uses platform
>>> specific names conditionally, such as WindowsError when running on
>>> Windows.
>>> Cython complains about non-builtin names here. I'm considering to turn
>>> that
>>> into a visible warning instead of an error, so that the name would
>>> instead
>>> be looked up dynamically to let the code fail at runtime *iff* it reaches
>>> the name lookup.
>>
>> Given the usefulness of the error, and the (relative) lack of issues
>> with it so far, I'd rather not turn it into only a warning by default
>> (though an option might be nice). Another option would be to whitelist
>> the presumably small, finite set of names that are platform-dependent.
>
> I agree, this has caught countless bugs in the past. I think a whitelist
> makes sense, but note that this does not obey Python semantics, either. In
> Python, any unknown name is just fine as long as it's not being looked up.
> Even though the use cases for this are clearly less common than the cases
> where it bites users.

Well, we certainly want to provide a way for users to disable it, and
would trigger it with the -pedantic flag.

> I'm currently changing the builtins caching support to simply not cache
> unknown names, so that they will be looked up at runtime at the point where
> they are used (even though, of cause, they are compile time errors by
> default). In combination with a whitelist and with an option to make unknown
> builtins a warning instead of an error, this will give us a pretty good
> default trade-off between Python semantics, safety and performance, with an
> easy option for better Python compatibility.

+1

>>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled,
>>> although
>>> that doesn't even seem to make that a big difference. The baseline is a
>>> self-compiled Python 2.7.1+ (about a month old).
>>
>> Cool. ?So basically everything is faster, usually somewhere between a
>> 50-100% improvement. There's lots of room for improvement, though a
>> JIT has a significant advantage that we don't get for untyped code.
>
> Sure, we won't be as fast as PyPy for plain untyped Python code. But the
> benchmark suite gives us a clear target, both in terms of performance and
> compatibility.

My thoughts as well. With good control flow in place, we can also look
into generating multiple branches for code (e.g. the likely, fast one
that assumes no overflows, etc. and bailing to the slow, safe one.)

- Robert

From robertwb at math.washington.edu  Sat Apr 16 10:51:39 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 Apr 2011 01:51:39 -0700
Subject: [Cython] Code examples missing in Cython User's Guide
In-Reply-To: <BANLkTim1nXMnPWoj8RqX-T30uWj1mbFG0g@mail.gmail.com>
References: <BANLkTikJEq6L73VtgZofci1ojbBh=Eu+DA@mail.gmail.com>
	<BANLkTinB329Y4=5itnwmpc4+0OeHODYLzg@mail.gmail.com>
	<BANLkTim1nXMnPWoj8RqX-T30uWj1mbFG0g@mail.gmail.com>
Message-ID: <BANLkTin2RZJUajc9QfF27VxXDcucmYYh4Q@mail.gmail.com>

On Thu, Apr 14, 2011 at 12:05 PM, Chris Lasher <chris.lasher at gmail.com> wrote:
> Thanks Arthur. I actually found the code examples in a grandparent directory
> of the User's Guide documentation source directory. I have submitted a pull
> request on GitHub which corrects the User's Guide Tutorial documentation so
> that it now includes the code.
>
> https://github.com/cython/cython/pull/24
>
> Could a Cython dev please review and approve this change

Done.

> and then build and
> update the docs on the Cython website for the sake of other Cython newbies?

This will be easier to do once we have a separate maintenance branch,
as we'd rather the documentation match the current, not next, release.
I don't think we (yet) have any new features mentioned in the docs, so
I'll go ahead an re-generate them (there's been a lot of good cleanup
in there, mostly thanks to Francesc Alted). The lastest docs can be
found at https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/.

- Robert

> On Tue, Apr 12, 2011 at 1:39 PM, Arthur de Souza Ribeiro
> <arthurdesribeiro at gmail.com> wrote:
>>
>> Hey Chris, the code for primes and fib examples, are in the directory
>> 'Demos' of the repository...
>> Best Regards.
>> []s
>> Arthur
>> 2011/4/12 Chris Lasher <chris.lasher at gmail.com>
>>>
>>> My apologies for cross-posting this from cython-users, but I realized I
>>> should have sent this bug report to cython-devel.
>>>
>>> Several code examples are missing from the User's Guide due to the source
>>> code files being moved or deleted. See for example the Tutorial page on the
>>> User's Guide http://docs.cython.org/src/userguide/tutorial.html The code for
>>> both fib.pyx and primes.pyx (and their setup.py files) is absent from the
>>> document.
>>>
>>> I looked at the ReST files and they are trying to source the files from
>>> an "examples" directory. Looking through the git repository, I wasn't able
>>> to locate this directory. Has it been moved or deleted?
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From d.s.seljebotn at astro.uio.no  Sat Apr 16 18:42:07 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 Apr 2011 18:42:07 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>	<4DA73D22.9050605@astro.uio.no>	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>	<4DA743EF.7080200@astro.uio.no>	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
Message-ID: <4DA9C6DF.1060406@astro.uio.no>

(Moving discussion from http://markflorisson.wordpress.com/, where Mark 
said:)

"""
Started a new branch https://github.com/markflorisson88/cython/tree/openmp .

Now the question is whether sharing attributes should be propagated 
outwards. e.g. if you do

for i in prange(m):
     for j in prange(n):
         sum += i * j

then ?sum? is a reduction for the inner parallel loop, but not for the 
outer one. So the user would currently have to rewrite this to

for i in prange(m):
     for j in prange(n):
         sum += i * j
     sum += 0

which seems a bit silly  . Of course, we could just disable nested 
parallelism, or tell the users to use a prange and a ?for from? in such 
cases.
"""

Dag: Interesting. The first one is definitely the behaviour we want, as 
long as it doesn't cause unintended consequences.

I don't really think it will -- the important thing is that that the 
order of loop iteration evaluation must be unimportant. And that is 
still true (for the outer loop, as well as for the inner) in your first 
example.

Question: When you have nested pranges, what will happen is that two 
nested OpenMP parallel blocks are used, right? And do you know if there 
is complete freedom/"reentrancy" in that variables that are 
thread-private in an outer parallel block and be shared in an inner one, 
and vice versa?

If so I'd think that this algorithm should work and feel natural:

  - In each prange, for the purposes of variable 
private/shared/reduction inference, consider all internal "prange" just 
as if they had been "range"; no special treatment.

  - Recurse to children pranges.

DS

From stefan_ml at behnel.de  Sat Apr 16 21:13:10 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Apr 2011 21:13:10 +0200
Subject: [Cython] Interpretting cython -a
In-Reply-To: <10374.32454.qm@web112112.mail.gq1.yahoo.com>
References: <10374.32454.qm@web112112.mail.gq1.yahoo.com>
Message-ID: <4DA9EA46.9000800@behnel.de>

Stuart Axon, 16.04.2011 19:44:
> Do I want things to be yellower or whiter?

White is C, yellow is Python.


> If I add an annotation and the colours don't change then I guess it's doing
> nothing, right.

You also want to click on the line in question and check the C code before 
an afterwards. If it doesn't change due to a type annotation, it's possible 
that Cython already inferred that type anyway, or that the annotation 
simply doesn't change the code, e.g. due to some other types it interacts with.

Stefan

From vitja.makarov at gmail.com  Sun Apr 17 17:57:28 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 17 Apr 2011 19:57:28 +0400
Subject: [Cython] Recent bugs in generators
Message-ID: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>

Hi!

1. Lambda-generator:

Previous implementation was inspired by Python2.6:
>>> list((lambda:((yield 1), (yield 2)))())
[1, 2, (None, None)]

things changed in python2.7:
>>> list((lambda:((yield 1), (yield 2)))())
[1, 2]

2. GeneratorExit is initialized to StopIteration when running
generators_py doctests

This is strange behaviour of doctest module, try this:
x = __builtins__
def foo():
    """
    >>> type(x)
    <type 'module'>
    """

3. check_yield_in_exception()

Cython calls __Pyx_ExceptionReset when except block is done, so when
yield is there no exception reset is called.

I'm not sure how to fix this.

import sys

def foo():
    """
    >>> list(foo())
    [<type 'exceptions.ValueError'>, None]
    """
    try:
        raise ValueError
    except ValueError:
        yield sys.exc_info()[0]
        yield sys.exc_info()[0] # exc_info is lost here


Here is quick fix for 1 and 2 https://github.com/cython/cython/pull/25

-- 
vitja.

From arthurdesribeiro at gmail.com  Sun Apr 17 20:07:38 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Sun, 17 Apr 2011 15:07:38 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DA7E653.4060801@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
	<4DA7E653.4060801@behnel.de>
Message-ID: <BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>

2011/4/15 Stefan Behnel <stefan_ml at behnel.de>

> [please avoid top-posting]
>
> Arthur de Souza Ribeiro, 15.04.2011 04:31:
>
>  I've created the .pyx files and it passed in all python tests.
>>
>
> Fine.
>
> As far as I can see, you only added static types in some places.

Did you test if they are actually required (maybe using "cython -a")? Some
> of them look rather counterproductive and should lead to a major slow-down.


In fact, I didn't, but, after you told me to do that, I run cython -a and
removed some unnecessary types.


> I added comments to your initial commit.
>

Hi Stefan, about your first comment : "And it's better to let Cython know
that this name refers to a function."  in line 69 of encoder.pyx file I
didn't understand well what does that mean, can you explain more this
comment?

About the other comments, I think I solved them all, any problem with them
or other ones, please tell me. I'll try to fix.


> Note that it's not obvious from your initial commit what you actually
> changed. It would have been better to import the original file first, rename
> it to .pyx, and then commit your changes.
>

I created a directory named 'Diff files' where I put the files generated by
'diff' command that i run in my computer, if you think it still be better if
I commit and then change, there is no problem for me...


>
> It appears that you accidentally added your .c and .so files to your repo.
>
>
> https://github.com/arthursribeiro/JSON-module
>
>
Removed them.


>
>  To test them, as I said, I copied the .py test files to my project
>> directory, generated the .so files, import them instead of python modules
>> and run. I run every test file and it passed in all of them. To run the
>> tests, run the file 'run-tests.sh'
>>
>> I used just .pyx in this module, should I reimplement it using pxd with
>> the
>> normal .py?
>>
>
> Not at this point. I think it's more important to get some performance
> numbers to see how your module behaves compared to the C accelerator module
> (_json.c). I think the best approach to this project would actually be to
> start with profiling the Python implementation to see where performance
> problems occur (or to look through _json.c to see what the CPython
> developers considered performance critical), and then put the focus on
> trying to speed up only those parts of the Python implementation, by adding
> static types and potentially even rewriting them in a way that Cython can
> optimise them better.
>

I've profilled the module I created and the module that is in Python 3.2,
the result is that the cython module spent about 73% less time then python's
one, the output to the module was like this (blue for cython, red for
python):

The behavior between my module and python's one seems to be the same I think
that's the way it should be.

JSONModule nested_dict
         10004 function calls in 0.268 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10000    0.196    0.000    0.196    0.000 :0(dumps)
        1    0.000    0.000    0.268    0.268 :0(exec)
        1    0.000    0.000    0.000    0.000 :0(setprofile)
        1    0.072    0.072    0.268    0.268 <string>:1(<module>)
        1    0.000    0.000    0.268    0.268 profile:0(for ii in
range(10000):  fun(thing))
        0    0.000             0.000          profile:0(profiler)


json nested_dict
         60004 function calls in 1.016 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.016    1.016 :0(exec)
    20000    0.136    0.000    0.136    0.000 :0(isinstance)
    10000    0.120    0.000    0.120    0.000 :0(join)
        1    0.000    0.000    0.000    0.000 :0(setprofile)
        1    0.088    0.088    1.016    1.016 <string>:1(<module>)
    10000    0.136    0.000    0.928    0.000 __init__.py:180(dumps)
    10000    0.308    0.000    0.792    0.000 encoder.py:172(encode)
    10000    0.228    0.000    0.228    0.000 encoder.py:193(iterencode)
        1    0.000    0.000    1.016    1.016 profile:0(for ii in
range(10000):  fun(thing))
        0    0.000             0.000          profile:0(profiler)


JSONModule ustring
         10004 function calls in 0.140 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10000    0.072    0.000    0.072    0.000 :0(dumps)
        1    0.000    0.000    0.140    0.140 :0(exec)
        1    0.000    0.000    0.000    0.000 :0(setprofile)
        1    0.068    0.068    0.140    0.140 <string>:1(<module>)
        1    0.000    0.000    0.140    0.140 profile:0(for ii in
range(10000):  fun(thing))
        0    0.000             0.000          profile:0(profiler)


json ustring
         40004 function calls in 0.580 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10000    0.092    0.000    0.092    0.000 :0(encode_basestring_ascii)
        1    0.004    0.004    0.580    0.580 :0(exec)
    10000    0.060    0.000    0.060    0.000 :0(isinstance)
        1    0.000    0.000    0.000    0.000 :0(setprofile)
        1    0.100    0.100    0.576    0.576 <string>:1(<module>)
    10000    0.152    0.000    0.476    0.000 __init__.py:180(dumps)
    10000    0.172    0.000    0.324    0.000 encoder.py:172(encode)
        1    0.000    0.000    0.580    0.580 profile:0(for ii in
range(10000):  fun(thing))
        0    0.000             0.000          profile:0(profiler)

The code is upated in repository, any comments that you might have, please,
let me know. Thank you very much for your feedback.

Best Regards.

[]s

Arthur


>
> Stefan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110417/fb55539c/attachment.html>

From sturla at molden.no  Sun Apr 17 20:24:11 2011
From: sturla at molden.no (Sturla Molden)
Date: Sun, 17 Apr 2011 20:24:11 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>	<4DA3F455.8070006@behnel.de>	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>	<4DA4984D.3020007@behnel.de>	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>	<4DA5D624.50609@behnel.de>	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>	<4DA7E653.4060801@behnel.de>
	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
Message-ID: <4DAB304B.4010801@molden.no>

Den 17.04.2011 20:07, skrev Arthur de Souza Ribeiro:
>
> I've profilled the module I created and the module that is in Python 
> 3.2, the result is that the cython module spent about 73% less time 
> then python's one, the output to the module was like this (blue for 
> cython, red for python):
>
>

The number of function calls are different. For nested_dict, you have 
37320 calls per second for Cython and 59059 calls per second for Python. 
I am not convinced that is better.

Sturla

From stefan_ml at behnel.de  Sun Apr 17 21:16:39 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Apr 2011 21:16:39 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DAB304B.4010801@molden.no>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>	<4DA3F455.8070006@behnel.de>	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>	<4DA4984D.3020007@behnel.de>	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>	<4DA5D624.50609@behnel.de>	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>	<4DA7E653.4060801@behnel.de>	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
	<4DAB304B.4010801@molden.no>
Message-ID: <4DAB3C97.6070208@behnel.de>

Sturla Molden, 17.04.2011 20:24:
> Den 17.04.2011 20:07, skrev Arthur de Souza Ribeiro:
>> I've profilled the module I created and the module that is in Python 3.2,
>> the result is that the cython module spent about 73% less time then
>> python's one, the output to the module was like this (blue for cython,
>> red for python):
>
> The number of function calls are different. For nested_dict, you have 37320
> calls per second for Cython and 59059 calls per second for Python. I am not
> convinced that is better.

Note that there are 20000 calls to isinstance(), which Cython handles 
internally. The profiler cannot see those.

However, the different number of functions calls also makes the profiling 
results less comparable, since there are fewer calls into the profiler. 
This leads to a lower performance penalty for Cython in the absolute 
timings, and consequently to an unfair comparison.

Stefan

From sturla at molden.no  Sun Apr 17 22:11:19 2011
From: sturla at molden.no (Sturla Molden)
Date: Sun, 17 Apr 2011 22:11:19 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DAB3C97.6070208@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>	<4DA3F455.8070006@behnel.de>	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>	<4DA4984D.3020007@behnel.de>	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>	<4DA5D624.50609@behnel.de>	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>	<4DA7E653.4060801@behnel.de>	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>	<4DAB304B.4010801@molden.no>
	<4DAB3C97.6070208@behnel.de>
Message-ID: <4DAB4967.5020507@molden.no>

Den 17.04.2011 21:16, skrev Stefan Behnel:
>
> However, the different number of functions calls also makes the 
> profiling results less comparable, since there are fewer calls into 
> the profiler. This leads to a lower performance penalty for Cython in 
> the absolute timings, and consequently to an unfair comparison.
>

As I understand it, the profiler will give a profile of a module.

To measure absolute performance, one should use timeit or just time.clock.

Sturla



From stefan_ml at behnel.de  Sun Apr 17 22:56:48 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Apr 2011 22:56:48 +0200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
	<4DA7E653.4060801@behnel.de>
	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
Message-ID: <4DAB5410.4050404@behnel.de>

Arthur de Souza Ribeiro, 17.04.2011 20:07:
> Hi Stefan, about your first comment : "And it's better to let Cython know
> that this name refers to a function."  in line 69 of encoder.pyx file I
> didn't understand well what does that mean, can you explain more this
> comment?

Hmm, sorry, I think that was not so important. That code line is only used 
to override the Python implementation with the implementation from the 
external C accelerator module. To do that, it assigns either of the two 
functions to a name. So, when that name is called in the code, Cython 
cannot know that it actually is a function, and has to resort to Python 
calling, whereas a visible c(p)def function that is defined inside of the 
same module could be called faster.

I missed the fact that this name isn't really used inside of the module, so 
whether Cython knows that it's a function or not isn't really all that 
important.

I added another comment to this commit, though:

https://github.com/arthursribeiro/JSON-module/commit/e2d80e0aeab6d39ff2d9b847843423ebdb9c57b7#diff-4


> About the other comments, I think I solved them all, any problem with them
> or other ones, please tell me. I'll try to fix.

It looks like you fixed a good deal of them.

I actually tried to work with your code, but I'm not sure how you are 
building it. Could you give me a hint on that?

Where did you actually take the original code from? Python 3.2? Or from 
Python's hg branch?


>> Note that it's not obvious from your initial commit what you actually
>> changed. It would have been better to import the original file first, rename
>> it to .pyx, and then commit your changes.
>
> I created a directory named 'Diff files' where I put the files generated by
> 'diff' command that i run in my computer, if you think it still be better if
> I commit and then change, there is no problem for me...

Diff only gives you the final outcome. Committing on top of the original 
files has the advantage of making the incremental changes visible 
separately. That makes it clearer what you tried, and a good commit comment 
will then make it clear why you did it.


>> I think it's more important to get some performance
>> numbers to see how your module behaves compared to the C accelerator module
>> (_json.c). I think the best approach to this project would actually be to
>> start with profiling the Python implementation to see where performance
>> problems occur (or to look through _json.c to see what the CPython
>> developers considered performance critical), and then put the focus on
>> trying to speed up only those parts of the Python implementation, by adding
>> static types and potentially even rewriting them in a way that Cython can
>> optimise them better.
>
> I've profilled the module I created and the module that is in Python 3.2,
> the result is that the cython module spent about 73% less time then python's

That's a common mistake when profiling: the actual time it takes to run is 
not meaningful. Depending on how far the two profiled programs differ, they 
may interact with the profiler in more or less intensive ways (as is 
clearly the case here), so the total time it takes for the programs to run 
can differ quite heavily under profiling, even if the non-profiled programs 
run at exactly the same speed.

Also, I don't think you have enabled profiling for the Cython code. You can 
do that by passing the "profile=True" directive to the compiler, or by 
putting it at the top of the source files. That will add module-inner 
function calls to the profiling output. Note, however, that enabling 
profiling will slow down the execution, so disable it when you measure 
absolute run times.

http://docs.cython.org/src/tutorial/profiling_tutorial.html


> (blue for cython, red for python):

Colours tend to pass rather badly through mailing lists. Many people 
disable the HTML presentation of e-mails, and plain text does not have 
colours. But it was still obvious enough what you meant.


> The behavior between my module and python's one seems to be the same I think
> that's the way it should be.
>
> JSONModule nested_dict
>           10004 function calls in 0.268 seconds
>
>     Ordered by: standard name
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>      10000    0.196    0.000    0.196    0.000 :0(dumps)

This is a pretty short list (I stripped the uninteresting parts). The 
profile right below shows a lot more entries in encoder.py. It would be 
good to see these calls in the Cython code as well.


> json nested_dict
>           60004 function calls in 1.016 seconds
>
>     Ordered by: standard name
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>          1    0.000    0.000    1.016    1.016 :0(exec)
>      20000    0.136    0.000    0.136    0.000 :0(isinstance)
>      10000    0.120    0.000    0.120    0.000 :0(join)
>          1    0.000    0.000    0.000    0.000 :0(setprofile)
>          1    0.088    0.088    1.016    1.016<string>:1(<module>)
>      10000    0.136    0.000    0.928    0.000 __init__.py:180(dumps)
>      10000    0.308    0.000    0.792    0.000 encoder.py:172(encode)
>      10000    0.228    0.000    0.228    0.000 encoder.py:193(iterencode)
> [...]
> JSONModule ustring
>           10004 function calls in 0.140 seconds
>
>     Ordered by: standard name
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>      10000    0.072    0.000    0.072    0.000 :0(dumps)
> [...]
>
> json ustring
>           40004 function calls in 0.580 seconds
>
>     Ordered by: standard name
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>      10000    0.092    0.000    0.092    0.000 :0(encode_basestring_ascii)
>          1    0.004    0.004    0.580    0.580 :0(exec)
>      10000    0.060    0.000    0.060    0.000 :0(isinstance)
>          1    0.000    0.000    0.000    0.000 :0(setprofile)
>          1    0.100    0.100    0.576    0.576<string>:1(<module>)
>      10000    0.152    0.000    0.476    0.000 __init__.py:180(dumps)
>      10000    0.172    0.000    0.324    0.000 encoder.py:172(encode)
>
> The code is upated in repository, any comments that you might have, please,
> let me know. Thank you very much for your feedback.

Thank you for the numbers. Could you add absolute timings using timeit? And 
maybe also try with larger input data?

ISTM that a lot of overhead comes from calls that Cython can easily 
optimise all by itself: isinstance() and (bytes|unicode).join(). That's the 
kind of observation that previously let me suggest to start by benchmarking 
and profiling in the first place. Cython compiled code has quite different 
performance characteristics from code executing in CPython's interpreter, 
so it's important to start by getting an idea of how the code behaves when 
compiled, and then optimising it in the places where it still needs to run 
faster.

Optimisation is an incremental process, and you will often end up reverting 
changes along the way when you see that they did not improve the 
performance, or maybe just made it so slightly faster that the speed 
improvement is not worth the code degradation of the optimisation change in 
question.

Could you try to come up with a short list of important code changes you 
made that let this module run faster, backed by some timings that show the 
difference with and without each change?

Stefan

From stefan_ml at behnel.de  Sun Apr 17 23:05:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Apr 2011 23:05:45 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
Message-ID: <4DAB5629.2020500@behnel.de>

Vitja Makarov, 17.04.2011 17:57:
> 3. check_yield_in_exception()

I added this because I found a failing pyregr test that uses it (testing 
the @contextmanager decorator).


> Cython calls __Pyx_ExceptionReset when except block is done, so when
> yield is there no exception reset is called.
>
> I'm not sure how to fix this.

I'm not completely sure either.


> import sys
>
> def foo():
>      """
>      >>>  list(foo())
>      [<type 'exceptions.ValueError'>, None]
>      """
>      try:
>          raise ValueError
>      except ValueError:
>          yield sys.exc_info()[0]
>          yield sys.exc_info()[0] # exc_info is lost here

I think (!), the difference here is that CPython actually keeps the 
exception in the generator frame. We don't have a frame, so we have to 
emulate it using the closure class. I guess we'll have to store away the 
exception into the closure when we yield while an exception is being 
handled, and restore it afterwards. Note: this is not the exception that is 
freshly *being* raised (the "_cur*" fields in the thread state), it's the 
exception that *was* raised and is now being handled, i.e. the thread state 
fields without the "_cur", that are reflected by sys.exc_info().

Stefan

From vitja.makarov at gmail.com  Mon Apr 18 06:38:20 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 18 Apr 2011 08:38:20 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DAB5629.2020500@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
Message-ID: <BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>

2011/4/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 17.04.2011 17:57:
>>
>> 3. check_yield_in_exception()
>
> I added this because I found a failing pyregr test that uses it (testing the
> @contextmanager decorator).
>
>
>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>> yield is there no exception reset is called.
>>
>> I'm not sure how to fix this.
>
> I'm not completely sure either.
>
>
>> import sys
>>
>> def foo():
>> ? ? """
>> ? ? >>> ?list(foo())
>> ? ? [<type 'exceptions.ValueError'>, None]
>> ? ? """
>> ? ? try:
>> ? ? ? ? raise ValueError
>> ? ? except ValueError:
>> ? ? ? ? yield sys.exc_info()[0]
>> ? ? ? ? yield sys.exc_info()[0] # exc_info is lost here
>
> I think (!), the difference here is that CPython actually keeps the
> exception in the generator frame. We don't have a frame, so we have to
> emulate it using the closure class. I guess we'll have to store away the
> exception into the closure when we yield while an exception is being
> handled, and restore it afterwards. Note: this is not the exception that is
> freshly *being* raised (the "_cur*" fields in the thread state), it's the
> exception that *was* raised and is now being handled, i.e. the thread state
> fields without the "_cur", that are reflected by sys.exc_info().
>

Interesting difference between py2 and py3:

def foo():
    try:
        raise ValueError
    except ValueError:
        yield
        raise
list(foo())

  File "xxx.py", line 7, in <module>
    list(foo())
  File "xxx.py", line 6, in foo
    raise
TypeError: exceptions must be old-style classes or derived from
BaseException, not NoneType

It seems that exception info is completely lost (tried 2.6, 2.7) and
seems to be fixed in python3.

Btw exception info temps are already saved and restored between yields.


-- 
vitja.

From stefan_ml at behnel.de  Mon Apr 18 06:57:27 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 18 Apr 2011 06:57:27 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
Message-ID: <4DABC4B7.8030904@behnel.de>

Vitja Makarov, 18.04.2011 06:38:
> 2011/4/18 Stefan Behnel<stefan_ml at behnel.de>:
>> Vitja Makarov, 17.04.2011 17:57:
>>>
>>> 3. check_yield_in_exception()
>>
>> I added this because I found a failing pyregr test that uses it (testing the
>> @contextmanager decorator).
>>
>>
>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>> yield is there no exception reset is called.
>>>
>>> I'm not sure how to fix this.
>>
>> I'm not completely sure either.
>>
>>
>>> import sys
>>>
>>> def foo():
>>>      """
>>>      >>>    list(foo())
>>>      [<type 'exceptions.ValueError'>, None]
>>>      """
>>>      try:
>>>          raise ValueError
>>>      except ValueError:
>>>          yield sys.exc_info()[0]
>>>          yield sys.exc_info()[0] # exc_info is lost here
>>
>> I think (!), the difference here is that CPython actually keeps the
>> exception in the generator frame. We don't have a frame, so we have to
>> emulate it using the closure class. I guess we'll have to store away the
>> exception into the closure when we yield while an exception is being
>> handled, and restore it afterwards. Note: this is not the exception that is
>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>> exception that *was* raised and is now being handled, i.e. the thread state
>> fields without the "_cur", that are reflected by sys.exc_info().
>
> Interesting difference between py2 and py3:
>
> def foo():
>      try:
>          raise ValueError
>      except ValueError:
>          yield
>          raise
> list(foo())
>
>    File "xxx.py", line 7, in<module>
>      list(foo())
>    File "xxx.py", line 6, in foo
>      raise
> TypeError: exceptions must be old-style classes or derived from
> BaseException, not NoneType
>
> It seems that exception info is completely lost (tried 2.6, 2.7) and
> seems to be fixed in python3.

Not surprising. The implementation is completely different in Py2 and Py3, 
both in CPython and in Cython. It's actually much simpler in Cython under 
Py3, due to better semantics and C-API support. That also implies that 
there's much less Cython can do wrong in that environment. ;-)


> Btw exception info temps are already saved and restored between yields.

Right, but the exc_info itself is not reset and recovered around the yield. 
As I said above, generators have their own lifetime frame in CPython, and 
exceptions don't leak from that. So, whenever it's the generator (or code 
called by it) that raises an exception, that must be kept local to the 
generator.

Stefan

From stefan_ml at behnel.de  Mon Apr 18 11:19:10 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 18 Apr 2011 11:19:10 +0200
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <20110407150110.GA13395@ubuntu>
References: <20110407150110.GA13395@ubuntu>
Message-ID: <4DAC020E.2010707@behnel.de>

Romain Guillebert, 07.04.2011 17:01:
> I proposed the Summer of Code project regarding the Python backend for
> Cython.
>
> As I said in my proposal this would translate Cython code to Python +
> FFI code (I don't know yet if it will use ctypes or something specific
> to PyPy). PyPy's ctypes is now really fast and this will allow people to
> port their Cython code to PyPy.
>
> For the moment I've been mostly in touch with the PyPy people and they
> seem happy with my proposal.
>
> Of course I'm available for questions.

Hi Romain,

it's usually required for GSoC students to provide a patch for the project 
they will be participating in. It appears that you have provided patches 
for PyPy that were well accepted, but since you'd be working mostly on 
Cython, it would be helpful for us to get an idea about how well you know 
the Cython source code by now.

So, here's a little exam. If you were to implement support for the 
globals() builtin in Cython, what would you consider the necessary parts of 
the implementation? How would you approach this task? Feel free to ask for 
any information or pointers you need to solve this.

Stefan

From markflorisson88 at gmail.com  Mon Apr 18 13:06:19 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 18 Apr 2011 13:06:19 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DA9C6DF.1060406@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
Message-ID: <BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>

On 16 April 2011 18:42, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> (Moving discussion from http://markflorisson.wordpress.com/, where Mark
> said:)

Ok, sure, it was just an issue I was wondering about at that moment,
but it's a tricky issue, so thanks.

> """
> Started a new branch https://github.com/markflorisson88/cython/tree/openmp .
>
> Now the question is whether sharing attributes should be propagated
> outwards. e.g. if you do
>
> for i in prange(m):
> ? ?for j in prange(n):
> ? ? ? ?sum += i * j
>
> then ?sum? is a reduction for the inner parallel loop, but not for the outer
> one. So the user would currently have to rewrite this to
>
> for i in prange(m):
> ? ?for j in prange(n):
> ? ? ? ?sum += i * j
> ? ?sum += 0
>
> which seems a bit silly ?. Of course, we could just disable nested
> parallelism, or tell the users to use a prange and a ?for from? in such
> cases.
> """
>
> Dag: Interesting. The first one is definitely the behaviour we want, as long
> as it doesn't cause unintended consequences.
>
> I don't really think it will -- the important thing is that that the order
> of loop iteration evaluation must be unimportant. And that is still true
> (for the outer loop, as well as for the inner) in your first example.
>
> Question: When you have nested pranges, what will happen is that two nested
> OpenMP parallel blocks are used, right? And do you know if there is complete
> freedom/"reentrancy" in that variables that are thread-private in an outer
> parallel block and be shared in an inner one, and vice versa?

An implementation may or may not support it, and if it is supported
the behaviour can be configured through omp_set_nested(). So we should
consider the case where it is supported and enabled.

If you have a lastprivate or reduction, and after the loop these are
(reduced and) assigned to the original variable. So if that happens
inside a parallel construct which does not declare the variable
private to the construct, you actually have a race. So e.g. the nested
prange currently races in the outer parallel range.

> If so I'd think that this algorithm should work and feel natural:
>
> ?- In each prange, for the purposes of variable private/shared/reduction
> inference, consider all internal "prange" just as if they had been "range";
> no special treatment.
>
> ?- Recurse to children pranges.

Right, that is most natural. Algorithmically, reductions and
lastprivates (as those can have races if placed in inner parallel
constructs) propagate outwards towards the outermost parallel block,
or up to the first parallel with block, or up to the first construct
that already determined the sharing attribute.

e.g.

with parallel:
     with parallel:
        for i in prange(n):
            for j in prange(n):
                sum += i * j
     # sum is well-defined here
# sum is undefined here

Here 'sum' is a reduction for the two innermost loops. 'sum' is not
private for the inner parallel with block, as a prange in a parallel
with block is a worksharing loop that binds to that parallel with
block. However, the outermost parallel with block declares sum (and i
and j) private, so after that block all those variables become
undefined.

However, in the outermost parallel with block, sum will have to be
initialized to 0 before anything else, or be declared firstprivate,
otherwise 'sum' is undefined to begin with. Do you think declaring it
firstprivate would be the way to go, or should we make it private and
issue a warning or perhaps even an error?

> DS
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From vitja.makarov at gmail.com  Mon Apr 18 15:19:18 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 18 Apr 2011 17:19:18 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DABC4B7.8030904@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
Message-ID: <BANLkTikqLSsHiCqKT1ZH4zfTtsCQ0UYcDQ@mail.gmail.com>

2011/4/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.04.2011 06:38:
>>
>> 2011/4/18 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 17.04.2011 17:57:
>>>>
>>>> 3. check_yield_in_exception()
>>>
>>> I added this because I found a failing pyregr test that uses it (testing
>>> the
>>> @contextmanager decorator).
>>>
>>>
>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>> yield is there no exception reset is called.
>>>>
>>>> I'm not sure how to fix this.
>>>
>>> I'm not completely sure either.
>>>
>>>
>>>> import sys
>>>>
>>>> def foo():
>>>> ? ? """
>>>> ? ? >>> ? ?list(foo())
>>>> ? ? [<type 'exceptions.ValueError'>, None]
>>>> ? ? """
>>>> ? ? try:
>>>> ? ? ? ? raise ValueError
>>>> ? ? except ValueError:
>>>> ? ? ? ? yield sys.exc_info()[0]
>>>> ? ? ? ? yield sys.exc_info()[0] # exc_info is lost here
>>>
>>> I think (!), the difference here is that CPython actually keeps the
>>> exception in the generator frame. We don't have a frame, so we have to
>>> emulate it using the closure class. I guess we'll have to store away the
>>> exception into the closure when we yield while an exception is being
>>> handled, and restore it afterwards. Note: this is not the exception that
>>> is
>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>> exception that *was* raised and is now being handled, i.e. the thread
>>> state
>>> fields without the "_cur", that are reflected by sys.exc_info().
>>
>> Interesting difference between py2 and py3:
>>
>> def foo():
>> ? ? try:
>> ? ? ? ? raise ValueError
>> ? ? except ValueError:
>> ? ? ? ? yield
>> ? ? ? ? raise
>> list(foo())
>>
>> ? File "xxx.py", line 7, in<module>
>> ? ? list(foo())
>> ? File "xxx.py", line 6, in foo
>> ? ? raise
>> TypeError: exceptions must be old-style classes or derived from
>> BaseException, not NoneType
>>
>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>> seems to be fixed in python3.
>
> Not surprising. The implementation is completely different in Py2 and Py3,
> both in CPython and in Cython. It's actually much simpler in Cython under
> Py3, due to better semantics and C-API support. That also implies that
> there's much less Cython can do wrong in that environment. ;-)
>
>
>> Btw exception info temps are already saved and restored between yields.
>
> Right, but the exc_info itself is not reset and recovered around the yield.
> As I said above, generators have their own lifetime frame in CPython, and
> exceptions don't leak from that. So, whenever it's the generator (or code
> called by it) that raises an exception, that must be kept local to the
> generator.
>

There is one more interesting thing:

def test_yield_inside_genexp():
    """
    >>> o = test_yield_inside_genexp()
    >>> list(o)
    [0, None, 1, None, 2, None]
    """
    return ((yield i) for i in range(3))


-- 
vitja.

From markflorisson88 at gmail.com  Mon Apr 18 15:57:30 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 18 Apr 2011 15:57:30 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
Message-ID: <BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>

On 18 April 2011 13:06, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 April 2011 18:42, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> (Moving discussion from http://markflorisson.wordpress.com/, where Mark
>> said:)
>
> Ok, sure, it was just an issue I was wondering about at that moment,
> but it's a tricky issue, so thanks.
>
>> """
>> Started a new branch https://github.com/markflorisson88/cython/tree/openmp .
>>
>> Now the question is whether sharing attributes should be propagated
>> outwards. e.g. if you do
>>
>> for i in prange(m):
>> ? ?for j in prange(n):
>> ? ? ? ?sum += i * j
>>
>> then ?sum? is a reduction for the inner parallel loop, but not for the outer
>> one. So the user would currently have to rewrite this to
>>
>> for i in prange(m):
>> ? ?for j in prange(n):
>> ? ? ? ?sum += i * j
>> ? ?sum += 0
>>
>> which seems a bit silly ?. Of course, we could just disable nested
>> parallelism, or tell the users to use a prange and a ?for from? in such
>> cases.
>> """
>>
>> Dag: Interesting. The first one is definitely the behaviour we want, as long
>> as it doesn't cause unintended consequences.
>>
>> I don't really think it will -- the important thing is that that the order
>> of loop iteration evaluation must be unimportant. And that is still true
>> (for the outer loop, as well as for the inner) in your first example.
>>
>> Question: When you have nested pranges, what will happen is that two nested
>> OpenMP parallel blocks are used, right? And do you know if there is complete
>> freedom/"reentrancy" in that variables that are thread-private in an outer
>> parallel block and be shared in an inner one, and vice versa?
>
> An implementation may or may not support it, and if it is supported
> the behaviour can be configured through omp_set_nested(). So we should
> consider the case where it is supported and enabled.
>
> If you have a lastprivate or reduction, and after the loop these are
> (reduced and) assigned to the original variable. So if that happens
> inside a parallel construct which does not declare the variable
> private to the construct, you actually have a race. So e.g. the nested
> prange currently races in the outer parallel range.
>
>> If so I'd think that this algorithm should work and feel natural:
>>
>> ?- In each prange, for the purposes of variable private/shared/reduction
>> inference, consider all internal "prange" just as if they had been "range";
>> no special treatment.
>>
>> ?- Recurse to children pranges.
>
> Right, that is most natural. Algorithmically, reductions and
> lastprivates (as those can have races if placed in inner parallel
> constructs) propagate outwards towards the outermost parallel block,
> or up to the first parallel with block, or up to the first construct
> that already determined the sharing attribute.
>
> e.g.
>
> with parallel:
> ? ? with parallel:
> ? ? ? ?for i in prange(n):
> ? ? ? ? ? ?for j in prange(n):
> ? ? ? ? ? ? ? ?sum += i * j
> ? ? # sum is well-defined here
> # sum is undefined here
>
> Here 'sum' is a reduction for the two innermost loops. 'sum' is not
> private for the inner parallel with block, as a prange in a parallel
> with block is a worksharing loop that binds to that parallel with
> block. However, the outermost parallel with block declares sum (and i
> and j) private, so after that block all those variables become
> undefined.
>
> However, in the outermost parallel with block, sum will have to be
> initialized to 0 before anything else, or be declared firstprivate,
> otherwise 'sum' is undefined to begin with. Do you think declaring it
> firstprivate would be the way to go, or should we make it private and
> issue a warning or perhaps even an error?
>
>> DS
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

Everything seems to be working, although now the user has to be
careful with nested parallel blocks as variables can be private there
(and not firstprivate), i.e., the user has to do initialization at the
right place (e.g. in the outermost parallel block that determines it
private). I'm thinking of adding a warning, as the C compiler does.

Two issues are remaining:

1) explicit declarations of firstprivates

Do we still want those?

2) buffer auxiliary vars

When unpacking numpy buffers and using typed numpy arrays, can
reassignment or updates of a buffer-related variable ever occur in
nogil code sections? I'm thinking this is not possible and therefore
all buffer variables may be shared in parallel (for) sections?

From d.s.seljebotn at astro.uio.no  Mon Apr 18 16:01:51 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 18 Apr 2011 16:01:51 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
Message-ID: <a9589bb5-6780-4e3d-a520-e847b3826c97@email.android.com>

(apologies for top post)

This all seems to scream 'disallow' to me, in particular since some openmp implementations may not support it etc.

At any rate I feel 'parallel/parallel/prange/prange' is going to far; so next step could be to only allowing 'parallel/prange/parallel/prange'.

But really, my feeling is that if you really do need this then you can always write a seperate function for the inner loop (I honestly can't think of a usecase anyway...). So I'd really drop it; at least until the rest of the gsoc project is completed :)

DS
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

mark florisson <markflorisson88 at gmail.com> wrote:

On 16 April 2011 18:42, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where Mark > said:) Ok, sure, it was just an issue I was wondering about at that moment, but it's a tricky issue, so thanks. > """ > Started a new branch https://github.com/markflorisson88/cython/tree/openmp . > > Now the question is whether sharing attributes should be propagated > outwards. e.g. if you do > > for i in prange(m): >    for j in prange(n): >        sum += i * j > > then ?sum? is a reduction for the inner parallel loop, but not for the outer > one. So the user would currently have to rewrite this to > > for i in prange(m): >    for j in prange(n): >        sum += i * j >    sum += 0 > > which seems a bit silly  . Of course, we could just disable nested > parallelism, or tell the users to use a prange and a ?for from? in such > cases. > """ > > Dag: Interesting. The first one is definitely the behaviour we want, as long
> as it doesn't cause unintended consequences. > > I don't really think it will -- the important thing is that that the order > of loop iteration evaluation must be unimportant. And that is still true > (for the outer loop, as well as for the inner) in your first example. > > Question: When you have nested pranges, what will happen is that two nested > OpenMP parallel blocks are used, right? And do you know if there is complete > freedom/"reentrancy" in that variables that are thread-private in an outer > parallel block and be shared in an inner one, and vice versa? An implementation may or may not support it, and if it is supported the behaviour can be configured through omp_set_nested(). So we should consider the case where it is supported and enabled. If you have a lastprivate or reduction, and after the loop these are (reduced and) assigned to the original variable. So if that happens inside a parallel construct which does not declare the variable private to the construct, you
actually have a race. So e.g. the nested prange currently races in the outer parallel range. > If so I'd think that this algorithm should work and feel natural: > >  - In each prange, for the purposes of variable private/shared/reduction > inference, consider all internal "prange" just as if they had been "range"; > no special treatment. > >  - Recurse to children pranges. Right, that is most natural. Algorithmically, reductions and lastprivates (as those can have races if placed in inner parallel constructs) propagate outwards towards the outermost parallel block, or up to the first parallel with block, or up to the first construct that already determined the sharing attribute. e.g. with parallel: with parallel: for i in prange(n): for j in prange(n): sum += i * j # sum is well-defined here # sum is undefined here Here 'sum' is a reduction for the two innermost loops. 'sum' is not private for the inner parallel with block, as a prange in a parallel with block is a worksharing loop
that binds to that parallel with block. However, the outermost parallel with block declares sum (and i and j) private, so after that block all those variables become undefined. However, in the outermost parallel with block, sum will have to be initialized to 0 before anything else, or be declared firstprivate, otherwise 'sum' is undefined to begin with. Do you think declaring it firstprivate would be the way to go, or should we make it private and issue a warning or perhaps even an error? > DS >_____________________________________________
> cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel >_____________________________________________
cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110418/c42cac71/attachment-0001.html>

From markflorisson88 at gmail.com  Mon Apr 18 16:05:44 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 18 Apr 2011 16:05:44 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <a9589bb5-6780-4e3d-a520-e847b3826c97@email.android.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<a9589bb5-6780-4e3d-a520-e847b3826c97@email.android.com>
Message-ID: <BANLkTimCHP=wtxYUmrkGcsJ9K8iwYTKaRQ@mail.gmail.com>

On 18 April 2011 16:01, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> (apologies for top post)

No problem, it means I have to scroll less :)

> This all seems to scream 'disallow' to me, in particular since some openmp
> implementations may not support it etc.
>
> At any rate I feel 'parallel/parallel/prange/prange' is going to far; so
> next step could be to only allowing 'parallel/prange/parallel/prange'.
>
> But really, my feeling is that if you really do need this then you can
> always write a seperate function for the inner loop (I honestly can't think
> of a usecase anyway...). So I'd really drop it; at least until the rest of
> the gsoc project is completed :)

Ok, sure, I'll disallow it. Then the user won't be able to make
mistakes and I don't have to detect the case and issue a warning for
inner reductions or lastprivates :).

> DS
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> mark florisson <markflorisson88 at gmail.com> wrote:
>>
>> On 16 April 2011 18:42, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>> wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where
>> Mark > said:) Ok, sure, it was just an issue I was wondering about at that
>> moment, but it's a tricky issue, so thanks. > """ > Started a new branch
>> https://github.com/markflorisson88/cython/tree/openmp . > > Now the question
>> is whether sharing attributes should be propagated > outwards. e.g. if you
>> do > > for i in prange(m): > ? ?for j in prange(n): > ? ? ? ?sum += i * j >
>> > then ?sum? is a reduction for the inner parallel loop, but not for the
>> outer > one. So the user would currently have to rewrite this to > > for i
>> in prange(m): > ? ?for j in prange(n): > ? ? ? ?sum += i * j > ? ?sum += 0 >
>> > which seems a bit silly ?. Of course, we could just disable nested >
>> parallelism, or tell the users to use a prange and a ?for from? in such >
>> cases. > """ > > Dag: Interesting. The first one is definitely the behaviour
>> we want, as long > as it doesn't cause unintended consequences. > > I don't
>> really think it will -- the important thing is that that the order > of loop
>> iteration evaluation must be unimportant. And that is still true > (for the
>> outer loop, as well as for the inner) in your first example. > > Question:
>> When you have nested pranges, what will happen is that two nested > OpenMP
>> parallel blocks are used, right? And do you know if there is complete >
>> freedom/"reentrancy" in that variables that are thread-private in an outer >
>> parallel block and be shared in an inner one, and vice versa? An
>> implementation may or may not support it, and if it is supported the
>> behaviour can be configured through omp_set_nested(). So we should consider
>> the case where it is supported and enabled. If you have a lastprivate or
>> reduction, and after the loop these are (reduced and) assigned to the
>> original variable. So if that happens inside a parallel construct which does
>> not declare the variable private to the construct, you actually have a race.
>> So e.g. the nested prange currently races in the outer parallel range. > If
>> so I'd think that this algorithm should work and feel natural: > > ?- In
>> each prange, for the purposes of variable private/shared/reduction >
>> inference, consider all internal "prange" just as if they had been "range";
>> > no special treatment. > > ?- Recurse to children pranges. Right, that is
>> most natural. Algorithmically, reductions and lastprivates (as those can
>> have races if placed in inner parallel constructs) propagate outwards
>> towards the outermost parallel block, or up to the first parallel with
>> block, or up to the first construct that already determined the sharing
>> attribute. e.g. with parallel: with parallel: for i in prange(n): for j in
>> prange(n): sum += i * j # sum is well-defined here # sum is undefined here
>> Here 'sum' is a reduction for the two innermost loops. 'sum' is not private
>> for the inner parallel with block, as a prange in a parallel with block is a
>> worksharing loop that binds to that parallel with block. However, the
>> outermost parallel with block declares sum (and i and j) private, so after
>> that block all those variables become undefined. However, in the outermost
>> parallel with block, sum will have to be initialized to 0 before anything
>> else, or be declared firstprivate, otherwise 'sum' is undefined to begin
>> with. Do you think declaring it firstprivate would be the way to go, or
>> should we make it private and issue a warning or perhaps even an error? > DS
>> >
>> ________________________________
>> > cython-devel mailing list > cython-devel at python.org >
>> > http://mail.python.org/mailman/listinfo/cython-devel >
>> ________________________________
>> cython-devel mailing list cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

From d.s.seljebotn at astro.uio.no  Mon Apr 18 16:41:47 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 18 Apr 2011 16:41:47 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
Message-ID: <62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>

Excellent! Sounds great! (as I won't have my laptop for some days I can't have a look yet but I will later)

You're right about (the current) buffers and the gil. A testcase explicitly for them would be good.

Firstprivate etc: i think it'd be nice myself, but it is probably better to take a break from it at this point so that we can think more about that and not do anything rash; perhaps open up a specific thread on them and ask for more general input. Perhaps you want to take a break or task-switch to something else (fused types?) until I can get around to review and merge what you have so far? You'll know best what works for you though. If you decide to implement explicit threadprivate variables because you've got the flow I certainly wom't object myself.


-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

mark florisson <markflorisson88 at gmail.com> wrote:

On 18 April 2011 13:06, mark florisson <markflorisson88 at gmail.com> wrote: > On 16 April 2011 18:42, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote: >> (Moving discussion from http://markflorisson.wordpress.com/, where Mark >> said:) > > Ok, sure, it was just an issue I was wondering about at that moment, > but it's a tricky issue, so thanks. > >> """ >> Started a new branch https://github.com/markflorisson88/cython/tree/openmp . >> >> Now the question is whether sharing attributes should be propagated >> outwards. e.g. if you do >> >> for i in prange(m): >>    for j in prange(n): >>        sum += i * j >> >> then ?sum? is a reduction for the inner parallel loop, but not for the outer >> one. So the user would currently have to rewrite this to >> >> for i in prange(m): >>    for j in prange(n): >>        sum += i * j >>    sum += 0 >> >> which seems a bit silly  . Of course, we could just disable nested >> parallelism, or tell the users to use a prange and a ?for
from? in such >> cases. >> """ >> >> Dag: Interesting. The first one is definitely the behaviour we want, as long >> as it doesn't cause unintended consequences. >> >> I don't really think it will -- the important thing is that that the order >> of loop iteration evaluation must be unimportant. And that is still true >> (for the outer loop, as well as for the inner) in your first example. >> >> Question: When you have nested pranges, what will happen is that two nested >> OpenMP parallel blocks are used, right? And do you know if there is complete >> freedom/"reentrancy" in that variables that are thread-private in an outer >> parallel block and be shared in an inner one, and vice versa? > > An implementation may or may not support it, and if it is supported > the behaviour can be configured through omp_set_nested(). So we should > consider the case where it is supported and enabled. > > If you have a lastprivate or reduction, and after the loop these are > (reduced and) assigned
to the original variable. So if that happens > inside a parallel construct which does not declare the variable > private to the construct, you actually have a race. So e.g. the nested > prange currently races in the outer parallel range. > >> If so I'd think that this algorithm should work and feel natural: >> >>  - In each prange, for the purposes of variable private/shared/reduction >> inference, consider all internal "prange" just as if they had been "range"; >> no special treatment. >> >>  - Recurse to children pranges. > > Right, that is most natural. Algorithmically, reductions and > lastprivates (as those can have races if placed in inner parallel > constructs) propagate outwards towards the outermost parallel block, > or up to the first parallel with block, or up to the first construct > that already determined the sharing attribute. > > e.g. > > with parallel: >     with parallel: >        for i in prange(n): >            for j in prange(n): >                sum += i * j > 
   # sum is well-defined here > # sum is undefined here > > Here 'sum' is a reduction for the two innermost loops. 'sum' is not > private for the inner parallel with block, as a prange in a parallel > with block is a worksharing loop that binds to that parallel with > block. However, the outermost parallel with block declares sum (and i > and j) private, so after that block all those variables become > undefined. > > However, in the outermost parallel with block, sum will have to be > initialized to 0 before anything else, or be declared firstprivate, > otherwise 'sum' is undefined to begin with. Do you think declaring it > firstprivate would be the way to go, or should we make it private and > issue a warning or perhaps even an error? > >> DS >>_____________________________________________
>> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > Everything seems to be working, although now the user has to be careful with nested parallel blocks as variables can be private there (and not firstprivate), i.e., the user has to do initialization at the right place (e.g. in the outermost parallel block that determines it private). I'm thinking of adding a warning, as the C compiler does. Two issues are remaining: 1) explicit declarations of firstprivates Do we still want those? 2) buffer auxiliary vars When unpacking numpy buffers and using typed numpy arrays, can reassignment or updates of a buffer-related variable ever occur in nogil code sections? I'm thinking this is not possible and therefore all buffer variables may be shared in parallel (for) sections?_____________________________________________
cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110418/1c2eb88d/attachment.html>

From markflorisson88 at gmail.com  Mon Apr 18 16:51:17 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 18 Apr 2011 16:51:17 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
Message-ID: <BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>

On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> Excellent! Sounds great! (as I won't have my laptop for some days I can't
> have a look yet but I will later)
>
> You're right about (the current) buffers and the gil. A testcase explicitly
> for them would be good.
>
> Firstprivate etc: i think it'd be nice myself, but it is probably better to
> take a break from it at this point so that we can think more about that and
> not do anything rash; perhaps open up a specific thread on them and ask for
> more general input. Perhaps you want to take a break or task-switch to
> something else (fused types?) until I can get around to review and merge
> what you have so far? You'll know best what works for you though. If you
> decide to implement explicit threadprivate variables because you've got the
> flow I certainly wom't object myself.
>
 Ok, cool, I'll move on :) I already included a test with a prange and
a numpy buffer with indexing.

From romain.py at gmail.com  Mon Apr 18 17:55:58 2011
From: romain.py at gmail.com (Romain Guillebert)
Date: Mon, 18 Apr 2011 16:55:58 +0100
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <4DAC020E.2010707@behnel.de>
Message-ID: <20110418155558.GA8935@ubuntu>

Hi

I investigated the code produced by Cython, and I see 3 cases which
should be handled :

* A pure python variable which has a value assigned (including None)
* A pure python variable which has no value assigned
* A C variable (we can't test if they are set of not)

The first and second one seem relatively easy to implement it can
reuse __pyx_string_tab. 
The code will call __Pyx_GetName on each element of the array, if it
returns NULL, don't put the variable in the dictionary, else put the
name and it's value in the dictionary.
For the last one, it will probably need the C name, C type and Python
name of each C variable, add the python name in the dictionary and wrap
the C value into a Python object.

However this will only provide read access to the globals. To offer
read/write access, making a proxy dictionary is probably the most
straightforward implementation (it can be rather inefficient though
since we need to do an if/else to compare the value we want to set with
every C globals).

Do you think I'm on the right way ?

Thanks
Romain

From markflorisson88 at gmail.com  Mon Apr 18 21:08:35 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 18 Apr 2011 21:08:35 +0200
Subject: [Cython] gilnanny
Message-ID: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>

Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
for every refnanny inc- and decref, so that it will issue a fatal
error whenever reference counting is done without the gil, to make
sure we never do any illegal things in nogil code blocks. While I'm at
it, I also want to add a pystate.pxd to the cpython includes.

From robertwb at math.washington.edu  Mon Apr 18 22:26:08 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 18 Apr 2011 13:26:08 -0700
Subject: [Cython] gilnanny
In-Reply-To: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>
References: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>
Message-ID: <BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>

On Mon, Apr 18, 2011 at 12:08 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
> for every refnanny inc- and decref, so that it will issue a fatal
> error whenever reference counting is done without the gil, to make
> sure we never do any illegal things in nogil code blocks.

Sounds like a good idea to me.

> While I'm at it, I also want to add a pystate.pxd to the cpython includes.

Sure, corresponding to pystate.h? Have you looked into how stable this
API is (e.g. Py2 vs. Py3)?

- Robert

From sturla at molden.no  Tue Apr 19 00:18:51 2011
From: sturla at molden.no (Sturla Molden)
Date: Tue, 19 Apr 2011 00:18:51 +0200
Subject: [Cython] gilnanny
In-Reply-To: <BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>
References: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>
	<BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>
Message-ID: <4DACB8CB.8030008@molden.no>

Den 18.04.2011 22:26, skrev Robert Bradshaw:
> On Mon, Apr 18, 2011 at 12:08 PM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
>> for every refnanny inc- and decref, so that it will issue a fatal
>> error whenever reference counting is done without the gil, to make
>> sure we never do any illegal things in nogil code blocks.
> Sounds like a good idea to me.
>

Have you ever considered to allow a "with gil:" statement? It seems this 
could be implemented using the simplified GIL API, i.e. the same way 
ctypes synchronizes callbacks to Python. Usecases would e.g. be 
computational code that sometimes needs to touch Python objects. E.g. 
append something to a list, slice a NumPy array, unbox a buffer into 
local scope, etc. A "with gil" statement could allow us to grab the GIL 
back for that.

Sturla

From stefan_ml at behnel.de  Tue Apr 19 07:08:13 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 07:08:13 +0200
Subject: [Cython] gilnanny
In-Reply-To: <4DACB8CB.8030008@molden.no>
References: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>	<BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>
	<4DACB8CB.8030008@molden.no>
Message-ID: <4DAD18BD.6060306@behnel.de>

Sturla Molden, 19.04.2011 00:18:
> Den 18.04.2011 22:26, skrev Robert Bradshaw:
>> On Mon, Apr 18, 2011 at 12:08 PM, mark florisson wrote:
>>> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
>>> for every refnanny inc- and decref, so that it will issue a fatal
>>> error whenever reference counting is done without the gil, to make
>>> sure we never do any illegal things in nogil code blocks.
>> Sounds like a good idea to me.
>>
>
> Have you ever considered to allow a "with gil:" statement?

Yes, that's what this is all about.

https://github.com/markflorisson88/cython/commits/master

Stefan

From stefan_ml at behnel.de  Tue Apr 19 07:17:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 07:17:45 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTikqLSsHiCqKT1ZH4zfTtsCQ0UYcDQ@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>	<4DAB5629.2020500@behnel.de>	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>	<4DABC4B7.8030904@behnel.de>
	<BANLkTikqLSsHiCqKT1ZH4zfTtsCQ0UYcDQ@mail.gmail.com>
Message-ID: <4DAD1AF9.5030608@behnel.de>

Vitja Makarov, 18.04.2011 15:19:
> There is one more interesting thing:
>
> def test_yield_inside_genexp():
>      """
>      >>>  o = test_yield_inside_genexp()
>      >>>  list(o)
>      [0, None, 1, None, 2, None]
>      """
>      return ((yield i) for i in range(3))

Impressively ugly.

I guess we should start testing these things with other Python 
implementations than CPython, in order to see if these corner cases are 
really being accepted and knowingly being implemented, or if it's just 
something that no-one has really cared about so far and that's actually 
worth discussing.

Stefan

From vitja.makarov at gmail.com  Tue Apr 19 07:49:19 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 19 Apr 2011 09:49:19 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DAD1AF9.5030608@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
	<BANLkTikqLSsHiCqKT1ZH4zfTtsCQ0UYcDQ@mail.gmail.com>
	<4DAD1AF9.5030608@behnel.de>
Message-ID: <BANLkTimy9m1qs3y1RJRYJYYkGZsEQK1u7A@mail.gmail.com>

2011/4/19 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.04.2011 15:19:
>>
>> There is one more interesting thing:
>>
>> def test_yield_inside_genexp():
>> ? ? """
>> ? ? >>> ?o = test_yield_inside_genexp()
>> ? ? >>> ?list(o)
>> ? ? [0, None, 1, None, 2, None]
>> ? ? """
>> ? ? return ((yield i) for i in range(3))
>
> Impressively ugly.
>
> I guess we should start testing these things with other Python
> implementations than CPython, in order to see if these corner cases are
> really being accepted and knowingly being implemented, or if it's just
> something that no-one has really cared about so far and that's actually
> worth discussing.
>

The case above is yield inside genexp and it works now.

Btw we've found one more funny thing this time with parser: 0lor(1)


-- 
vitja.

From stefan_ml at behnel.de  Tue Apr 19 07:59:04 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 07:59:04 +0200
Subject: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
In-Reply-To: <20110418155558.GA8935@ubuntu>
References: <20110418155558.GA8935@ubuntu>
Message-ID: <4DAD24A8.5060702@behnel.de>

Romain Guillebert, 18.04.2011 17:55:
> I investigated the code produced by Cython, and I see 3 cases which
> should be handled :
>
> * A pure python variable which has a value assigned (including None)
> * A pure python variable which has no value assigned
> * A C variable (we can't test if they are set of not)
>
> The first and second one seem relatively easy to implement it can
> reuse __pyx_string_tab.
> The code will call __Pyx_GetName on each element of the array, if it
> returns NULL, don't put the variable in the dictionary, else put the
> name and it's value in the dictionary.
> For the last one, it will probably need the C name, C type and Python
> name of each C variable, add the python name in the dictionary and wrap
> the C value into a Python object.
>
> However this will only provide read access to the globals. To offer
> read/write access, making a proxy dictionary is probably the most
> straightforward implementation (it can be rather inefficient though
> since we need to do an if/else to compare the value we want to set with
> every C globals).
>
> Do you think I'm on the right way ?

Thanks, Romain. Certainly sounds "good enough" to me.

Stefan

From markflorisson88 at gmail.com  Tue Apr 19 09:09:53 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 09:09:53 +0200
Subject: [Cython] gilnanny
In-Reply-To: <BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>
References: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>
	<BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>
Message-ID: <BANLkTimED_wcshQpn-E8GN5w1qm7hnq7vQ@mail.gmail.com>

On 18 April 2011 22:26, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Mon, Apr 18, 2011 at 12:08 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
>> for every refnanny inc- and decref, so that it will issue a fatal
>> error whenever reference counting is done without the gil, to make
>> sure we never do any illegal things in nogil code blocks.
>
> Sounds like a good idea to me.

Ok, cool :)

>> While I'm at it, I also want to add a pystate.pxd to the cpython includes.
>
> Sure, corresponding to pystate.h? Have you looked into how stable this
> API is (e.g. Py2 vs. Py3)?

I haven't looked at all the functions, but most of them are documented
in both Python 2 and Python 3. I'll be vigilant.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Apr 19 11:14:11 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 11:14:11 +0200
Subject: [Cython] Hudson account
Message-ID: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>

Can I get a Hudson account so I can setup my branches there? I can't
seem to login using my trac credentials.

From stefan_ml at behnel.de  Tue Apr 19 12:19:31 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 12:19:31 +0200
Subject: [Cython] Hudson account
In-Reply-To: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
Message-ID: <4DAD61B3.6090600@behnel.de>

mark florisson, 19.04.2011 11:14:
> Can I get a Hudson account so I can setup my branches there?

I've created one for you. Please tell me when you have changed your 
password, so that I can give you write access.

Please be careful not to break too much, and note that it's easier to work 
at the file system level if you want to do mass duplication/adaptation of 
jobs. If you tell me what you need, I can set it up for you.

It may actually be best to start from Viteks configured jobs, copy them and 
adapt their github URL and branch name.

Stefan

From markflorisson88 at gmail.com  Tue Apr 19 12:31:43 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 12:31:43 +0200
Subject: [Cython] Hudson account
In-Reply-To: <4DAD61B3.6090600@behnel.de>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
	<4DAD61B3.6090600@behnel.de>
Message-ID: <BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>

On 19 April 2011 12:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 19.04.2011 11:14:
>>
>> Can I get a Hudson account so I can setup my branches there?
>
> I've created one for you. Please tell me when you have changed your
> password, so that I can give you write access.

Done.

> Please be careful not to break too much, and note that it's easier to work
> at the file system level if you want to do mass duplication/adaptation of
> jobs. If you tell me what you need, I can set it up for you.

All I want is to run the master branch (it has the 'with gil'
statement') and the openmp branch. I also want the fusedtypes branch,
which I'll create in a minute or two.

> It may actually be best to start from Viteks configured jobs, copy them and
> adapt their github URL and branch name.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Apr 19 12:42:32 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 12:42:32 +0200
Subject: [Cython] Hudson account
In-Reply-To: <BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
	<4DAD61B3.6090600@behnel.de>
	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
Message-ID: <BANLkTimn4Qj6kARbP6AfmhrSWrAOfRw=6Q@mail.gmail.com>

On 19 April 2011 12:31, mark florisson <markflorisson88 at gmail.com> wrote:
> On 19 April 2011 12:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 19.04.2011 11:14:
>>>
>>> Can I get a Hudson account so I can setup my branches there?
>>
>> I've created one for you. Please tell me when you have changed your
>> password, so that I can give you write access.
>
> Done.
>
>> Please be careful not to break too much, and note that it's easier to work
>> at the file system level if you want to do mass duplication/adaptation of
>> jobs. If you tell me what you need, I can set it up for you.
>
> All I want is to run the master branch (it has the 'with gil'
> statement') and the openmp branch. I also want the fusedtypes branch,
> which I'll create in a minute or two.

Oh, as for the versions, I think at least 2.3, 2.7 and py3k would be a
good idea.
Does the hudson server have numpy installed?

>> It may actually be best to start from Viteks configured jobs, copy them and
>> adapt their github URL and branch name.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

From stefan_ml at behnel.de  Tue Apr 19 12:53:36 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 12:53:36 +0200
Subject: [Cython] Hudson account
In-Reply-To: <BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>	<4DAD61B3.6090600@behnel.de>
	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
Message-ID: <4DAD69B0.1060400@behnel.de>

mark florisson, 19.04.2011 12:31:
> On 19 April 2011 12:19, Stefan Behnel wrote:
>> mark florisson, 19.04.2011 11:14:
>>>
>>> Can I get a Hudson account so I can setup my branches there?
>>
>> I've created one for you. Please tell me when you have changed your
>> password, so that I can give you write access.
>
> Done.

Ok, you should have write rights now.


>> Please be careful not to break too much, and note that it's easier to work
>> at the file system level if you want to do mass duplication/adaptation of
>> jobs. If you tell me what you need, I can set it up for you.
>
> All I want is to run the master branch (it has the 'with gil'
> statement') and the openmp branch. I also want the fusedtypes branch,
> which I'll create in a minute or two.

That's three branches in total, plus the builds in three CPython versions 
(2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new 
jobs that will compete with the others.

I would expect that the three branches would rarely be changed all at the 
same time (except when you update them from the main branch), but that may 
still result in quite some additional delays due to the longer build queue.

Vitek currently has one set of build jobs for himself and can change the 
branches at need. Question to the others: should we continue to do this, or 
set up a full set of build jobs for each git branch we want to test?

Stefan

From markflorisson88 at gmail.com  Tue Apr 19 13:09:00 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 13:09:00 +0200
Subject: [Cython] Hudson account
In-Reply-To: <4DAD69B0.1060400@behnel.de>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
	<4DAD61B3.6090600@behnel.de>
	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
	<4DAD69B0.1060400@behnel.de>
Message-ID: <BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>

On 19 April 2011 12:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 19.04.2011 12:31:
>>
>> On 19 April 2011 12:19, Stefan Behnel wrote:
>>>
>>> mark florisson, 19.04.2011 11:14:
>>>>
>>>> Can I get a Hudson account so I can setup my branches there?
>>>
>>> I've created one for you. Please tell me when you have changed your
>>> password, so that I can give you write access.
>>
>> Done.
>
> Ok, you should have write rights now.
>
>
>>> Please be careful not to break too much, and note that it's easier to
>>> work
>>> at the file system level if you want to do mass duplication/adaptation of
>>> jobs. If you tell me what you need, I can set it up for you.
>>
>> All I want is to run the master branch (it has the 'with gil'
>> statement') and the openmp branch. I also want the fusedtypes branch,
>> which I'll create in a minute or two.
>
> That's three branches in total, plus the builds in three CPython versions
> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new
> jobs that will compete with the others.
>
> I would expect that the three branches would rarely be changed all at the
> same time (except when you update them from the main branch), but that may
> still result in quite some additional delays due to the longer build queue.

Switching the branches would be fine with me, so 3 Python versions x 1
branch x 2xC/C++ = 6 jobs. Is it necessary to separate C from C++
testing?

> Vitek currently has one set of build jobs for himself and can change the
> branches at need. Question to the others: should we continue to do this, or
> set up a full set of build jobs for each git branch we want to test?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Tue Apr 19 13:36:53 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 13:36:53 +0200
Subject: [Cython] Hudson account
In-Reply-To: <BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>	<4DAD61B3.6090600@behnel.de>	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>	<4DAD69B0.1060400@behnel.de>
	<BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>
Message-ID: <4DAD73D5.5040202@behnel.de>

mark florisson, 19.04.2011 13:09:
> On 19 April 2011 12:53, Stefan Behnel wrote:
>> mark florisson, 19.04.2011 12:31:
>>> All I want is to run the master branch (it has the 'with gil'
>>> statement') and the openmp branch. I also want the fusedtypes branch,
>>> which I'll create in a minute or two.
>>
>> That's three branches in total, plus the builds in three CPython versions
>> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new
>> jobs that will compete with the others.
>>
>> I would expect that the three branches would rarely be changed all at the
>> same time (except when you update them from the main branch), but that may
>> still result in quite some additional delays due to the longer build queue.
>
> Switching the branches would be fine with me, so 3 Python versions x 1
> branch x 2xC/C++ = 6 jobs.

10 actually (1+3+6), but only the (6) test jobs are long running.

The job tree first builds an sdist from github, then builds bdists from 
that in all CPython versions, then tests the result in the test jobs.


> Is it necessary to separate C from C++ testing?

Not necessary, it just gives quicker feedback, right after the first test 
job is finished. If you test both in one job, it just runs twice as long.

The problem is not so much the sheer number of jobs (we have several 
hundred Hudson jobs up at work). It's the time it takes to run the ones 
that get triggered, and especially long running jobs that fill up the pipeline.

If you can live with a longer response time (usually just a couple of 
minutes), I'd suggest merging the test jobs, so that only one of them fills 
up the pipeline per build, instead of two.

Stefan

From markflorisson88 at gmail.com  Tue Apr 19 13:46:17 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 13:46:17 +0200
Subject: [Cython] Hudson account
In-Reply-To: <4DAD73D5.5040202@behnel.de>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
	<4DAD61B3.6090600@behnel.de>
	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
	<4DAD69B0.1060400@behnel.de>
	<BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>
	<4DAD73D5.5040202@behnel.de>
Message-ID: <BANLkTimF97idyPHTyBKBcntDA5-bMr6KEA@mail.gmail.com>

On 19 April 2011 13:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 19.04.2011 13:09:
>>
>> On 19 April 2011 12:53, Stefan Behnel wrote:
>>>
>>> mark florisson, 19.04.2011 12:31:
>>>>
>>>> All I want is to run the master branch (it has the 'with gil'
>>>> statement') and the openmp branch. I also want the fusedtypes branch,
>>>> which I'll create in a minute or two.
>>>
>>> That's three branches in total, plus the builds in three CPython versions
>>> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30
>>> new
>>> jobs that will compete with the others.
>>>
>>> I would expect that the three branches would rarely be changed all at the
>>> same time (except when you update them from the main branch), but that
>>> may
>>> still result in quite some additional delays due to the longer build
>>> queue.
>>
>> Switching the branches would be fine with me, so 3 Python versions x 1
>> branch x 2xC/C++ = 6 jobs.
>
> 10 actually (1+3+6), but only the (6) test jobs are long running.
>
> The job tree first builds an sdist from github, then builds bdists from that
> in all CPython versions, then tests the result in the test jobs.
>
I see, ok.

>> Is it necessary to separate C from C++ testing?
>
> Not necessary, it just gives quicker feedback, right after the first test
> job is finished. If you test both in one job, it just runs twice as long.
>
> The problem is not so much the sheer number of jobs (we have several hundred
> Hudson jobs up at work). It's the time it takes to run the ones that get
> triggered, and especially long running jobs that fill up the pipeline.
>
> If you can live with a longer response time (usually just a couple of
> minutes), I'd suggest merging the test jobs, so that only one of them fills
> up the pipeline per build, instead of two.

Ok. So are you setting it up, or should I do it?

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Tue Apr 19 18:27:27 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Apr 2011 18:27:27 +0200
Subject: [Cython] Hudson account
In-Reply-To: <BANLkTimF97idyPHTyBKBcntDA5-bMr6KEA@mail.gmail.com>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>	<4DAD61B3.6090600@behnel.de>	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>	<4DAD69B0.1060400@behnel.de>	<BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>	<4DAD73D5.5040202@behnel.de>
	<BANLkTimF97idyPHTyBKBcntDA5-bMr6KEA@mail.gmail.com>
Message-ID: <4DADB7EF.5090703@behnel.de>

mark florisson, 19.04.2011 13:46:
> So are you setting it up

Done. You now have your first red Hudson jobs. ;)

Stefan

From markflorisson88 at gmail.com  Tue Apr 19 18:37:39 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 19 Apr 2011 18:37:39 +0200
Subject: [Cython] Hudson account
In-Reply-To: <4DADB7EF.5090703@behnel.de>
References: <BANLkTi=d2ng1b66hLqU=RcAN+D6Jh-y0Lw@mail.gmail.com>
	<4DAD61B3.6090600@behnel.de>
	<BANLkTin6ZrmdV6CVH3J9Z3AdG1SMV2StAw@mail.gmail.com>
	<4DAD69B0.1060400@behnel.de>
	<BANLkTi=2c31DWyAx6Tu9uc+68MZPQ7GL1g@mail.gmail.com>
	<4DAD73D5.5040202@behnel.de>
	<BANLkTimF97idyPHTyBKBcntDA5-bMr6KEA@mail.gmail.com>
	<4DADB7EF.5090703@behnel.de>
Message-ID: <BANLkTimnr_doX_+-G59OpsHUkCN8bcePVw@mail.gmail.com>

On 19 April 2011 18:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 19.04.2011 13:46:
>>
>> So are you setting it up
>
> Done. You now have your first red Hudson jobs. ;)

Thanks a lot for setting it up! Red means good right? :)

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From sturla at molden.no  Tue Apr 19 21:49:56 2011
From: sturla at molden.no (Sturla Molden)
Date: Tue, 19 Apr 2011 21:49:56 +0200
Subject: [Cython] gilnanny
In-Reply-To: <4DAD18BD.6060306@behnel.de>
References: <BANLkTiknMBnmHDc0aujdgsLYFkC992NF3w@mail.gmail.com>	<BANLkTinxz8RBx_3MLNAygG-JNAchsfmH1g@mail.gmail.com>	<4DACB8CB.8030008@molden.no>
	<4DAD18BD.6060306@behnel.de>
Message-ID: <4DADE764.90905@molden.no>

Den 19.04.2011 07:08, skrev Stefan Behnel:
>
> Yes, that's what this is all about.
>
> https://github.com/markflorisson88/cython/commits/master

Great :)

Sturla

From bpederse at gmail.com  Tue Apr 19 22:45:55 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Tue, 19 Apr 2011 14:45:55 -0600
Subject: [Cython] libcpp.string operators
Message-ID: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>

hi, i have been using a stub for the c++ <string> in a lot of my work.
i decided to have a go at doing a proper string.pxd, pasted here:

https://gist.github.com/929604

other than the operators, it's mostly implemented with tests. but when
i try the operators
with this test:

def test_equal_operator(char *a, char *b):
    """
    >>> test_equal_operator("asdf", "asdf")
    True
    """
    cdef string s = string(a)
    cdef string t = string(b)
    cdef bint same = t == s
    return same

and this declaration in the pxd:

    bint operator==(string&, string&)

i get the error below. any ideas what i'm doing wrong?
thanks,
-brent





=== Got errors: ===
152:23: Invalid types for '==' (string, string)


======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running cpp_stl_string
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 569, in run
    self.runCompileTest()
  File "runtests.py", line 400, in runCompileTest
    self.test_directory, self.expect_errors, self.annotate)
  File "runtests.py", line 546, in compile
    self.assertEquals(None, unexpected_error)
AssertionError: None != u"152:23: Invalid types for '==' (string, string)"

From bpederse at gmail.com  Wed Apr 20 02:08:15 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Tue, 19 Apr 2011 18:08:15 -0600
Subject: [Cython] libcpp.string operators
In-Reply-To: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
Message-ID: <BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>

On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen <bpederse at gmail.com> wrote:
> hi, i have been using a stub for the c++ <string> in a lot of my work.
> i decided to have a go at doing a proper string.pxd, pasted here:
>
> https://gist.github.com/929604
>
> other than the operators, it's mostly implemented with tests. but when
> i try the operators
> with this test:
>
> def test_equal_operator(char *a, char *b):
> ? ?"""
> ? ?>>> test_equal_operator("asdf", "asdf")
> ? ?True
> ? ?"""
> ? ?cdef string s = string(a)
> ? ?cdef string t = string(b)
> ? ?cdef bint same = t == s
> ? ?return same
>
> and this declaration in the pxd:
>
> ? ?bint operator==(string&, string&)


it seems:

        bint operator==(string&)

is the correct syntax. i had copied the stuff from vector.pxd

>
> i get the error below. any ideas what i'm doing wrong?
> thanks,
> -brent
>
>
>
>
>
> === Got errors: ===
> 152:23: Invalid types for '==' (string, string)
>
>
> ======================================================================
> ERROR: runTest (__main__.CythonRunTestCase)
> compiling (cpp) and running cpp_stl_string
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 569, in run
> ? ?self.runCompileTest()
> ?File "runtests.py", line 400, in runCompileTest
> ? ?self.test_directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 546, in compile
> ? ?self.assertEquals(None, unexpected_error)
> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>

From bpederse at gmail.com  Wed Apr 20 02:22:14 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Tue, 19 Apr 2011 18:22:14 -0600
Subject: [Cython] libcpp.string operators
In-Reply-To: <BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>
Message-ID: <BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>

On Tue, Apr 19, 2011 at 6:08 PM, Brent Pedersen <bpederse at gmail.com> wrote:
> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen <bpederse at gmail.com> wrote:
>> hi, i have been using a stub for the c++ <string> in a lot of my work.
>> i decided to have a go at doing a proper string.pxd, pasted here:
>>
>> https://gist.github.com/929604
>>
>> other than the operators, it's mostly implemented with tests. but when
>> i try the operators
>> with this test:
>>
>> def test_equal_operator(char *a, char *b):
>> ? ?"""
>> ? ?>>> test_equal_operator("asdf", "asdf")
>> ? ?True
>> ? ?"""
>> ? ?cdef string s = string(a)
>> ? ?cdef string t = string(b)
>> ? ?cdef bint same = t == s
>> ? ?return same
>>
>> and this declaration in the pxd:
>>
>> ? ?bint operator==(string&, string&)
>
>
> it seems:
>
> ? ? ? ?bint operator==(string&)
>
> is the correct syntax. i had copied the stuff from vector.pxd
>
>>
>> i get the error below. any ideas what i'm doing wrong?
>> thanks,
>> -brent
>>
>>
>>
>>
>>
>> === Got errors: ===
>> 152:23: Invalid types for '==' (string, string)
>>
>>
>> ======================================================================
>> ERROR: runTest (__main__.CythonRunTestCase)
>> compiling (cpp) and running cpp_stl_string
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ?File "runtests.py", line 569, in run
>> ? ?self.runCompileTest()
>> ?File "runtests.py", line 400, in runCompileTest
>> ? ?self.test_directory, self.expect_errors, self.annotate)
>> ?File "runtests.py", line 546, in compile
>> ? ?self.assertEquals(None, unexpected_error)
>> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>>
>

i updated the gist with the operators, didn't do iterators.
https://gist.github.com/929604

From arthurdesribeiro at gmail.com  Wed Apr 20 05:04:25 2011
From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro)
Date: Wed, 20 Apr 2011 00:04:25 -0300
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <4DAB5410.4050404@behnel.de>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
	<4DA7E653.4060801@behnel.de>
	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
	<4DAB5410.4050404@behnel.de>
Message-ID: <BANLkTikBE8MazDBHb-37-V5gR3jatvE_Kw@mail.gmail.com>

2011/4/17 Stefan Behnel <stefan_ml at behnel.de>

> Arthur de Souza Ribeiro, 17.04.2011 20:07:
>
>  Hi Stefan, about your first comment : "And it's better to let Cython know
>> that this name refers to a function."  in line 69 of encoder.pyx file I
>> didn't understand well what does that mean, can you explain more this
>> comment?
>>
>
> Hmm, sorry, I think that was not so important. That code line is only used
> to override the Python implementation with the implementation from the
> external C accelerator module. To do that, it assigns either of the two
> functions to a name. So, when that name is called in the code, Cython cannot
> know that it actually is a function, and has to resort to Python calling,
> whereas a visible c(p)def function that is defined inside of the same module
> could be called faster.
>
> I missed the fact that this name isn't really used inside of the module, so
> whether Cython knows that it's a function or not isn't really all that
> important.
>

So, I don't have to be worried about this, right?


>
> I added another comment to this commit, though:
>
>
> https://github.com/arthursribeiro/JSON-module/commit/e2d80e0aeab6d39ff2d9b847843423ebdb9c57b7#diff-4
>
>
>
I saw your comment and what I understood of it is that the alias that are
being attributed to the type names make code slower, I tried to compile in
cython the same way that it was in python, but, there is something wrong
with it. It says:

Error compiling Cython file:
------------------------------------------------------------
...
def _make_iterencode(dict markers, _default, _encoder, _indent, _floatstr,
        _key_separator, _item_separator, bint _sort_keys, bint _skipkeys,
bint _one_shot,
        ## HACK: hand-optimized bytecode; turn globals into locals
        ValueError=ValueError,
        dict=dict,
        float=float,
            ^
------------------------------------------------------------

encoder.pyx:273:13: Empty declarator

I turned that way because I think the user can maybe change what types are
going to be used and cython do not allow do these things like python. (for
reserved words)


>
>  About the other comments, I think I solved them all, any problem with them
>> or other ones, please tell me. I'll try to fix.
>>
>
> It looks like you fixed a good deal of them.
>
> I actually tried to work with your code, but I'm not sure how you are
> building it. Could you give me a hint on that?
>

I'm manually building them using setup.py files, for every module I create
one and build manually, I don't think that's the best way to do it, but, to
test things, that's the way I'm doing.


>
> Where did you actually take the original code from? Python 3.2? Or from
> Python's hg branch?
>
>
I took the original code from Python 3.2


>
>
>  Note that it's not obvious from your initial commit what you actually
>>> changed. It would have been better to import the original file first,
>>> rename
>>> it to .pyx, and then commit your changes.
>>>
>>
>> I created a directory named 'Diff files' where I put the files generated
>> by
>> 'diff' command that i run in my computer, if you think it still be better
>> if
>> I commit and then change, there is no problem for me...
>>
>
> Diff only gives you the final outcome. Committing on top of the original
> files has the advantage of making the incremental changes visible
> separately. That makes it clearer what you tried, and a good commit comment
> will then make it clear why you did it.
>
>
>
>  I think it's more important to get some performance
>>> numbers to see how your module behaves compared to the C accelerator
>>> module
>>> (_json.c). I think the best approach to this project would actually be to
>>> start with profiling the Python implementation to see where performance
>>> problems occur (or to look through _json.c to see what the CPython
>>> developers considered performance critical), and then put the focus on
>>> trying to speed up only those parts of the Python implementation, by
>>> adding
>>> static types and potentially even rewriting them in a way that Cython can
>>> optimise them better.
>>>
>>
>> I've profilled the module I created and the module that is in Python 3.2,
>> the result is that the cython module spent about 73% less time then
>> python's
>>
>
> That's a common mistake when profiling: the actual time it takes to run is
> not meaningful. Depending on how far the two profiled programs differ, they
> may interact with the profiler in more or less intensive ways (as is clearly
> the case here), so the total time it takes for the programs to run can
> differ quite heavily under profiling, even if the non-profiled programs run
> at exactly the same speed.
>
> Also, I don't think you have enabled profiling for the Cython code. You can
> do that by passing the "profile=True" directive to the compiler, or by
> putting it at the top of the source files. That will add module-inner
> function calls to the profiling output. Note, however, that enabling
> profiling will slow down the execution, so disable it when you measure
> absolute run times.
>
> http://docs.cython.org/src/tutorial/profiling_tutorial.html
>
>
As you said, I didn't enable profiling for Cython code, I did it and got a
bigger number of function calls if compared to the old ones. I added a new
test case for list object and profiled the code, as you said, They differ
exactly by the number of calls to isinstance function, the result stayed
like this:

------------------------------------ USING Profiler
------------------------------------
JSONModule nested_dict
Tue Apr 19 23:16:48 2011    Profile.prof

         200003 function calls in 0.964 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.964    0.964 {built-in method exec}
    50000    0.114    0.000    0.804    0.000 __init__.pyx:179(dumps)
    50000    0.217    0.000    0.690    0.000 encoder.pyx:193(encode)
    50000    0.473    0.000    0.473    0.000 encoder.pyx:214(iterencode)
    50000    0.089    0.000    0.893    0.000 {JSONModule.dumps}
        1    0.071    0.071    0.964    0.964 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


json nested_dict
Tue Apr 19 23:16:49 2011    Profile.prof

         300003 function calls in 1.350 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.350    1.350 {built-in method exec}
    50000    0.157    0.000    1.255    0.000 __init__.py:180(dumps)
    50000    0.115    0.000    0.115    0.000 {method 'join' of 'str'
objects}
    50000    0.558    0.000    0.558    0.000 encoder.py:193(iterencode)
    50000    0.317    0.000    1.099    0.000 encoder.py:172(encode)
        1    0.094    0.094    1.350    1.350 <string>:1(<module>)
   100000    0.108    0.000    0.108    0.000 {built-in method isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


JSONModule ustring
Tue Apr 19 23:16:49 2011    Profile.prof

         150003 function calls in 0.297 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.297    0.297 {built-in method exec}
    50000    0.099    0.000    0.160    0.000 __init__.pyx:179(dumps)
    50000    0.061    0.000    0.061    0.000 encoder.pyx:193(encode)
    50000    0.082    0.000    0.242    0.000 {JSONModule.dumps}
        1    0.055    0.055    0.297    0.297 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


json ustring
Tue Apr 19 23:16:50 2011    Profile.prof

         200003 function calls in 0.419 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.419    0.419 {built-in method exec}
    50000    0.118    0.000    0.346    0.000 __init__.py:180(dumps)
    50000    0.052    0.000    0.052    0.000 {built-in method
encode_basestring_ascii}
    50000    0.138    0.000    0.228    0.000 encoder.py:172(encode)
        1    0.072    0.072    0.419    0.419 <string>:1(<module>)
    50000    0.038    0.000    0.038    0.000 {built-in method isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


JSONModule xlist
Tue Apr 19 23:16:50 2011    Profile.prof

         200003 function calls in 0.651 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.651    0.651 {built-in method exec}
    50000    0.108    0.000    0.500    0.000 __init__.pyx:179(dumps)
    50000    0.154    0.000    0.392    0.000 encoder.pyx:193(encode)
    50000    0.238    0.000    0.238    0.000 encoder.pyx:214(iterencode)
    50000    0.086    0.000    0.585    0.000 {JSONModule.dumps}
        1    0.065    0.065    0.651    0.651 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


json xlist
Tue Apr 19 23:16:51 2011    Profile.prof

         300003 function calls in 1.029 seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.029    1.029 {built-in method exec}
    50000    0.145    0.000    0.940    0.000 __init__.py:180(dumps)
    50000    0.062    0.000    0.062    0.000 {method 'join' of 'str'
objects}
    50000    0.323    0.000    0.323    0.000 encoder.py:193(iterencode)
    50000    0.304    0.000    0.795    0.000 encoder.py:172(encode)
        1    0.089    0.089    1.029    1.029 <string>:1(<module>)
   100000    0.106    0.000    0.106    0.000 {built-in method isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}


----------------------------------------------------------------------------------------



>
>
>  (blue for cython, red for python):
>>
>
> Colours tend to pass rather badly through mailing lists. Many people
> disable the HTML presentation of e-mails, and plain text does not have
> colours. But it was still obvious enough what you meant.
>
>
Sorry about this.


>
>
> Thank you for the numbers. Could you add absolute timings using timeit? And
> maybe also try with larger input data?
>

Using timer, I got the following output:

------------------------------------ USING Timeit
--------------------------------------
JSONModule nested_dict spent 11.39 usec/pass (cython)
JSONModule ustring spent 0.94 usec/pass (cython)
JSONModule xlist spent 5.71 usec/pass (cython)

json nested_dict spent 16.61 usec/pass
json ustring spent 1.88 usec/pass
json xlist spent 10.38 usec/pass
----------------------------------------------------------------------------------------

The testcases are the same of the profile ones.


>
> ISTM that a lot of overhead comes from calls that Cython can easily
> optimise all by itself: isinstance() and (bytes|unicode).join(). That's the
> kind of observation that previously let me suggest to start by benchmarking
> and profiling in the first place. Cython compiled code has quite different
> performance characteristics from code executing in CPython's interpreter, so
> it's important to start by getting an idea of how the code behaves when
> compiled, and then optimising it in the places where it still needs to run
> faster.
>
>
As you said, starting profiling is a better approach, specially because
every change made reflects in time changes (using timeit of profiling).


> Optimisation is an incremental process, and you will often end up reverting
> changes along the way when you see that they did not improve the
> performance, or maybe just made it so slightly faster that the speed
> improvement is not worth the code degradation of the optimisation change in
> question.
>
> Could you try to come up with a short list of important code changes you
> made that let this module run faster, backed by some timings that show the
> difference with and without each change?
>

To let this module run faster, the bests changes were in classes definitions
(I'm going to show numbers soon) using cinit and defining the variables made
the code faster. Another changes that I made were in for loops. I created an
int variable and made to loop in range of a number instead of a 'for x in
...' statement. The other ones were about to add static types specially to
int and boolean types.


>
> Stefan
>


Thank you very much again.

Best Regards.

[]s

Arthur
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110420/9170737a/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Wed Apr 20 05:34:18 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 20 Apr 2011 15:34:18 +1200
Subject: [Cython] GSoC Proposal - Reimplement C modules in CPython's
 standard library in Cython.
In-Reply-To: <BANLkTikBE8MazDBHb-37-V5gR3jatvE_Kw@mail.gmail.com>
References: <BANLkTikpCWXNZLhRr3gtnzyGcGTHpABLVg@mail.gmail.com>
	<BANLkTi=qRV8pC5=ALCzSr-dVpryzbpWFpQ@mail.gmail.com>
	<BANLkTimt8x_VdtFFkzrqhaZJ=-zQqox8sg@mail.gmail.com>
	<BANLkTinxetOfK+w_JhhsRXUzj1W-i7MX_Q@mail.gmail.com>
	<4DA3F455.8070006@behnel.de>
	<BANLkTiktZA64rf8pnwvCKzfXBfdQFP=hAQ@mail.gmail.com>
	<4DA4984D.3020007@behnel.de>
	<BANLkTi=UAE5BWV+Bn3pVy0dgFmDMBJbBDA@mail.gmail.com>
	<4DA5D624.50609@behnel.de>
	<BANLkTimGbToEDAe0baSkum2J+_UjsTBoVw@mail.gmail.com>
	<4DA7E653.4060801@behnel.de>
	<BANLkTi=_fnYetBtwEuWE_WcKrnjXOanAPw@mail.gmail.com>
	<4DAB5410.4050404@behnel.de>
	<BANLkTikBE8MazDBHb-37-V5gR3jatvE_Kw@mail.gmail.com>
Message-ID: <4DAE543A.7080509@canterbury.ac.nz>

Arthur de Souza Ribeiro wrote:

> def _make_iterencode(dict markers, _default, _encoder, _indent, _floatstr,
>         _key_separator, _item_separator, bint _sort_keys, bint 
> _skipkeys, bint _one_shot,
>         ## HACK: hand-optimized bytecode; turn globals into locals
>         ValueError=ValueError,
>         dict=dict,
>         float=float,
>             ^
> ------------------------------------------------------------
> 
> encoder.pyx:273:13: Empty declarator

You may need to choose something other than 'float' for the local name to avoid
confusing the parser (it thinks you're about to declare a parameter of type
'float').

-- 
Greg

From vitja.makarov at gmail.com  Wed Apr 20 10:26:46 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 20 Apr 2011 12:26:46 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DABC4B7.8030904@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
Message-ID: <BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>

2011/4/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.04.2011 06:38:
>>
>> 2011/4/18 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 17.04.2011 17:57:
>>>>
>>>> 3. check_yield_in_exception()
>>>
>>> I added this because I found a failing pyregr test that uses it (testing
>>> the
>>> @contextmanager decorator).
>>>
>>>
>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>> yield is there no exception reset is called.
>>>>
>>>> I'm not sure how to fix this.
>>>
>>> I'm not completely sure either.
>>>
>>>
>>>> import sys
>>>>
>>>> def foo():
>>>> ? ? """
>>>> ? ? >>> ? ?list(foo())
>>>> ? ? [<type 'exceptions.ValueError'>, None]
>>>> ? ? """
>>>> ? ? try:
>>>> ? ? ? ? raise ValueError
>>>> ? ? except ValueError:
>>>> ? ? ? ? yield sys.exc_info()[0]
>>>> ? ? ? ? yield sys.exc_info()[0] # exc_info is lost here
>>>
>>> I think (!), the difference here is that CPython actually keeps the
>>> exception in the generator frame. We don't have a frame, so we have to
>>> emulate it using the closure class. I guess we'll have to store away the
>>> exception into the closure when we yield while an exception is being
>>> handled, and restore it afterwards. Note: this is not the exception that
>>> is
>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>> exception that *was* raised and is now being handled, i.e. the thread
>>> state
>>> fields without the "_cur", that are reflected by sys.exc_info().
>>
>> Interesting difference between py2 and py3:
>>
>> def foo():
>> ? ? try:
>> ? ? ? ? raise ValueError
>> ? ? except ValueError:
>> ? ? ? ? yield
>> ? ? ? ? raise
>> list(foo())
>>
>> ? File "xxx.py", line 7, in<module>
>> ? ? list(foo())
>> ? File "xxx.py", line 6, in foo
>> ? ? raise
>> TypeError: exceptions must be old-style classes or derived from
>> BaseException, not NoneType
>>
>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>> seems to be fixed in python3.
>
> Not surprising. The implementation is completely different in Py2 and Py3,
> both in CPython and in Cython. It's actually much simpler in Cython under
> Py3, due to better semantics and C-API support. That also implies that
> there's much less Cython can do wrong in that environment. ;-)
>
>
>> Btw exception info temps are already saved and restored between yields.
>
> Right, but the exc_info itself is not reset and recovered around the yield.
> As I said above, generators have their own lifetime frame in CPython, and
> exceptions don't leak from that. So, whenever it's the generator (or code
> called by it) that raises an exception, that must be kept local to the
> generator.
>

Please review:
https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858

I've added __Pyx_Generator_SwapExceptions() method and call it right
before resume switch and before return from yield. It swaps generator
exception state with thread local.


-- 
vitja.

From stefan_ml at behnel.de  Wed Apr 20 11:43:49 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 11:43:49 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>	<4DAB5629.2020500@behnel.de>	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>	<4DABC4B7.8030904@behnel.de>
	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>
Message-ID: <4DAEAAD5.5080908@behnel.de>

Vitja Makarov, 20.04.2011 10:26:
> 2011/4/18 Stefan Behnel:
>> Vitja Makarov, 18.04.2011 06:38:
>>>
>>> 2011/4/18 Stefan Behnel:
>>>>
>>>> Vitja Makarov, 17.04.2011 17:57:
>>>>>
>>>>> 3. check_yield_in_exception()
>>>>
>>>> I added this because I found a failing pyregr test that uses it (testing
>>>> the
>>>> @contextmanager decorator).
>>>>
>>>>
>>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>>> yield is there no exception reset is called.
>>>>>
>>>>> I'm not sure how to fix this.
>>>>
>>>> I'm not completely sure either.
>>>>
>>>>
>>>>> import sys
>>>>>
>>>>> def foo():
>>>>>      """
>>>>>      >>>      list(foo())
>>>>>      [<type 'exceptions.ValueError'>, None]
>>>>>      """
>>>>>      try:
>>>>>          raise ValueError
>>>>>      except ValueError:
>>>>>          yield sys.exc_info()[0]
>>>>>          yield sys.exc_info()[0] # exc_info is lost here
>>>>
>>>> I think (!), the difference here is that CPython actually keeps the
>>>> exception in the generator frame. We don't have a frame, so we have to
>>>> emulate it using the closure class. I guess we'll have to store away the
>>>> exception into the closure when we yield while an exception is being
>>>> handled, and restore it afterwards. Note: this is not the exception that
>>>> is
>>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>>> exception that *was* raised and is now being handled, i.e. the thread
>>>> state
>>>> fields without the "_cur", that are reflected by sys.exc_info().
>>>
>>> Interesting difference between py2 and py3:
>>>
>>> def foo():
>>>      try:
>>>          raise ValueError
>>>      except ValueError:
>>>          yield
>>>          raise
>>> list(foo())
>>>
>>>    File "xxx.py", line 7, in<module>
>>>      list(foo())
>>>    File "xxx.py", line 6, in foo
>>>      raise
>>> TypeError: exceptions must be old-style classes or derived from
>>> BaseException, not NoneType
>>>
>>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>>> seems to be fixed in python3.
>>
>> Not surprising. The implementation is completely different in Py2 and Py3,
>> both in CPython and in Cython. It's actually much simpler in Cython under
>> Py3, due to better semantics and C-API support. That also implies that
>> there's much less Cython can do wrong in that environment. ;-)
>>
>>
>>> Btw exception info temps are already saved and restored between yields.
>>
>> Right, but the exc_info itself is not reset and recovered around the yield.
>> As I said above, generators have their own lifetime frame in CPython, and
>> exceptions don't leak from that. So, whenever it's the generator (or code
>> called by it) that raises an exception, that must be kept local to the
>> generator.
>
> Please review:
> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>
> I've added __Pyx_Generator_SwapExceptions() method and call it right
> before resume switch and before return from yield. It swaps generator
> exception state with thread local.

Looks good to me. I assume this fixes the problem? Then please push it into 
mainline.

Stefan

From vitja.makarov at gmail.com  Wed Apr 20 11:50:53 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 20 Apr 2011 13:50:53 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DAEAAD5.5080908@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>
	<4DAEAAD5.5080908@behnel.de>
Message-ID: <BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>

2011/4/20 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 20.04.2011 10:26:
>>
>> 2011/4/18 Stefan Behnel:
>>>
>>> Vitja Makarov, 18.04.2011 06:38:
>>>>
>>>> 2011/4/18 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 17.04.2011 17:57:
>>>>>>
>>>>>> 3. check_yield_in_exception()
>>>>>
>>>>> I added this because I found a failing pyregr test that uses it
>>>>> (testing
>>>>> the
>>>>> @contextmanager decorator).
>>>>>
>>>>>
>>>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>>>> yield is there no exception reset is called.
>>>>>>
>>>>>> I'm not sure how to fix this.
>>>>>
>>>>> I'm not completely sure either.
>>>>>
>>>>>
>>>>>> import sys
>>>>>>
>>>>>> def foo():
>>>>>> ? ? """
>>>>>> ? ? >>> ? ? ?list(foo())
>>>>>> ? ? [<type 'exceptions.ValueError'>, None]
>>>>>> ? ? """
>>>>>> ? ? try:
>>>>>> ? ? ? ? raise ValueError
>>>>>> ? ? except ValueError:
>>>>>> ? ? ? ? yield sys.exc_info()[0]
>>>>>> ? ? ? ? yield sys.exc_info()[0] # exc_info is lost here
>>>>>
>>>>> I think (!), the difference here is that CPython actually keeps the
>>>>> exception in the generator frame. We don't have a frame, so we have to
>>>>> emulate it using the closure class. I guess we'll have to store away
>>>>> the
>>>>> exception into the closure when we yield while an exception is being
>>>>> handled, and restore it afterwards. Note: this is not the exception
>>>>> that
>>>>> is
>>>>> freshly *being* raised (the "_cur*" fields in the thread state), it's
>>>>> the
>>>>> exception that *was* raised and is now being handled, i.e. the thread
>>>>> state
>>>>> fields without the "_cur", that are reflected by sys.exc_info().
>>>>
>>>> Interesting difference between py2 and py3:
>>>>
>>>> def foo():
>>>> ? ? try:
>>>> ? ? ? ? raise ValueError
>>>> ? ? except ValueError:
>>>> ? ? ? ? yield
>>>> ? ? ? ? raise
>>>> list(foo())
>>>>
>>>> ? File "xxx.py", line 7, in<module>
>>>> ? ? list(foo())
>>>> ? File "xxx.py", line 6, in foo
>>>> ? ? raise
>>>> TypeError: exceptions must be old-style classes or derived from
>>>> BaseException, not NoneType
>>>>
>>>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>>>> seems to be fixed in python3.
>>>
>>> Not surprising. The implementation is completely different in Py2 and
>>> Py3,
>>> both in CPython and in Cython. It's actually much simpler in Cython under
>>> Py3, due to better semantics and C-API support. That also implies that
>>> there's much less Cython can do wrong in that environment. ;-)
>>>
>>>
>>>> Btw exception info temps are already saved and restored between yields.
>>>
>>> Right, but the exc_info itself is not reset and recovered around the
>>> yield.
>>> As I said above, generators have their own lifetime frame in CPython, and
>>> exceptions don't leak from that. So, whenever it's the generator (or code
>>> called by it) that raises an exception, that must be kept local to the
>>> generator.
>>
>> Please review:
>>
>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>
>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>> before resume switch and before return from yield. It swaps generator
>> exception state with thread local.
>
> Looks good to me. I assume this fixes the problem? Then please push it into
> mainline.
>

old pull request is still there ;)

https://github.com/cython/cython/pull/25

Does __Pyx_ExceptionReset() steal references to args, so they should
not be decrefed later?


-- 
vitja.

From stefan_ml at behnel.de  Wed Apr 20 12:16:31 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 12:16:31 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>	<4DAB5629.2020500@behnel.de>	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>	<4DABC4B7.8030904@behnel.de>	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>	<4DAEAAD5.5080908@behnel.de>
	<BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>
Message-ID: <4DAEB27F.1050404@behnel.de>

Vitja Makarov, 20.04.2011 11:50:
> 2011/4/20 Stefan Behnel:
>> Vitja Makarov, 20.04.2011 10:26:
>>> 2011/4/18 Stefan Behnel:
>>>> generators have their own lifetime frame in CPython, and
>>>> exceptions don't leak from that. So, whenever it's the generator (or code
>>>> called by it) that raises an exception, that must be kept local to the
>>>> generator.
>>>
>>> Please review:
>>>
>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>
>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>> before resume switch and before return from yield. It swaps generator
>>> exception state with thread local.
>>
>> Looks good to me. I assume this fixes the problem? Then please push it into
>> mainline.
>
> old pull request is still there ;)
>
> https://github.com/cython/cython/pull/25
>
> Does __Pyx_ExceptionReset() steal references to args, so they should
> not be decrefed later?

Hmm, good call. The refcounting looks correct over the two function calls, 
but it would be nicer if it could be avoided instead. We are really just 
swapping around pointers here, no refcounting is needed.

I think it's worth using a dedicated utility function for this.

Oh, and one more thing: what should be done when exiting the generator 
normally? Would that just raise GeneratorExit anyway, so that we can 
swallow any original outer exception? I think there might be a problem with 
Python 3, where the GeneratorExit should be raised in the context of the 
original exception. Worth testing how CPython behaves here...

Stefan

From vitja.makarov at gmail.com  Wed Apr 20 12:51:35 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 20 Apr 2011 14:51:35 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DAEB27F.1050404@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>
	<4DAEAAD5.5080908@behnel.de>
	<BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>
	<4DAEB27F.1050404@behnel.de>
Message-ID: <BANLkTikZg8vuPcyVNHbDq=VaFSk4VboyFg@mail.gmail.com>

2011/4/20 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 20.04.2011 11:50:
>>
>> 2011/4/20 Stefan Behnel:
>>>
>>> Vitja Makarov, 20.04.2011 10:26:
>>>>
>>>> 2011/4/18 Stefan Behnel:
>>>>>
>>>>> generators have their own lifetime frame in CPython, and
>>>>> exceptions don't leak from that. So, whenever it's the generator (or
>>>>> code
>>>>> called by it) that raises an exception, that must be kept local to the
>>>>> generator.
>>>>
>>>> Please review:
>>>>
>>>>
>>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>>
>>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>>> before resume switch and before return from yield. It swaps generator
>>>> exception state with thread local.
>>>
>>> Looks good to me. I assume this fixes the problem? Then please push it
>>> into
>>> mainline.
>>
>> old pull request is still there ;)
>>
>> https://github.com/cython/cython/pull/25
>>
>> Does __Pyx_ExceptionReset() steal references to args, so they should
>> not be decrefed later?
>
> Hmm, good call. The refcounting looks correct over the two function calls,
> but it would be nicer if it could be avoided instead. We are really just
> swapping around pointers here, no refcounting is needed.
>
> I think it's worth using a dedicated utility function for this.
>

Do you mean new utility call __Pyx_ExceptionSwap()?

> Oh, and one more thing: what should be done when exiting the generator
> normally? Would that just raise GeneratorExit anyway, so that we can swallow
> any original outer exception? I think there might be a problem with Python
> 3, where the GeneratorExit should be raised in the context of the original
> exception. Worth testing how CPython behaves here...
>


Right! It's better to wrap call to generator:

ExceptionSwap();
generator_body();
ExceptionSwap();

Ok, I'll take a look.

-- 
vitja.

From stefan_ml at behnel.de  Wed Apr 20 13:45:02 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 13:45:02 +0200
Subject: [Cython] Recent bugs in generators
In-Reply-To: <BANLkTikZg8vuPcyVNHbDq=VaFSk4VboyFg@mail.gmail.com>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>	<4DAB5629.2020500@behnel.de>	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>	<4DABC4B7.8030904@behnel.de>	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>	<4DAEAAD5.5080908@behnel.de>	<BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>	<4DAEB27F.1050404@behnel.de>
	<BANLkTikZg8vuPcyVNHbDq=VaFSk4VboyFg@mail.gmail.com>
Message-ID: <4DAEC73E.2030906@behnel.de>

Vitja Makarov, 20.04.2011 12:51:
> 2011/4/20 Stefan Behnel<stefan_ml at behnel.de>:
>> Vitja Makarov, 20.04.2011 11:50:
>>>
>>> 2011/4/20 Stefan Behnel:
>>>>
>>>> Vitja Makarov, 20.04.2011 10:26:
>>>>>
>>>>> 2011/4/18 Stefan Behnel:
>>>>>>
>>>>>> generators have their own lifetime frame in CPython, and
>>>>>> exceptions don't leak from that. So, whenever it's the generator (or
>>>>>> code
>>>>>> called by it) that raises an exception, that must be kept local to the
>>>>>> generator.
>>>>>
>>>>> Please review:
>>>>>
>>>>>
>>>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>>>
>>>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>>>> before resume switch and before return from yield. It swaps generator
>>>>> exception state with thread local.
>>>>
>>>> Looks good to me. I assume this fixes the problem? Then please push it
>>>> into
>>>> mainline.
>>>
>>> old pull request is still there ;)
>>>
>>> https://github.com/cython/cython/pull/25
>>>
>>> Does __Pyx_ExceptionReset() steal references to args, so they should
>>> not be decrefed later?
>>
>> Hmm, good call. The refcounting looks correct over the two function calls,
>> but it would be nicer if it could be avoided instead. We are really just
>> swapping around pointers here, no refcounting is needed.
>>
>> I think it's worth using a dedicated utility function for this.
>>
>
> Do you mean new utility call __Pyx_ExceptionSwap()?

Yes. I think it doesn't even have to be specific to the generator code. 
Just pass in "&gen->exc_type" etc.


>> Oh, and one more thing: what should be done when exiting the generator
>> normally? Would that just raise GeneratorExit anyway, so that we can swallow
>> any original outer exception? I think there might be a problem with Python
>> 3, where the GeneratorExit should be raised in the context of the original
>> exception. Worth testing how CPython behaves here...
>>
>
>
> Right! It's better to wrap call to generator:
>
> ExceptionSwap();
> generator_body();
> ExceptionSwap();

Good idea.


> Ok, I'll take a look.

Cool. Thanks!

Stefan

From vitja.makarov at gmail.com  Wed Apr 20 15:16:09 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 20 Apr 2011 17:16:09 +0400
Subject: [Cython] Recent bugs in generators
In-Reply-To: <4DAEC73E.2030906@behnel.de>
References: <BANLkTinCy+iQJoxVNU=K_pwEibpbiwgW2A@mail.gmail.com>
	<4DAB5629.2020500@behnel.de>
	<BANLkTi=k93qwUiWS+gswh22DAMJbR8fY0g@mail.gmail.com>
	<4DABC4B7.8030904@behnel.de>
	<BANLkTikp2wkocS07nuqCeDTXquB+psRiug@mail.gmail.com>
	<4DAEAAD5.5080908@behnel.de>
	<BANLkTim30+i+vTzqZwfi9ADnQ_KQX9hKGA@mail.gmail.com>
	<4DAEB27F.1050404@behnel.de>
	<BANLkTikZg8vuPcyVNHbDq=VaFSk4VboyFg@mail.gmail.com>
	<4DAEC73E.2030906@behnel.de>
Message-ID: <BANLkTiknFn28-B3DcUFj+boY1ri+Cv68LQ@mail.gmail.com>

2011/4/20 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 20.04.2011 12:51:
>>
>> 2011/4/20 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 20.04.2011 11:50:
>>>>
>>>> 2011/4/20 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 20.04.2011 10:26:
>>>>>>
>>>>>> 2011/4/18 Stefan Behnel:
>>>>>>>
>>>>>>> generators have their own lifetime frame in CPython, and
>>>>>>> exceptions don't leak from that. So, whenever it's the generator (or
>>>>>>> code
>>>>>>> called by it) that raises an exception, that must be kept local to
>>>>>>> the
>>>>>>> generator.
>>>>>>
>>>>>> Please review:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>>>>
>>>>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>>>>> before resume switch and before return from yield. It swaps generator
>>>>>> exception state with thread local.
>>>>>
>>>>> Looks good to me. I assume this fixes the problem? Then please push it
>>>>> into
>>>>> mainline.
>>>>
>>>> old pull request is still there ;)
>>>>
>>>> https://github.com/cython/cython/pull/25
>>>>
>>>> Does __Pyx_ExceptionReset() steal references to args, so they should
>>>> not be decrefed later?
>>>
>>> Hmm, good call. The refcounting looks correct over the two function
>>> calls,
>>> but it would be nicer if it could be avoided instead. We are really just
>>> swapping around pointers here, no refcounting is needed.
>>>
>>> I think it's worth using a dedicated utility function for this.
>>>
>>
>> Do you mean new utility call __Pyx_ExceptionSwap()?
>
> Yes. I think it doesn't even have to be specific to the generator code. Just
> pass in "&gen->exc_type" etc.
>
>
>>> Oh, and one more thing: what should be done when exiting the generator
>>> normally? Would that just raise GeneratorExit anyway, so that we can
>>> swallow
>>> any original outer exception? I think there might be a problem with
>>> Python
>>> 3, where the GeneratorExit should be raised in the context of the
>>> original
>>> exception. Worth testing how CPython behaves here...
>>>
>>
>>
>> Right! It's better to wrap call to generator:
>>
>> ExceptionSwap();
>> generator_body();
>> ExceptionSwap();
>
> Good idea.
>
>
>> Ok, I'll take a look.
>
> Cool. Thanks!
>

https://github.com/vitek/cython/compare/73014aaed1...01286645d0

Another one try ;)


-- 
vitja.

From dalcinl at gmail.com  Wed Apr 20 16:09:54 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 Apr 2011 11:09:54 -0300
Subject: [Cython] libcpp.string operators
In-Reply-To: <BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>
	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>
Message-ID: <BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>

On 19 April 2011 21:22, Brent Pedersen <bpederse at gmail.com> wrote:
> On Tue, Apr 19, 2011 at 6:08 PM, Brent Pedersen <bpederse at gmail.com> wrote:
>> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen <bpederse at gmail.com> wrote:
>>> hi, i have been using a stub for the c++ <string> in a lot of my work.
>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>
>>> https://gist.github.com/929604
>>>
>>> other than the operators, it's mostly implemented with tests. but when
>>> i try the operators
>>> with this test:
>>>
>>> def test_equal_operator(char *a, char *b):
>>> ? ?"""
>>> ? ?>>> test_equal_operator("asdf", "asdf")
>>> ? ?True
>>> ? ?"""
>>> ? ?cdef string s = string(a)
>>> ? ?cdef string t = string(b)
>>> ? ?cdef bint same = t == s
>>> ? ?return same
>>>
>>> and this declaration in the pxd:
>>>
>>> ? ?bint operator==(string&, string&)
>>
>>
>> it seems:
>>
>> ? ? ? ?bint operator==(string&)
>>
>> is the correct syntax. i had copied the stuff from vector.pxd
>>
>>>
>>> i get the error below. any ideas what i'm doing wrong?
>>> thanks,
>>> -brent
>>>
>>>
>>>
>>>
>>>
>>> === Got errors: ===
>>> 152:23: Invalid types for '==' (string, string)
>>>
>>>
>>> ======================================================================
>>> ERROR: runTest (__main__.CythonRunTestCase)
>>> compiling (cpp) and running cpp_stl_string
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> ?File "runtests.py", line 569, in run
>>> ? ?self.runCompileTest()
>>> ?File "runtests.py", line 400, in runCompileTest
>>> ? ?self.test_directory, self.expect_errors, self.annotate)
>>> ?File "runtests.py", line 546, in compile
>>> ? ?self.assertEquals(None, unexpected_error)
>>> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>>>
>>
>
> i updated the gist with the operators, didn't do iterators.
> https://gist.github.com/929604
>

Looks pretty good. Could you fork the devel repo, add the pxd and the
test at appropriate places  and make a pull request? This is just in
order to give you credits for your work.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Wed Apr 20 17:58:01 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 17:58:01 +0200
Subject: [Cython] libcpp.string operators
In-Reply-To: <BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>
	<BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>
Message-ID: <4DAF0289.2050306@behnel.de>

Lisandro Dalcin, 20.04.2011 16:09:
> On 19 April 2011 21:22, Brent Pedersen<bpederse at gmail.com>  wrote:
>> On Tue, Apr 19, 2011 at 6:08 PM, Brent Pedersen<bpederse at gmail.com>  wrote:
>>> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen<bpederse at gmail.com>  wrote:
>>>> hi, i have been using a stub for the c++<string>  in a lot of my work.
>>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>>
>>>> https://gist.github.com/929604
>>>>
>>>> other than the operators, it's mostly implemented with tests. but when
>>>> i try the operators
>>>> with this test:
>>>>
>>>> def test_equal_operator(char *a, char *b):
>>>>     """
>>>>     >>>  test_equal_operator("asdf", "asdf")
>>>>     True
>>>>     """
>>>>     cdef string s = string(a)
>>>>     cdef string t = string(b)
>>>>     cdef bint same = t == s
>>>>     return same
>>>>
>>>> and this declaration in the pxd:
>>>>
>>>>     bint operator==(string&, string&)
>>>
>>>
>>> it seems:
>>>
>>>         bint operator==(string&)
>>>
>>> is the correct syntax. i had copied the stuff from vector.pxd
>>>
>>>>
>>>> i get the error below. any ideas what i'm doing wrong?
>>>> thanks,
>>>> -brent
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> === Got errors: ===
>>>> 152:23: Invalid types for '==' (string, string)
>>>>
>>>>
>>>> ======================================================================
>>>> ERROR: runTest (__main__.CythonRunTestCase)
>>>> compiling (cpp) and running cpp_stl_string
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>>   File "runtests.py", line 569, in run
>>>>     self.runCompileTest()
>>>>   File "runtests.py", line 400, in runCompileTest
>>>>     self.test_directory, self.expect_errors, self.annotate)
>>>>   File "runtests.py", line 546, in compile
>>>>     self.assertEquals(None, unexpected_error)
>>>> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>>>>
>>>
>>
>> i updated the gist with the operators, didn't do iterators.
>> https://gist.github.com/929604
>>
>
> Looks pretty good. Could you fork the devel repo, add the pxd and the
> test at appropriate places  and make a pull request? This is just in
> order to give you credits for your work.

Looks like this wasn't tested with Python 3, there are 16 failing tests 
more now.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/952/

Stefan

From bpederse at gmail.com  Wed Apr 20 18:29:09 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 20 Apr 2011 10:29:09 -0600
Subject: [Cython] libcpp.string operators
In-Reply-To: <4DAF0289.2050306@behnel.de>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>
	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>
	<BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>
	<4DAF0289.2050306@behnel.de>
Message-ID: <BANLkTimn_OAhW2MaQ+JHf6UADpHALgPuyw@mail.gmail.com>

On Wed, Apr 20, 2011 at 9:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 20.04.2011 16:09:
>>
>> On 19 April 2011 21:22, Brent Pedersen<bpederse at gmail.com> ?wrote:
>>>
>>> On Tue, Apr 19, 2011 at 6:08 PM, Brent Pedersen<bpederse at gmail.com>
>>> ?wrote:
>>>>
>>>> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen<bpederse at gmail.com>
>>>> ?wrote:
>>>>>
>>>>> hi, i have been using a stub for the c++<string> ?in a lot of my work.
>>>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>>>
>>>>> https://gist.github.com/929604
>>>>>
>>>>> other than the operators, it's mostly implemented with tests. but when
>>>>> i try the operators
>>>>> with this test:
>>>>>
>>>>> def test_equal_operator(char *a, char *b):
>>>>> ? ?"""
>>>>> ? ?>>> ?test_equal_operator("asdf", "asdf")
>>>>> ? ?True
>>>>> ? ?"""
>>>>> ? ?cdef string s = string(a)
>>>>> ? ?cdef string t = string(b)
>>>>> ? ?cdef bint same = t == s
>>>>> ? ?return same
>>>>>
>>>>> and this declaration in the pxd:
>>>>>
>>>>> ? ?bint operator==(string&, string&)
>>>>
>>>>
>>>> it seems:
>>>>
>>>> ? ? ? ?bint operator==(string&)
>>>>
>>>> is the correct syntax. i had copied the stuff from vector.pxd
>>>>
>>>>>
>>>>> i get the error below. any ideas what i'm doing wrong?
>>>>> thanks,
>>>>> -brent
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> === Got errors: ===
>>>>> 152:23: Invalid types for '==' (string, string)
>>>>>
>>>>>
>>>>> ======================================================================
>>>>> ERROR: runTest (__main__.CythonRunTestCase)
>>>>> compiling (cpp) and running cpp_stl_string
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>> ?File "runtests.py", line 569, in run
>>>>> ? ?self.runCompileTest()
>>>>> ?File "runtests.py", line 400, in runCompileTest
>>>>> ? ?self.test_directory, self.expect_errors, self.annotate)
>>>>> ?File "runtests.py", line 546, in compile
>>>>> ? ?self.assertEquals(None, unexpected_error)
>>>>> AssertionError: None != u"152:23: Invalid types for '==' (string,
>>>>> string)"
>>>>>
>>>>
>>>
>>> i updated the gist with the operators, didn't do iterators.
>>> https://gist.github.com/929604
>>>
>>
>> Looks pretty good. Could you fork the devel repo, add the pxd and the
>> test at appropriate places ?and make a pull request? This is just in
>> order to give you credits for your work.
>
> Looks like this wasn't tested with Python 3, there are 16 failing tests more
> now.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/952/
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

is the solution to prefix the string literals with 'b'? e.g.

    >>> test_indexing(b"asdf")

or something else? that passes in python 2.6, 2.7 and 3.2 for me.

From stefan_ml at behnel.de  Wed Apr 20 18:30:19 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 18:30:19 +0200
Subject: [Cython] libcpp.string operators
In-Reply-To: <4DAF0289.2050306@behnel.de>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>	<BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>
	<4DAF0289.2050306@behnel.de>
Message-ID: <4DAF0A1B.1080306@behnel.de>

Stefan Behnel, 20.04.2011 17:58:
> Lisandro Dalcin, 20.04.2011 16:09:
>> On 19 April 2011 21:22, Brent Pedersen wrote:
>>>>> hi, i have been using a stub for the c++<string> in a lot of my work.
>>>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>>>
>>>>> https://gist.github.com/929604
>>
>> Looks pretty good. Could you fork the devel repo, add the pxd and the
>> test at appropriate places and make a pull request? This is just in
>> order to give you credits for your work.
>
> Looks like this wasn't tested with Python 3, there are 16 failing tests
> more now.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/952/

I fixed it up.

Stefan

From stefan_ml at behnel.de  Wed Apr 20 18:34:04 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Apr 2011 18:34:04 +0200
Subject: [Cython] libcpp.string operators
In-Reply-To: <BANLkTimn_OAhW2MaQ+JHf6UADpHALgPuyw@mail.gmail.com>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>	<BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>	<4DAF0289.2050306@behnel.de>
	<BANLkTimn_OAhW2MaQ+JHf6UADpHALgPuyw@mail.gmail.com>
Message-ID: <4DAF0AFC.6040709@behnel.de>

Brent Pedersen, 20.04.2011 18:29:
> On Wed, Apr 20, 2011 at 9:58 AM, Stefan Behnel wrote:
>> Lisandro Dalcin, 20.04.2011 16:09:
>>>>> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen wrote:
>>>>>>
>>>>>> hi, i have been using a stub for the c++<string>    in a lot of my work.
>>>>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>>>>
>>>>>> https://gist.github.com/929604
>>>>>>
>>> Looks pretty good. Could you fork the devel repo, add the pxd and the
>>> test at appropriate places  and make a pull request? This is just in
>>> order to give you credits for your work.
>>
>> Looks like this wasn't tested with Python 3, there are 16 failing tests more
>> now.
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/952/
>>
> is the solution to prefix the string literals with 'b'? e.g.
>
>      >>>  test_indexing(b"asdf")
>
> or something else? that passes in python 2.6, 2.7 and 3.2 for me.

... but not in Py2.3-2.5. No, you need to either use byte strings created 
in Cython code, or use a hack that works on both Py2 and Py3, such as 
'xyz'.encode('ASCII').

Stefan

From bpederse at gmail.com  Wed Apr 20 18:40:07 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 20 Apr 2011 10:40:07 -0600
Subject: [Cython] libcpp.string operators
In-Reply-To: <4DAF0AFC.6040709@behnel.de>
References: <BANLkTik6Bi+sED6jMKQYeB5QMrxvCDQmTA@mail.gmail.com>
	<BANLkTinDSyecoCskW7d=z=Wfus6G9WTk5Q@mail.gmail.com>
	<BANLkTi=ue1YZ-47-A5knaesw=T8J04VMzQ@mail.gmail.com>
	<BANLkTimsOtCJMA5jdBVcqeEaurKgLWb0nQ@mail.gmail.com>
	<4DAF0289.2050306@behnel.de>
	<BANLkTimn_OAhW2MaQ+JHf6UADpHALgPuyw@mail.gmail.com>
	<4DAF0AFC.6040709@behnel.de>
Message-ID: <BANLkTinTWEp-brGYOhFrKh0PT81Mha0MsA@mail.gmail.com>

On Wed, Apr 20, 2011 at 10:34 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Brent Pedersen, 20.04.2011 18:29:
>>
>> On Wed, Apr 20, 2011 at 9:58 AM, Stefan Behnel wrote:
>>>
>>> Lisandro Dalcin, 20.04.2011 16:09:
>>>>>>
>>>>>> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen wrote:
>>>>>>>
>>>>>>> hi, i have been using a stub for the c++<string> ? ?in a lot of my
>>>>>>> work.
>>>>>>> i decided to have a go at doing a proper string.pxd, pasted here:
>>>>>>>
>>>>>>> https://gist.github.com/929604
>>>>>>>
>>>> Looks pretty good. Could you fork the devel repo, add the pxd and the
>>>> test at appropriate places ?and make a pull request? This is just in
>>>> order to give you credits for your work.
>>>
>>> Looks like this wasn't tested with Python 3, there are 16 failing tests
>>> more
>>> now.
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/952/
>>>
>> is the solution to prefix the string literals with 'b'? e.g.
>>
>> ? ? >>> ?test_indexing(b"asdf")
>>
>> or something else? that passes in python 2.6, 2.7 and 3.2 for me.
>
> ... but not in Py2.3-2.5. No, you need to either use byte strings created in
> Cython code, or use a hack that works on both Py2 and Py3, such as
> 'xyz'.encode('ASCII').
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

good to know. thanks.

From robertwb at math.washington.edu  Wed Apr 20 21:08:20 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 Apr 2011 12:08:20 -0700
Subject: [Cython] On cdef extern from syntax
In-Reply-To: <BANLkTikjWU_Rs5wdhhWhm5cmd9VFUkTDdg@mail.gmail.com>
References: <BANLkTikjWU_Rs5wdhhWhm5cmd9VFUkTDdg@mail.gmail.com>
Message-ID: <BANLkTimTyrvDO77F4MLDAD-NLZ2m9MDsXg@mail.gmail.com>

On Wed, Apr 20, 2011 at 11:08 AM, Fabrizio Milo aka misto
<mistobaan at gmail.com> wrote:
> Hi,
>
> I was wondering if has been ever discussed to implement the cython
> cdef extern from syntax as a with statement:
>
> with cython.header_importer("") as cy:
> ? ? ? cy.ctypedef(" unsigned int uint8 ")
> ? ? ? cy.cfunc( " void * myfunc() )")
>
> or also
>
> with cython.header_importer("") as cy:
> ? ? cy("""
> ? ? ctypdef ?...
>
> ? ? """)
>
> Maybe there is already something like that ?

I don't think the with statement really makes sense here, as it is a
list of global declarations, and the with statement is all about
making something local to a block.

- Robert

From vitja.makarov at gmail.com  Thu Apr 21 08:57:31 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 21 Apr 2011 10:57:31 +0400
Subject: [Cython] Build module script
Message-ID: <BANLkTikazMS=YgBULn0BbRvD7bd=pcn7RQ@mail.gmail.com>

Now we have cythonrun build script, may be it's time to create script
for easy module building?

-- 
vitja.

From markflorisson88 at gmail.com  Thu Apr 21 10:08:48 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 10:08:48 +0200
Subject: [Cython] Build module script
In-Reply-To: <BANLkTikazMS=YgBULn0BbRvD7bd=pcn7RQ@mail.gmail.com>
References: <BANLkTikazMS=YgBULn0BbRvD7bd=pcn7RQ@mail.gmail.com>
Message-ID: <BANLkTi=4MT=qajSWBrnS2Rj2h6F9o=hBSA@mail.gmail.com>

On 21 April 2011 08:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Now we have cythonrun build script, may be it's time to create script
> for easy module building?
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

We have cythonize():
http://wiki.cython.org/enhancements/distutils_preprocessing . Is
something like that what you mean? Unfortunately I believe it doesn't
parse sys.argv (yet). Perhaps a recursive option would also be useful.

From vitja.makarov at gmail.com  Thu Apr 21 10:23:45 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 21 Apr 2011 12:23:45 +0400
Subject: [Cython] Build module script
In-Reply-To: <BANLkTi=4MT=qajSWBrnS2Rj2h6F9o=hBSA@mail.gmail.com>
References: <BANLkTikazMS=YgBULn0BbRvD7bd=pcn7RQ@mail.gmail.com>
	<BANLkTi=4MT=qajSWBrnS2Rj2h6F9o=hBSA@mail.gmail.com>
Message-ID: <BANLkTinrFuGBb8j9R3_snyVfOXNzJa-dZw@mail.gmail.com>

2011/4/21 mark florisson <markflorisson88 at gmail.com>:
> On 21 April 2011 08:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> Now we have cythonrun build script, may be it's time to create script
>> for easy module building?
>>
>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> We have cythonize():
> http://wiki.cython.org/enhancements/distutils_preprocessing . Is
> something like that what you mean? Unfortunately I believe it doesn't
> parse sys.argv (yet). Perhaps a recursive option would also be useful.

Not exactly cythonize is for distutils and friends.

I mean simple script that run cython compiler and then build shared
python module.
It's much better then manually run cython then gcc or write makefile for this.

-- 
vitja.

From robertwb at math.washington.edu  Thu Apr 21 10:37:29 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 Apr 2011 01:37:29 -0700
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
Message-ID: <BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>

On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>> have a look yet but I will later)
>>
>> You're right about (the current) buffers and the gil. A testcase explicitly
>> for them would be good.
>>
>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>> take a break from it at this point so that we can think more about that and
>> not do anything rash; perhaps open up a specific thread on them and ask for
>> more general input. Perhaps you want to take a break or task-switch to
>> something else (fused types?) until I can get around to review and merge
>> what you have so far? You'll know best what works for you though. If you
>> decide to implement explicit threadprivate variables because you've got the
>> flow I certainly wom't object myself.
>>
> ?Ok, cool, I'll move on :) I already included a test with a prange and
> a numpy buffer with indexing.

Wow, you're just plowing away at this. Very cool.

+1 to disallowing nested prange, that seems to get really messy with
little benefit.

In terms of the CEP, I'm still unconvinced that firstprivate is not
safe to infer, but lets leave the initial values undefined rather than
specifying them to be NaNs (we can do that as an implementation if you
want), which will give us flexibility to change later once we've had a
chance to play around with it.

The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
We also probably want some way of explicitly marking a variable as
shared and still be able to assign to/flush/sync it. Perhaps the
parallel context could be used for these declarations, i.e.

    with parallel(threadlocal=a, shared=(b,c)):
        ...

which would be considered an "expert" usecase.

For all the discussion of threadsavailable/threadid, the most common
usecase I see is for allocating a large shared buffer and partitioning
it. This seems better handled by allocating separate thread-local
buffers, no? I still like the context idea, but everything in a
parallel block before and after the loop(s) also seems like a natural
place to put any setup/teardown code (though the context has the
advantage that __exit__ is always called, even if exceptions are
raised, which makes cleanup a lot easier to handle).

- Robert

From markflorisson88 at gmail.com  Thu Apr 21 10:43:19 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 10:43:19 +0200
Subject: [Cython] Build module script
In-Reply-To: <BANLkTinrFuGBb8j9R3_snyVfOXNzJa-dZw@mail.gmail.com>
References: <BANLkTikazMS=YgBULn0BbRvD7bd=pcn7RQ@mail.gmail.com>
	<BANLkTi=4MT=qajSWBrnS2Rj2h6F9o=hBSA@mail.gmail.com>
	<BANLkTinrFuGBb8j9R3_snyVfOXNzJa-dZw@mail.gmail.com>
Message-ID: <BANLkTimAVtZ-_hY7bNZpzFLeZh20Pscw-w@mail.gmail.com>

On 21 April 2011 10:23, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/4/21 mark florisson <markflorisson88 at gmail.com>:
>> On 21 April 2011 08:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> Now we have cythonrun build script, may be it's time to create script
>>> for easy module building?
>>>
>>> --
>>> vitja.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> We have cythonize():
>> http://wiki.cython.org/enhancements/distutils_preprocessing . Is
>> something like that what you mean? Unfortunately I believe it doesn't
>> parse sys.argv (yet). Perhaps a recursive option would also be useful.
>
> Not exactly cythonize is for distutils and friends.
>
> I mean simple script that run cython compiler and then build shared
> python module.
> It's much better then manually run cython then gcc or write makefile for this.

Ah, I see. Yeah that might be convenient. Although there is always pyximport.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Thu Apr 21 10:59:25 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 10:59:25 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
Message-ID: <BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>

On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>> have a look yet but I will later)
>>>
>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>> for them would be good.
>>>
>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>> take a break from it at this point so that we can think more about that and
>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>> more general input. Perhaps you want to take a break or task-switch to
>>> something else (fused types?) until I can get around to review and merge
>>> what you have so far? You'll know best what works for you though. If you
>>> decide to implement explicit threadprivate variables because you've got the
>>> flow I certainly wom't object myself.
>>>
>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>> a numpy buffer with indexing.
>
> Wow, you're just plowing away at this. Very cool.
>
> +1 to disallowing nested prange, that seems to get really messy with
> little benefit.
>
> In terms of the CEP, I'm still unconvinced that firstprivate is not
> safe to infer, but lets leave the initial values undefined rather than
> specifying them to be NaNs (we can do that as an implementation if you
> want), which will give us flexibility to change later once we've had a
> chance to play around with it.

Yes, they are currently undefined (and not initialized to NaN etc).
The thing is that without the control flow analysis (or perhaps not
until runtime) you won't know whether a variable is initialized at all
before the parallel section, so making it firstprivate might actually
copy an undefined value (perhaps with a trap representation!) into the
thread-private copy, which might invalidate valid code. e.g. consider

x_is_initialized = False
if condition:
    x = 1
    x_is_initialized = True

for i in prange(10, schedule='static'):
    if x_is_initialized:
        printf("%d\n", x)
    x = i

> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
> We also probably want some way of explicitly marking a variable as
> shared and still be able to assign to/flush/sync it. Perhaps the
> parallel context could be used for these declarations, i.e.
>
> ? ?with parallel(threadlocal=a, shared=(b,c)):
> ? ? ? ?...
>
> which would be considered an "expert" usecase.

Indeed, assigning to elements in an array instead doesn't seem very
convenient :)

> For all the discussion of threadsavailable/threadid, the most common
> usecase I see is for allocating a large shared buffer and partitioning
> it. This seems better handled by allocating separate thread-local
> buffers, no? I still like the context idea, but everything in a
> parallel block before and after the loop(s) also seems like a natural
> place to put any setup/teardown code (though the context has the
> advantage that __exit__ is always called, even if exceptions are
> raised, which makes cleanup a lot easier to handle).

Currently 'with gil' isn't merged into that branch, and if it will, it
will be disallowed, as I'm not yet sure how (if at all) it could be
handled with regard to exceptions. It seems a lot easier to disallow
it and have the user write a 'with gil' function, from which nothing
can propagate.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Thu Apr 21 11:21:18 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 11:21:18 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
Message-ID: <BANLkTin=OoQagZ+hyH0GbxwQmHZBqXy1Xw@mail.gmail.com>

On 21 April 2011 10:59, mark florisson <markflorisson88 at gmail.com> wrote:
> On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>> take a break from it at this point so that we can think more about that and
>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got the
>>>> flow I certainly wom't object myself.
>>>>
>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
> ? ?x = 1
> ? ?x_is_initialized = True
>
> for i in prange(10, schedule='static'):
> ? ?if x_is_initialized:
> ? ? ? ?printf("%d\n", x)
> ? ?x = i

Erm, that snippet I posted is invalid in any case, as x will be
private. So guess initializing things to NaN in such would have to
occur in the parallel section that should enclose the for. So e.g.
we'd have to do

#pragma omp parallel private(x)
{
    x = INT_MAX;
    #pragma omp for lastprivate(i)
    for (...)
        ...
}

Which would then mean that 'x' cannot be lastprivate anymore :). So
it's either "uninitialized and undefined" or "firstprivate". I
personally prefer the former for the implicit route.

I do like the threadlocal=a stuff to parallel, it's basically what I
proposed a while back except that you don't make them strings, but
better because most of your variables can be inferred, so the
messiness is gone.

>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> ? ?with parallel(threadlocal=a, shared=(b,c)):
>> ? ? ? ?...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.
>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

From robertwb at math.washington.edu  Thu Apr 21 11:18:16 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 Apr 2011 02:18:16 -0700
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
Message-ID: <BANLkTi=43Se1ukB1f0ao7erHTu2C442H2Q@mail.gmail.com>

On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>> take a break from it at this point so that we can think more about that and
>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got the
>>>> flow I certainly wom't object myself.
>>>>
>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
> ? ?x = 1
> ? ?x_is_initialized = True
>
> for i in prange(10, schedule='static'):
> ? ?if x_is_initialized:
> ? ? ? ?printf("%d\n", x)
> ? ?x = i

I'm still failing to see how this is a problem (or anything new, as
opposed to this same example with an ordinary range).

>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> ? ?with parallel(threadlocal=a, shared=(b,c)):
>> ? ? ? ?...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.

Not being able to propagate exceptions is a pretty strong
constraint--even if the implementation doesn't yet support it, it'd be
nice to have an API that makes it possible as a future feature.

- Robert

From robertwb at math.washington.edu  Thu Apr 21 11:37:56 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 Apr 2011 02:37:56 -0700
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTin=OoQagZ+hyH0GbxwQmHZBqXy1Xw@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
	<BANLkTin=OoQagZ+hyH0GbxwQmHZBqXy1Xw@mail.gmail.com>
Message-ID: <BANLkTi=U1hBxsPpqK7qCTpYTXao94qUTPw@mail.gmail.com>

On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 21 April 2011 10:59, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>> have a look yet but I will later)
>>>>>
>>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>>> for them would be good.
>>>>>
>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>>> take a break from it at this point so that we can think more about that and
>>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>> something else (fused types?) until I can get around to review and merge
>>>>> what you have so far? You'll know best what works for you though. If you
>>>>> decide to implement explicit threadprivate variables because you've got the
>>>>> flow I certainly wom't object myself.
>>>>>
>>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>>> a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>> ? ?x = 1
>> ? ?x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>> ? ?if x_is_initialized:
>> ? ? ? ?printf("%d\n", x)
>> ? ?x = i
>
> Erm, that snippet I posted is invalid in any case, as x will be
> private. So guess initializing things to NaN in such would have to
> occur in the parallel section that should enclose the for. So e.g.
> we'd have to do
>
> #pragma omp parallel private(x)
> {
> ? ?x = INT_MAX;
> ? ?#pragma omp for lastprivate(i)
> ? ?for (...)
> ? ? ? ?...
> }
>
> Which would then mean that 'x' cannot be lastprivate anymore :). So
> it's either "uninitialized and undefined" or "firstprivate". I
> personally prefer the former for the implicit route.

A variable can't be both first and last private? In any case, as long
as we don't promise anything about them now, we can decide later.

> I do like the threadlocal=a stuff to parallel, it's basically what I
> proposed a while back except that you don't make them strings, but
> better because most of your variables can be inferred, so the
> messiness is gone.

Yep.

- Robert

From markflorisson88 at gmail.com  Thu Apr 21 11:46:16 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 11:46:16 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=U1hBxsPpqK7qCTpYTXao94qUTPw@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
	<BANLkTin=OoQagZ+hyH0GbxwQmHZBqXy1Xw@mail.gmail.com>
	<BANLkTi=U1hBxsPpqK7qCTpYTXao94qUTPw@mail.gmail.com>
Message-ID: <BANLkTin4MGrOBMTHqWeAdZ6d-S_eeVakCQ@mail.gmail.com>

On 21 April 2011 11:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 21 April 2011 10:59, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>>> have a look yet but I will later)
>>>>>>
>>>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>>>> for them would be good.
>>>>>>
>>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>>>> take a break from it at this point so that we can think more about that and
>>>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>>> something else (fused types?) until I can get around to review and merge
>>>>>> what you have so far? You'll know best what works for you though. If you
>>>>>> decide to implement explicit threadprivate variables because you've got the
>>>>>> flow I certainly wom't object myself.
>>>>>>
>>>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>>>> a numpy buffer with indexing.
>>>>
>>>> Wow, you're just plowing away at this. Very cool.
>>>>
>>>> +1 to disallowing nested prange, that seems to get really messy with
>>>> little benefit.
>>>>
>>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>>> safe to infer, but lets leave the initial values undefined rather than
>>>> specifying them to be NaNs (we can do that as an implementation if you
>>>> want), which will give us flexibility to change later once we've had a
>>>> chance to play around with it.
>>>
>>> Yes, they are currently undefined (and not initialized to NaN etc).
>>> The thing is that without the control flow analysis (or perhaps not
>>> until runtime) you won't know whether a variable is initialized at all
>>> before the parallel section, so making it firstprivate might actually
>>> copy an undefined value (perhaps with a trap representation!) into the
>>> thread-private copy, which might invalidate valid code. e.g. consider
>>>
>>> x_is_initialized = False
>>> if condition:
>>> ? ?x = 1
>>> ? ?x_is_initialized = True
>>>
>>> for i in prange(10, schedule='static'):
>>> ? ?if x_is_initialized:
>>> ? ? ? ?printf("%d\n", x)
>>> ? ?x = i
>>
>> Erm, that snippet I posted is invalid in any case, as x will be
>> private. So guess initializing things to NaN in such would have to
>> occur in the parallel section that should enclose the for. So e.g.
>> we'd have to do
>>
>> #pragma omp parallel private(x)
>> {
>> ? ?x = INT_MAX;
>> ? ?#pragma omp for lastprivate(i)
>> ? ?for (...)
>> ? ? ? ?...
>> }
>>
>> Which would then mean that 'x' cannot be lastprivate anymore :). So
>> it's either "uninitialized and undefined" or "firstprivate". I
>> personally prefer the former for the implicit route.
>
> A variable can't be both first and last private? In any case, as long
> as we don't promise anything about them now, we can decide later.

It can be, but not if the binding parallel region declares it private.
So we wouldn't actually need the snippet above, we could just do

x = INT_MAX;
#pragma omp parallel for firstprivate(x) lastprivate(i, x)
for (...)
    ...

Yeah, that would work.

>> I do like the threadlocal=a stuff to parallel, it's basically what I
>> proposed a while back except that you don't make them strings, but
>> better because most of your variables can be inferred, so the
>> messiness is gone.
>
> Yep.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Thu Apr 21 11:48:53 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 21 Apr 2011 11:48:53 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTi=43Se1ukB1f0ao7erHTu2C442H2Q@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<BANLkTi=-yE11wq2DsahsJ7=3J30p24CsjQ@mail.gmail.com>
	<BANLkTi=43Se1ukB1f0ao7erHTu2C442H2Q@mail.gmail.com>
Message-ID: <BANLkTimqFSXpvOK4qAMHe2nNJALfWtdDBQ@mail.gmail.com>

On 21 April 2011 11:18, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 21 April 2011 10:37, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>> have a look yet but I will later)
>>>>>
>>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>>> for them would be good.
>>>>>
>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>>> take a break from it at this point so that we can think more about that and
>>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>> something else (fused types?) until I can get around to review and merge
>>>>> what you have so far? You'll know best what works for you though. If you
>>>>> decide to implement explicit threadprivate variables because you've got the
>>>>> flow I certainly wom't object myself.
>>>>>
>>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>>> a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>> ? ?x = 1
>> ? ?x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>> ? ?if x_is_initialized:
>> ? ? ? ?printf("%d\n", x)
>> ? ?x = i
>
> I'm still failing to see how this is a problem (or anything new, as
> opposed to this same example with an ordinary range).
>>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>>> We also probably want some way of explicitly marking a variable as
>>> shared and still be able to assign to/flush/sync it. Perhaps the
>>> parallel context could be used for these declarations, i.e.
>>>
>>> ? ?with parallel(threadlocal=a, shared=(b,c)):
>>> ? ? ? ?...
>>>
>>> which would be considered an "expert" usecase.
>>
>> Indeed, assigning to elements in an array instead doesn't seem very
>> convenient :)
>>
>>> For all the discussion of threadsavailable/threadid, the most common
>>> usecase I see is for allocating a large shared buffer and partitioning
>>> it. This seems better handled by allocating separate thread-local
>>> buffers, no? I still like the context idea, but everything in a
>>> parallel block before and after the loop(s) also seems like a natural
>>> place to put any setup/teardown code (though the context has the
>>> advantage that __exit__ is always called, even if exceptions are
>>> raised, which makes cleanup a lot easier to handle).
>>
>> Currently 'with gil' isn't merged into that branch, and if it will, it
>> will be disallowed, as I'm not yet sure how (if at all) it could be
>> handled with regard to exceptions. It seems a lot easier to disallow
>> it and have the user write a 'with gil' function, from which nothing
>> can propagate.
>
> Not being able to propagate exceptions is a pretty strong
> constraint--even if the implementation doesn't yet support it, it'd be
> nice to have an API that makes it possible as a future feature.

It would be possible, with some modifications to try/finally. I think
it'd be best to stabilize and merge with gil first.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From cb at mit.edu  Thu Apr 21 15:59:49 2011
From: cb at mit.edu (Chuck Blake)
Date: Thu, 21 Apr 2011 09:59:49 -0400
Subject: [Cython] Build module script
In-Reply-To: <BANLkTimAVtZ-_hY7bNZpzFLeZh20Pscw-w@mail.gmail.com>
Message-ID: <20110421135949.GA7638@pdos.lcs.mit.edu>

>>> On 21 April 2011 08:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> I mean simple script that run cython compiler and then build shared
>> python module.
>> It's much better then manually run cython then gcc or write makefile for this.
>
>Ah, I see. Yeah that might be convenient. Although there is always pyximport.

I have a script I can contribute called pycc that might be a bit more
than "simple".

  pycc [-c] [pyrexc|cython opts] [-- C-compiler opts] <FILE.pyx> [Clink opts]

as in

  pycc -c foo.pyx

to produce foo.so in the simple case, or maybe, just as an example

  pycc -c -- -fmudflap foo.pyx -lmudflap

if you had gcc as a compiler and wanted to turn on mudflap.

The script itself is a Python script which uses an
    exec(file(os.environ["HOME"] + "/.pyccrc").read())
to configure things in the Python language itself for what C compiler and
what default compile flags, C & Python include paths, libraries, etc.

It only works on Unix right now.  I'm not sure how hard it would be
to adapt it to work on both Unix and Windows.

It actually also works for "programs" rather than "modules" if you
drop the -c as in

  pycc -- -I/usr/include/X11 whatever.pyx -lX11

to produce a --embed generated program called "whatever".  In all,
it makes for a pycc that behaves a bit more like traditional compiler
front ends such as 'gcc' that take source all the way to a variety
of outputs usable by the host OS.  It works well in traditional
Makefiles or just from the command line as one-off invocations as
you're doing initial writing and testing or even "just seeing"
if Cython works/helps in some isolated situation with some module
or script.

--

There's also some complexity that dates to pre-embed days, but
which I personally find useful (or Pyrex users might find useful
since I believe Pyrex does not have --embed).  If you do not have
the "-c" and you do have a ##MAIN## comment in the source file,
it will indent all code after ##MAIN## into a "Main()" function
called at startup.  If there's no special comment it falls back
to --embed.  (we could spell this some other way if anyone wants
more like the compiler option comment syntax "# cython: MAIN").

The upside of this feature is that with almost no mucking about
with your Python script other than adding a comment, it makes
variables local to Main() which allows optimizations in the main
script logic that potentially make things much faster.  I see 2X
speed-ups in a few of my own speed critical programs.  It probably
would speed up CPython interpretation, too, but not nearly as much
as the Cython operation in my various experiments.

The downside of the feature is that does fiddle with global/local
scope or mixed statements + defines + statments + defines.  Those
are all "very uncommon" patterns in my usage.  Also, it does have a
big ##MAIN## warning marker to let a reader know something special
might be going on semantics-wise.  And, hey, 2X speed for a 1-line
diff. :)

So, you can probably see why I keep that feature around, but I could
also see if people didn't want an auto-functionizing-of-Main feature.
Having it around also doesn't seem harmful if the spelling of the
comment marker is nigh impossible to occur by accident.  If you
wanted me to delete that from the script before submission, I could,
but it'd require a bit work and testing.  Let me know if you want it.

--

An entirely alternate proposal that might be better, but would also be
more involved would be to make the "cython" program grow functionality
to make it more like a traditional compiler front end.  As things stand,
"cython" (inheriting use-concept from Pyrex) works more like "cc -E"
where output needs to fed to another stage to be compiled/linked.

From d.s.seljebotn at astro.uio.no  Thu Apr 21 20:13:18 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 Apr 2011 20:13:18 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>	<4DA60013.6060901@astro.uio.no>	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>	<4DA73D22.9050605@astro.uio.no>	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>	<4DA743EF.7080200@astro.uio.no>	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>	<4DA74CEC.2000609@astro.uio.no>	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>	<4DA9C6DF.1060406@astro.uio.no>	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
Message-ID: <4DB073BE.7010909@astro.uio.no>

On 04/21/2011 10:37 AM, Robert Bradshaw wrote:
> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> On 18 April 2011 16:41, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>> have a look yet but I will later)
>>>
>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>> for them would be good.
>>>
>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>> take a break from it at this point so that we can think more about that and
>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>> more general input. Perhaps you want to take a break or task-switch to
>>> something else (fused types?) until I can get around to review and merge
>>> what you have so far? You'll know best what works for you though. If you
>>> decide to implement explicit threadprivate variables because you've got the
>>> flow I certainly wom't object myself.
>>>
>>   Ok, cool, I'll move on :) I already included a test with a prange and
>> a numpy buffer with indexing.
>
> Wow, you're just plowing away at this. Very cool.
>
> +1 to disallowing nested prange, that seems to get really messy with
> little benefit.
>
> In terms of the CEP, I'm still unconvinced that firstprivate is not
> safe to infer, but lets leave the initial values undefined rather than
> specifying them to be NaNs (we can do that as an implementation if you
> want), which will give us flexibility to change later once we've had a
> chance to play around with it.

I don't see any technical issues with inferring firstprivate, the 
question is whether we want to. I suggest not inferring it in order to 
make this safer: One should be able to just try to change a loop from 
"range" to "prange", and either a) have things fail very hard, or b) 
just work correctly and be able to trust the results.

Note that when I suggest using NaN, it is as initial values for EACH 
ITERATION, not per-thread initialization. It is not about "firstprivate" 
or not, but about disabling thread-private variables entirely in favor 
of "per-iteration" variables.

I believe that by talking about "readonly" and "per-iteration" 
variables, rather than "thread-shared" and "thread-private" variables, 
this can be used much more safely and with virtually no knowledge of the 
details of threading. Again, what's in my mind are scientific 
programmers with (too) little training.

In the end it's a matter of taste and what is most convenient to more 
users. But I believe the case of needing real thread-private variables 
that preserves per-thread values across iterations (and thus also can 
possibly benefit from firstprivate) is seldomly enough used that an 
explicit declaration is OK, in particular when it buys us so much in 
safety in the common case.

To be very precise,

cdef double x, z
for i in prange(n):
     x = f(x)
     z = f(i)
     ...

goes to

cdef double x, z
for i in prange(n):
     x = z = nan
     x = f(x)
     z = f(i)
     ...

and we leave it to the C compiler to (trivially) optimize away "z = 
nan". And, yes, it is a stopgap solution until we've got control flow 
analysis so that we can outright disallow such uses of x (without 
threadprivate declaration, which also gives firstprivate behaviour).



>
> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
> We also probably want some way of explicitly marking a variable as
> shared and still be able to assign to/flush/sync it. Perhaps the
> parallel context could be used for these declarations, i.e.
>
>      with parallel(threadlocal=a, shared=(b,c)):
>          ...
>
> which would be considered an "expert" usecase.

I'm not set on the syntax for threadlocal variables; although your 
proposal feels funny/very unpythonic, almost like a C macro. For some 
inspiration, here's the Python solution (with no obvious place to put 
the type):

import threading
mydata = threading.local()
mydata.myvar = ... # value is threadprivate

> For all the discussion of threadsavailable/threadid, the most common
> usecase I see is for allocating a large shared buffer and partitioning
> it. This seems better handled by allocating separate thread-local
> buffers, no? I still like the context idea, but everything in a
> parallel block before and after the loop(s) also seems like a natural
> place to put any setup/teardown code (though the context has the
> advantage that __exit__ is always called, even if exceptions are
> raised, which makes cleanup a lot easier to handle).

I'd *really* like to have try/finally available in cython.parallel block 
for this, although I realize that may have to wait for a while. A big 
part of our discussions at the workshop were about how to handle 
exceptions; I guess there'll be a "phase 2" of this where 
break/continue/raise is dealt with.

Dag Sverre

From robertwb at math.washington.edu  Thu Apr 21 21:04:24 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 Apr 2011 12:04:24 -0700
Subject: [Cython] prange CEP updated
In-Reply-To: <4DB073BE.7010909@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<4DB073BE.7010909@astro.uio.no>
Message-ID: <BANLkTikoZ2Y_Ty6__bUkrBNKVYJ46ieWHw@mail.gmail.com>

On Thu, Apr 21, 2011 at 11:13 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/21/2011 10:37 AM, Robert Bradshaw wrote:
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> I don't see any technical issues with inferring firstprivate, the question
> is whether we want to. I suggest not inferring it in order to make this
> safer: One should be able to just try to change a loop from "range" to
> "prange", and either a) have things fail very hard, or b) just work
> correctly and be able to trust the results.
>
> Note that when I suggest using NaN, it is as initial values for EACH
> ITERATION, not per-thread initialization. It is not about "firstprivate" or
> not, but about disabling thread-private variables entirely in favor of
> "per-iteration" variables.
>
> I believe that by talking about "readonly" and "per-iteration" variables,
> rather than "thread-shared" and "thread-private" variables, this can be used
> much more safely and with virtually no knowledge of the details of
> threading. Again, what's in my mind are scientific programmers with (too)
> little training.
>
> In the end it's a matter of taste and what is most convenient to more users.
> But I believe the case of needing real thread-private variables that
> preserves per-thread values across iterations (and thus also can possibly
> benefit from firstprivate) is seldomly enough used that an explicit
> declaration is OK, in particular when it buys us so much in safety in the
> common case.
>
> To be very precise,
>
> cdef double x, z
> for i in prange(n):
> ? ?x = f(x)
> ? ?z = f(i)
> ? ?...
>
> goes to
>
> cdef double x, z
> for i in prange(n):
> ? ?x = z = nan
> ? ?x = f(x)
> ? ?z = f(i)
> ? ?...
>
> and we leave it to the C compiler to (trivially) optimize away "z = nan".
> And, yes, it is a stopgap solution until we've got control flow analysis so
> that we can outright disallow such uses of x (without threadprivate
> declaration, which also gives firstprivate behaviour).

OK, I had totally missed that these are per-iteration. In that case,
it makes more sense.

>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> ? ? with parallel(threadlocal=a, shared=(b,c)):
>> ? ? ? ? ...
>>
>> which would be considered an "expert" usecase.
>
> I'm not set on the syntax for threadlocal variables; although your proposal
> feels funny/very unpythonic, almost like a C macro. For some inspiration,
> here's the Python solution (with no obvious place to put the type):
>
> import threading
> mydata = threading.local()
> mydata.myvar = ... # value is threadprivate

That's nice and Pythonic, though I'm not sure how we would handle
typing and the passing of "mydata" around if we wanted to go that
route. We have cython.locals, we could introduce
cython.parallel.threadlocals(a=int), though this is a bit magical as
well.

>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> I'd *really* like to have try/finally available in cython.parallel block for
> this, although I realize that may have to wait for a while. A big part of
> our discussions at the workshop were about how to handle exceptions; I guess
> there'll be a "phase 2" of this where break/continue/raise is dealt with.

Yeah, this is definitely an (important) second or third phase.

- Robert

From sturla at molden.no  Fri Apr 22 23:40:55 2011
From: sturla at molden.no (Sturla Molden)
Date: Fri, 22 Apr 2011 23:40:55 +0200
Subject: [Cython] [SciPy-User] Central File Exchange for Scipy
In-Reply-To: <BANLkTiksYb5xC4zfXDryiZRRjVFTSdW-gw@mail.gmail.com>
References: <4DAD0411.6010405@creativetrax.com>
	<ioji8h$av6$1@dough.gmane.org>	<4DADCADA.1010905@creativetrax.com>	<4DADD381.60206@creativetrax.com>
	<iokm48$dc5$2@dough.gmane.org>	<4DADF03C.3080804@creativetrax.com>
	<iokujp$h46$2@dough.gmane.org>	<4DAE15F8.4090007@creativetrax.com>
	<iomkhu$41h$1@dough.gmane.org>	<4DAFE5C1.2090705@creativetrax.com>
	<iop8dd$uer$1@dough.gmane.org>	<BANLkTinrwE10+j-fisTQhpRfm_ovyU1tcQ@mail.gmail.com>	<BANLkTimPLv9r_9-GkQ7bbAO0-fC9wK=5rQ@mail.gmail.com>	<BANLkTi=O-yWTw4ZeGMn3cBtyEqJCU9+u5g@mail.gmail.com>
	<BANLkTiksYb5xC4zfXDryiZRRjVFTSdW-gw@mail.gmail.com>
Message-ID: <4DB1F5E7.2050502@molden.no>

Den 22.04.2011 23:22, skrev josef.pktd at gmail.com:
>
> after a facelift
>
> http://www.mathworks.com/matlabcentral/fileexchange/?sort=date_desc_updated&term=
>
> Josef
>

This is indeed something I miss for scientific Python as well.

I also miss a similar file exchange for Cython (not limited to 
scientific computing).


Here is another site for comparison (yes I know about the SciPy cookbook):

http://code.activestate.com/recipes/



Sturla













From stefan_ml at behnel.de  Mon Apr 25 07:52:05 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 25 Apr 2011 07:52:05 +0200
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <4D9DA560.8070505@behnel.de>
References: <4D9DA400.4060105@behnel.de> <4D9DA560.8070505@behnel.de>
Message-ID: <4DB50C05.5060701@behnel.de>

Stefan Behnel, 07.04.2011 13:52:
> Stefan Behnel, 07.04.2011 13:46:
>> I just noticed that the CPython pyregr tests have jumped up from ~14
>> minutes for a run to ~4 hours when we added generator support.
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>
>> I currently have no idea why that is (well, it's likely because we compile
>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It would
>> be great if someone could find the time to analyse this problem. The
>> current run time makes it basically impossible to keep these tests enabled.
>
> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
> tests take 30-45 minutes, which is very long, but not completely out of
> bounds. I've disabled the Py2.6 pyregr tests for now.

There seems to be a huge memory leak which almost certainly accounts for 
this. The Python process that runs the pyregr suite ends up with about 50GB 
of memory at the end, also in the latest Py3k builds.

I have no idea where it may be, but it started to show when we merged the 
generator support. That's where I noticed the instant jump in the runtime.

Stefan

From vitja.makarov at gmail.com  Mon Apr 25 08:19:05 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 25 Apr 2011 10:19:05 +0400
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <4DB50C05.5060701@behnel.de>
References: <4D9DA400.4060105@behnel.de> <4D9DA560.8070505@behnel.de>
	<4DB50C05.5060701@behnel.de>
Message-ID: <BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>

2011/4/25 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 07.04.2011 13:52:
>>
>> Stefan Behnel, 07.04.2011 13:46:
>>>
>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>> minutes for a run to ~4 hours when we added generator support.
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>
>>> I currently have no idea why that is (well, it's likely because we
>>> compile
>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>> would
>>> be great if someone could find the time to analyse this problem. The
>>> current run time makes it basically impossible to keep these tests
>>> enabled.
>>
>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>> tests take 30-45 minutes, which is very long, but not completely out of
>> bounds. I've disabled the Py2.6 pyregr tests for now.
>
> There seems to be a huge memory leak which almost certainly accounts for
> this. The Python process that runs the pyregr suite ends up with about 50GB
> of memory at the end, also in the latest Py3k builds.
>
> I have no idea where it may be, but it started to show when we merged the
> generator support. That's where I noticed the instant jump in the runtime.
>

That's very strange for my branch it takes about 30 minutes that is ok.

-- 
vitja.

From stefan_ml at behnel.de  Mon Apr 25 09:20:26 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 25 Apr 2011 09:20:26 +0200
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
References: <4D9DA400.4060105@behnel.de>
	<4D9DA560.8070505@behnel.de>	<4DB50C05.5060701@behnel.de>
	<BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
Message-ID: <4DB520BA.3060109@behnel.de>

Vitja Makarov, 25.04.2011 08:19:
> 2011/4/25 Stefan Behnel:
>> Stefan Behnel, 07.04.2011 13:52:
>>>
>>> Stefan Behnel, 07.04.2011 13:46:
>>>>
>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>> minutes for a run to ~4 hours when we added generator support.
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>
>>>> I currently have no idea why that is (well, it's likely because we
>>>> compile
>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>> would
>>>> be great if someone could find the time to analyse this problem. The
>>>> current run time makes it basically impossible to keep these tests
>>>> enabled.
>>>
>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>>> tests take 30-45 minutes, which is very long, but not completely out of
>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>
>> There seems to be a huge memory leak which almost certainly accounts for
>> this. The Python process that runs the pyregr suite ends up with about 50GB
>> of memory at the end, also in the latest Py3k builds.
>>
>> I have no idea where it may be, but it started to show when we merged the
>> generator support. That's where I noticed the instant jump in the runtime.
>
> That's very strange for my branch it takes about 30 minutes that is ok.

Does your branch leak memory when you run a generator? Using a debug build 
of CPython, running a Cython generator for the first time persistently 
increases the reference counts for me (using the generators.pyx test):

    Python 2.7.1+ (2.7:c821d3d335e8, Apr 22 2011, 18:37:12)
    [GCC 4.4.3] on linux2
    >>> import generators as G
    [17021 refs]
    >>> import gc
    [17462 refs]
    >>> gc.collect()
    0
    [17465 refs]
    >>> gc.collect()
    0
    [17465 refs]
    >>> list(G.with_outer(1,2,3,4)())
    [1, 2, 3, 4]
    [17474 refs]
    >>> gc.collect()
    0
    [17470 refs]

It seems like this leaked five references. And it seems to be related to 
the generator being inside of a closure itself:

   >>> list(G.test_first_assignment())
   [5, 10, (5, 10)]
   [17475 refs]
   >>> gc.collect()
   0
   [17470 refs]

back to the last value here, but:

   >>> list(G.generator_nonlocal()(5))
   [2, 3, 4, 5, 6]
   [17481 refs]
   >>> gc.collect()
   0
   [17476 refs]

Another six references leaked. And it's only the first time the generator 
is run, running it a second time doesn't change anything:

   >>> list(G.generator_nonlocal()(5))
   [2, 3, 4, 5, 6]
   [17481 refs]
   >>> gc.collect()
   0
   [17476 refs]

Stefan

From stefan_ml at behnel.de  Mon Apr 25 10:49:05 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 25 Apr 2011 10:49:05 +0200
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
References: <4D9DA400.4060105@behnel.de>
	<4D9DA560.8070505@behnel.de>	<4DB50C05.5060701@behnel.de>
	<BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
Message-ID: <4DB53581.5090505@behnel.de>

Vitja Makarov, 25.04.2011 08:19:
> 2011/4/25 Stefan Behnel:
>> Stefan Behnel, 07.04.2011 13:52:
>>>
>>> Stefan Behnel, 07.04.2011 13:46:
>>>>
>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>> minutes for a run to ~4 hours when we added generator support.
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>
>>>> I currently have no idea why that is (well, it's likely because we
>>>> compile
>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>> would
>>>> be great if someone could find the time to analyse this problem. The
>>>> current run time makes it basically impossible to keep these tests
>>>> enabled.
>>>
>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>>> tests take 30-45 minutes, which is very long, but not completely out of
>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>
>> There seems to be a huge memory leak which almost certainly accounts for
>> this. The Python process that runs the pyregr suite ends up with about 50GB
>> of memory at the end, also in the latest Py3k builds.
>>
>> I have no idea where it may be, but it started to show when we merged the
>> generator support. That's where I noticed the instant jump in the runtime.
>
> That's very strange for my branch it takes about 30 minutes that is ok.

There's also a second path that's worth investigating. As part of the 
merge, there was another change that came in: the CythonPyregrTestCase 
implementation. This means that the regression tests are now being run 
differently than before. The massive memory consumption may simply be due 
to the mass of unit tests being loaded into memory.

Stefan

From vitja.makarov at gmail.com  Mon Apr 25 11:04:48 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 25 Apr 2011 13:04:48 +0400
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <4DB53581.5090505@behnel.de>
References: <4D9DA400.4060105@behnel.de> <4D9DA560.8070505@behnel.de>
	<4DB50C05.5060701@behnel.de>
	<BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
	<4DB53581.5090505@behnel.de>
Message-ID: <BANLkTimw+dGsENd-KhgHPKy6ENc33jjTcA@mail.gmail.com>

2011/4/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.04.2011 08:19:
>>
>> 2011/4/25 Stefan Behnel:
>>>
>>> Stefan Behnel, 07.04.2011 13:52:
>>>>
>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>
>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>
>>>>>
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>
>>>>> I currently have no idea why that is (well, it's likely because we
>>>>> compile
>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>> would
>>>>> be great if someone could find the time to analyse this problem. The
>>>>> current run time makes it basically impossible to keep these tests
>>>>> enabled.
>>>>
>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>> Py2.7
>>>> tests take 30-45 minutes, which is very long, but not completely out of
>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>
>>> There seems to be a huge memory leak which almost certainly accounts for
>>> this. The Python process that runs the pyregr suite ends up with about
>>> 50GB
>>> of memory at the end, also in the latest Py3k builds.
>>>
>>> I have no idea where it may be, but it started to show when we merged the
>>> generator support. That's where I noticed the instant jump in the
>>> runtime.
>>
>> That's very strange for my branch it takes about 30 minutes that is ok.
>
> There's also a second path that's worth investigating. As part of the merge,
> there was another change that came in: the CythonPyregrTestCase
> implementation. This means that the regression tests are now being run
> differently than before. The massive memory consumption may simply be due to
> the mass of unit tests being loaded into memory.
>


   def run_test():
..................................
        try:
            module = __import__(self.module)
            if hasattr(module, 'test_main'):
                module.test_main()
        except (unittest.SkipTest, support.ResourceDenied):
            result.addSkip(self, 'ok')


It seems that all the modules stay loaded so may be they should be
unloaded with del sys.modules[module_name]?

-- 
vitja.

From stefan_ml at behnel.de  Mon Apr 25 11:25:18 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 25 Apr 2011 11:25:18 +0200
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <BANLkTimw+dGsENd-KhgHPKy6ENc33jjTcA@mail.gmail.com>
References: <4D9DA400.4060105@behnel.de>
	<4D9DA560.8070505@behnel.de>	<4DB50C05.5060701@behnel.de>	<BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>	<4DB53581.5090505@behnel.de>
	<BANLkTimw+dGsENd-KhgHPKy6ENc33jjTcA@mail.gmail.com>
Message-ID: <4DB53DFE.9060601@behnel.de>

Vitja Makarov, 25.04.2011 11:04:
> 2011/4/25 Stefan Behnel<stefan_ml at behnel.de>:
>> Vitja Makarov, 25.04.2011 08:19:
>>>
>>> 2011/4/25 Stefan Behnel:
>>>>
>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>
>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>
>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>
>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>> compile
>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>> would
>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>> current run time makes it basically impossible to keep these tests
>>>>>> enabled.
>>>>>
>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>> Py2.7
>>>>> tests take 30-45 minutes, which is very long, but not completely out of
>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>
>>>> There seems to be a huge memory leak which almost certainly accounts for
>>>> this. The Python process that runs the pyregr suite ends up with about
>>>> 50GB
>>>> of memory at the end, also in the latest Py3k builds.
>>>>
>>>> I have no idea where it may be, but it started to show when we merged the
>>>> generator support. That's where I noticed the instant jump in the
>>>> runtime.
>>>
>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>
>> There's also a second path that's worth investigating. As part of the merge,
>> there was another change that came in: the CythonPyregrTestCase
>> implementation. This means that the regression tests are now being run
>> differently than before. The massive memory consumption may simply be due to
>> the mass of unit tests being loaded into memory.
>
>     def run_test():
> ..................................
>          try:
>              module = __import__(self.module)
>              if hasattr(module, 'test_main'):
>                  module.test_main()
>          except (unittest.SkipTest, support.ResourceDenied):
>              result.addSkip(self, 'ok')
>
>
> It seems that all the modules stay loaded so may be they should be
> unloaded with del sys.modules[module_name]?

(Binary) module unloading isn't really supported in CPython. There's PEP 
3121 that has the potential to change it, but it's not completely 
implemented, neither in CPython nor in Cython. A major problem is that 
unloading a module deletes its globals but not necessarily the code that 
uses them. For example, instances of types defined in the module can still 
be alive at that point.

The way runtests.py deals with this is forking before loading a module. 
However, this does not currently work with the "xmlrunner" which we use on 
Hudson, so we let all tests run in a single process there.

Stefan

From vitja.makarov at gmail.com  Mon Apr 25 18:51:34 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 25 Apr 2011 20:51:34 +0400
Subject: [Cython] Hudson pyregr testing takes too long
In-Reply-To: <4DB53DFE.9060601@behnel.de>
References: <4D9DA400.4060105@behnel.de> <4D9DA560.8070505@behnel.de>
	<4DB50C05.5060701@behnel.de>
	<BANLkTikZ2H2PbiuSzOKgzQymjnUM2q7Qiw@mail.gmail.com>
	<4DB53581.5090505@behnel.de>
	<BANLkTimw+dGsENd-KhgHPKy6ENc33jjTcA@mail.gmail.com>
	<4DB53DFE.9060601@behnel.de>
Message-ID: <BANLkTinY+-wZ0xaQsrR+hcYCBmn3i3KcaQ@mail.gmail.com>

2011/4/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.04.2011 11:04:
>>
>> 2011/4/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 25.04.2011 08:19:
>>>>
>>>> 2011/4/25 Stefan Behnel:
>>>>>
>>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>>
>>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>>
>>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>>
>>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>>> compile
>>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>>> would
>>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>>> current run time makes it basically impossible to keep these tests
>>>>>>> enabled.
>>>>>>
>>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>>> Py2.7
>>>>>> tests take 30-45 minutes, which is very long, but not completely out
>>>>>> of
>>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>>
>>>>> There seems to be a huge memory leak which almost certainly accounts
>>>>> for
>>>>> this. The Python process that runs the pyregr suite ends up with about
>>>>> 50GB
>>>>> of memory at the end, also in the latest Py3k builds.
>>>>>
>>>>> I have no idea where it may be, but it started to show when we merged
>>>>> the
>>>>> generator support. That's where I noticed the instant jump in the
>>>>> runtime.
>>>>
>>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>>
>>> There's also a second path that's worth investigating. As part of the
>>> merge,
>>> there was another change that came in: the CythonPyregrTestCase
>>> implementation. This means that the regression tests are now being run
>>> differently than before. The massive memory consumption may simply be due
>>> to
>>> the mass of unit tests being loaded into memory.
>>
>> ? ?def run_test():
>> ..................................
>> ? ? ? ? try:
>> ? ? ? ? ? ? module = __import__(self.module)
>> ? ? ? ? ? ? if hasattr(module, 'test_main'):
>> ? ? ? ? ? ? ? ? module.test_main()
>> ? ? ? ? except (unittest.SkipTest, support.ResourceDenied):
>> ? ? ? ? ? ? result.addSkip(self, 'ok')
>>
>>
>> It seems that all the modules stay loaded so may be they should be
>> unloaded with del sys.modules[module_name]?
>
> (Binary) module unloading isn't really supported in CPython. There's PEP
> 3121 that has the potential to change it, but it's not completely
> implemented, neither in CPython nor in Cython. A major problem is that
> unloading a module deletes its globals but not necessarily the code that
> uses them. For example, instances of types defined in the module can still
> be alive at that point.
>
> The way runtests.py deals with this is forking before loading a module.
> However, this does not currently work with the "xmlrunner" which we use on
> Hudson, so we let all tests run in a single process there.
>


Btw when running plain python code with generators total ref counter
doesn't get back to initial value.
I tried to trace scope and generator destructors and they are run as
expected. So I'm not sure about leaks in generators.

-- 
vitja.

From markflorisson88 at gmail.com  Tue Apr 26 16:23:51 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 26 Apr 2011 16:23:51 +0200
Subject: [Cython] Fused Types
Message-ID: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>

Hey,

I've been working a bit on fused types
(http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
generate code for every permutation of specific types. Currently it
only works for cdef functions. Now in SimpleCallNode I'm trying to use
PyrexTypes.best_match() to find the function with the best signature,
but it doesn't seem to take care of coercions, e.g. it calls
'assignable_from' on the dst_type (e.g. char *), but for
BuiltinObjectType (a str) this returns false. Why is this the case,
don't calls to overloaded C++ methods need to dispatch properly here
also?

Other issues are public and api declarations. So should we support
that at all? We could define a macro that calls a function that does
the dispatch. So e.g. for this

ctypedef cython.fused_type(typeA, typeB) dtype

cdef func(dtype x):
    ...

we would get two generated functions, say, __pyx_typeA_func and
__pyx_typeB_func. So we could have a macro get_func(dtype) or
something that then substitutes __pyx_get_func(#dtype), where
__pyx_get_func returns the pointer to the right function based on the
type names. I'm not sure we should support it, right now I just put
the mangled names in the header. At least the cdef functions will be
sharable between Cython implementation files.

I also noticed that for cdef functions with optional argument it gets
a struct as argument, but this struct doesn't seem to end up in the
header file when the function is declared public. I believe that also
the typedefs for ctypedef-ed things in the .pyx file don't make it
there when used to type a cdef function's arguments. Should that be
fixed?

Cheers,

Mark

From markflorisson88 at gmail.com  Tue Apr 26 16:25:38 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 26 Apr 2011 16:25:38 +0200
Subject: [Cython] prange CEP updated
In-Reply-To: <4DB073BE.7010909@astro.uio.no>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<4DB073BE.7010909@astro.uio.no>
Message-ID: <BANLkTin6CEm7Se6+fWpGfnoP3tk1AKJ=Qg@mail.gmail.com>

On 21 April 2011 20:13, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 04/21/2011 10:37 AM, Robert Bradshaw wrote:
>>
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> <markflorisson88 at gmail.com> ?wrote:
>>>
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>> ?wrote:
>>>>
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I
>>>> can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase
>>>> explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better
>>>> to
>>>> take a break from it at this point so that we can think more about that
>>>> and
>>>> not do anything rash; perhaps open up a specific thread on them and ask
>>>> for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got
>>>> the
>>>> flow I certainly wom't object myself.
>>>>
>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> I don't see any technical issues with inferring firstprivate, the question
> is whether we want to. I suggest not inferring it in order to make this
> safer: One should be able to just try to change a loop from "range" to
> "prange", and either a) have things fail very hard, or b) just work
> correctly and be able to trust the results.
>
> Note that when I suggest using NaN, it is as initial values for EACH
> ITERATION, not per-thread initialization. It is not about "firstprivate" or
> not, but about disabling thread-private variables entirely in favor of
> "per-iteration" variables.
>
> I believe that by talking about "readonly" and "per-iteration" variables,
> rather than "thread-shared" and "thread-private" variables, this can be used
> much more safely and with virtually no knowledge of the details of
> threading. Again, what's in my mind are scientific programmers with (too)
> little training.
>
> In the end it's a matter of taste and what is most convenient to more users.
> But I believe the case of needing real thread-private variables that
> preserves per-thread values across iterations (and thus also can possibly
> benefit from firstprivate) is seldomly enough used that an explicit
> declaration is OK, in particular when it buys us so much in safety in the
> common case.
>
> To be very precise,
>
> cdef double x, z
> for i in prange(n):
> ? ?x = f(x)
> ? ?z = f(i)
> ? ?...
>
> goes to
>
> cdef double x, z
> for i in prange(n):
> ? ?x = z = nan
> ? ?x = f(x)
> ? ?z = f(i)
> ? ?...
>
> and we leave it to the C compiler to (trivially) optimize away "z = nan".
> And, yes, it is a stopgap solution until we've got control flow analysis so
> that we can outright disallow such uses of x (without threadprivate
> declaration, which also gives firstprivate behaviour).
>
Ah, I see, sure, that sounds sensible. I'm currently working on fused
types, so when I finish that up I'll return to that.
>
>>
>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> ? ? with parallel(threadlocal=a, shared=(b,c)):
>> ? ? ? ? ...
>>
>> which would be considered an "expert" usecase.
>
> I'm not set on the syntax for threadlocal variables; although your proposal
> feels funny/very unpythonic, almost like a C macro. For some inspiration,
> here's the Python solution (with no obvious place to put the type):
>
> import threading
> mydata = threading.local()
> mydata.myvar = ... # value is threadprivate
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> I'd *really* like to have try/finally available in cython.parallel block for
> this, although I realize that may have to wait for a while. A big part of
> our discussions at the workshop were about how to handle exceptions; I guess
> there'll be a "phase 2" of this where break/continue/raise is dealt with.

I'll leave that until I finish fused types and the typed memory views.
Before I'd start on that I'd first review the with gil block and
ensure the tests pass in all python versions, and perhaps that should
be merged first before I pull it into the parallel branch? Otherwise
you're kind of forced to review both branches.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Tue Apr 26 16:43:06 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 26 Apr 2011 16:43:06 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
Message-ID: <4DB6D9FA.9030405@behnel.de>

mark florisson, 26.04.2011 16:23:
> I've been working a bit on fused types
> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
> generate code for every permutation of specific types. Currently it
> only works for cdef functions. Now in SimpleCallNode I'm trying to use
> PyrexTypes.best_match() to find the function with the best signature,
> but it doesn't seem to take care of coercions, e.g. it calls
> 'assignable_from' on the dst_type (e.g. char *), but for
> BuiltinObjectType (a str) this returns false.

Which is correct. "char*" cannot coerce from/to "str". It can coerce to 
"bytes", though.

http://wiki.cython.org/enhancements/stringliterals


> Why is this the case,
> don't calls to overloaded C++ methods need to dispatch properly here
> also?

If this doesn't work, I assume it just isn't implemented.


> Other issues are public and api declarations. So should we support
> that at all? We could define a macro that calls a function that does
> the dispatch. So e.g. for this
>
> ctypedef cython.fused_type(typeA, typeB) dtype
>
> cdef func(dtype x):
>      ...
>
> we would get two generated functions, say, __pyx_typeA_func and
> __pyx_typeB_func. So we could have a macro get_func(dtype) or
> something that then substitutes __pyx_get_func(#dtype), where
> __pyx_get_func returns the pointer to the right function based on the
> type names. I'm not sure we should support it, right now I just put
> the mangled names in the header. At least the cdef functions will be
> sharable between Cython implementation files.

I'm fine with explicitly forbidding this for now. It may eventually work 
for Python object types where we can properly dispatch, but it won't easily 
work for C types. It may work in C++, though.


> I also noticed that for cdef functions with optional argument it gets
> a struct as argument, but this struct doesn't seem to end up in the
> header file when the function is declared public. I believe that also
> the typedefs for ctypedef-ed things in the .pyx file don't make it
> there when used to type a cdef function's arguments. Should that be
> fixed?

No, I think this should also become a compiler error. These functions are 
not meant to be called from C code. It's a Cython convenience feature. As 
long as it's not correctly supported on both ends of the publicly exported 
C-API, it's best to keep users from using it at all.

Stefan

From stefan_ml at behnel.de  Tue Apr 26 16:50:55 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 26 Apr 2011 16:50:55 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DA8A8A3.6080600@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
Message-ID: <4DB6DBCF.2020301@behnel.de>

Stefan Behnel, 15.04.2011 22:20:
> Stefan Behnel, 11.04.2011 15:08:
>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>> works out well, we may have it up in a couple of days.
>
> ... or maybe not. It may take a little longer due to lack of time on his side.
>
>
>> I would expect that Cython won't be a big winner in this game, given that
>> it will only compile plain untyped Python code. It's also going to fail
>> entirely in some of the benchmarks. But I think it's worth having it up
>> there, simply as a way for us to see where we are performance-wise and to
>> get quick (nightly) feed-back about optimisations we try. The benchmark
>> suite is also a nice set of real-world Python code that will allow us to
>> find compliance issues.
>
> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got
> at least some of the benchmarks running. Note that they are actually simple
> ones, only a single module. Basically all complex benchmarks fail due to
> known bugs, such as Cython def functions not accepting attribute
> assignments (e.g. on wrapping). There's also a problem with code that uses
> platform specific names conditionally, such as WindowsError when running on
> Windows. Cython complains about non-builtin names here. I'm considering to
> turn that into a visible warning instead of an error, so that the name
> would instead be looked up dynamically to let the code fail at runtime
> *iff* it reaches the name lookup.
>
> Anyway, here are the numbers. I got them with "auto_cpdef" enabled,
> although that doesn't even seem to make that a big difference. The baseline
> is a self-compiled Python 2.7.1+ (about a month old).

[numbers stripped]

And here's the shiny graph:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html

It gets automatically rebuilt by this Hudson job:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/

Stefan

From markflorisson88 at gmail.com  Tue Apr 26 17:18:50 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 26 Apr 2011 17:18:50 +0200
Subject: [Cython] Fused Types
In-Reply-To: <4DB6D9FA.9030405@behnel.de>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
Message-ID: <BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>

On 26 April 2011 16:43, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 26.04.2011 16:23:
>>
>> I've been working a bit on fused types
>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>> generate code for every permutation of specific types. Currently it
>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>> PyrexTypes.best_match() to find the function with the best signature,
>> but it doesn't seem to take care of coercions, e.g. it calls
>> 'assignable_from' on the dst_type (e.g. char *), but for
>> BuiltinObjectType (a str) this returns false.
>
> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
> "bytes", though.
>
> http://wiki.cython.org/enhancements/stringliterals

Right, I see, so the thing is that I was using string literals which
should be inferred as the bytes type when they are assigned to a char
*. The thing is that because the type is fused, the argument may be
anything, so I was hoping best_match would figure it out for me.
Apparently it doesn't, and indeed, this example doesn't work:

cdef extern from "Foo.h":
    cdef cppclass Foo:
        Foo(char *)
        Foo(int)

cdef char *foo = "foo"
cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
method found")
cdef Foo* bar = new Foo(foo)   # <- this works

So that's pretty lame, I think I should fix that.

>
>> Why is this the case,
>> don't calls to overloaded C++ methods need to dispatch properly here
>> also?
>
> If this doesn't work, I assume it just isn't implemented.
>
>
>> Other issues are public and api declarations. So should we support
>> that at all? We could define a macro that calls a function that does
>> the dispatch. So e.g. for this
>>
>> ctypedef cython.fused_type(typeA, typeB) dtype
>>
>> cdef func(dtype x):
>> ? ? ...
>>
>> we would get two generated functions, say, __pyx_typeA_func and
>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>> something that then substitutes __pyx_get_func(#dtype), where
>> __pyx_get_func returns the pointer to the right function based on the
>> type names. I'm not sure we should support it, right now I just put
>> the mangled names in the header. At least the cdef functions will be
>> sharable between Cython implementation files.
>
> I'm fine with explicitly forbidding this for now. It may eventually work for
> Python object types where we can properly dispatch, but it won't easily work
> for C types. It may work in C++, though.
>

Ok, will do.

>> I also noticed that for cdef functions with optional argument it gets
>> a struct as argument, but this struct doesn't seem to end up in the
>> header file when the function is declared public. I believe that also
>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>> there when used to type a cdef function's arguments. Should that be
>> fixed?
>
> No, I think this should also become a compiler error. These functions are
> not meant to be called from C code. It's a Cython convenience feature. As
> long as it's not correctly supported on both ends of the publicly exported
> C-API, it's best to keep users from using it at all.

Ok, I'll try to make Cython issue an error for these cases.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Tue Apr 26 19:52:37 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 26 Apr 2011 10:52:37 -0700
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DB6DBCF.2020301@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
	<4DB6DBCF.2020301@behnel.de>
Message-ID: <BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>

On Tue, Apr 26, 2011 at 7:50 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 15.04.2011 22:20:
>>
>> Stefan Behnel, 11.04.2011 15:08:
>>>
>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>>> works out well, we may have it up in a couple of days.
>>
>> ... or maybe not. It may take a little longer due to lack of time on his
>> side.
>>
>>
>>> I would expect that Cython won't be a big winner in this game, given that
>>> it will only compile plain untyped Python code. It's also going to fail
>>> entirely in some of the benchmarks. But I think it's worth having it up
>>> there, simply as a way for us to see where we are performance-wise and to
>>> get quick (nightly) feed-back about optimisations we try. The benchmark
>>> suite is also a nice set of real-world Python code that will allow us to
>>> find compliance issues.
>>
>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got
>> at least some of the benchmarks running. Note that they are actually
>> simple
>> ones, only a single module. Basically all complex benchmarks fail due to
>> known bugs, such as Cython def functions not accepting attribute
>> assignments (e.g. on wrapping). There's also a problem with code that uses
>> platform specific names conditionally, such as WindowsError when running
>> on
>> Windows. Cython complains about non-builtin names here. I'm considering to
>> turn that into a visible warning instead of an error, so that the name
>> would instead be looked up dynamically to let the code fail at runtime
>> *iff* it reaches the name lookup.
>>
>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled,
>> although that doesn't even seem to make that a big difference. The
>> baseline
>> is a self-compiled Python 2.7.1+ (about a month old).
>
> [numbers stripped]
>
> And here's the shiny graph:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html
>
> It gets automatically rebuilt by this Hudson job:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/

Cool. Any history stored/displayed?

- Robert

From robertwb at math.washington.edu  Tue Apr 26 19:59:45 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 26 Apr 2011 10:59:45 -0700
Subject: [Cython] prange CEP updated
In-Reply-To: <BANLkTin6CEm7Se6+fWpGfnoP3tk1AKJ=Qg@mail.gmail.com>
References: <4D9B7BA9.2060509@astro.uio.no>
	<BANLkTimL0GEkYdRkJpTUdRmPcbzJAbkeYg@mail.gmail.com>
	<4DA60013.6060901@astro.uio.no>
	<BANLkTikr4ZKTzKJ4X22JjUZwm_P_ExBd-w@mail.gmail.com>
	<BANLkTimCgoPebsBhQtTaxSsacmz8eSbDQg@mail.gmail.com>
	<4DA73D22.9050605@astro.uio.no>
	<BANLkTimPyFbSagcrnhy4ZScBXqzFk3KAbw@mail.gmail.com>
	<4DA743EF.7080200@astro.uio.no>
	<BANLkTi=Ks=yECn-83Oje_pJ5zYPSxmJO=A@mail.gmail.com>
	<4DA74CEC.2000609@astro.uio.no>
	<BANLkTikYUoRVNwQxfEVqhxOkzyHA866yiA@mail.gmail.com>
	<4DA9C6DF.1060406@astro.uio.no>
	<BANLkTikNwGvs9YOSKx5wJDdqXw-Mbw2krQ@mail.gmail.com>
	<BANLkTinnn5L5BW3f-HeOQcXiHCUA9b46Tw@mail.gmail.com>
	<62739f06-a6e9-4cd9-99bd-e3af674fdc61@email.android.com>
	<BANLkTimdjvP2k1SmxzwefqRC6FhYVGi0Vg@mail.gmail.com>
	<BANLkTimFAkMvGUcoBYfePDMf9+52XNQ3kg@mail.gmail.com>
	<4DB073BE.7010909@astro.uio.no>
	<BANLkTin6CEm7Se6+fWpGfnoP3tk1AKJ=Qg@mail.gmail.com>
Message-ID: <BANLkTim5SNrDtWPqCYfCFPXyREw2YSiW7w@mail.gmail.com>

On Tue, Apr 26, 2011 at 7:25 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 21 April 2011 20:13, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> On 04/21/2011 10:37 AM, Robert Bradshaw wrote:
>>>
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> <markflorisson88 at gmail.com> ?wrote:
>>>>
>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>
>>>> ?wrote:
>>>>>
>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I
>>>>> can't
>>>>> have a look yet but I will later)
>>>>>
>>>>> You're right about (the current) buffers and the gil. A testcase
>>>>> explicitly
>>>>> for them would be good.
>>>>>
>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better
>>>>> to
>>>>> take a break from it at this point so that we can think more about that
>>>>> and
>>>>> not do anything rash; perhaps open up a specific thread on them and ask
>>>>> for
>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>> something else (fused types?) until I can get around to review and merge
>>>>> what you have so far? You'll know best what works for you though. If you
>>>>> decide to implement explicit threadprivate variables because you've got
>>>>> the
>>>>> flow I certainly wom't object myself.
>>>>>
>>>> ?Ok, cool, I'll move on :) I already included a test with a prange and
>>>> a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> I don't see any technical issues with inferring firstprivate, the question
>> is whether we want to. I suggest not inferring it in order to make this
>> safer: One should be able to just try to change a loop from "range" to
>> "prange", and either a) have things fail very hard, or b) just work
>> correctly and be able to trust the results.
>>
>> Note that when I suggest using NaN, it is as initial values for EACH
>> ITERATION, not per-thread initialization. It is not about "firstprivate" or
>> not, but about disabling thread-private variables entirely in favor of
>> "per-iteration" variables.
>>
>> I believe that by talking about "readonly" and "per-iteration" variables,
>> rather than "thread-shared" and "thread-private" variables, this can be used
>> much more safely and with virtually no knowledge of the details of
>> threading. Again, what's in my mind are scientific programmers with (too)
>> little training.
>>
>> In the end it's a matter of taste and what is most convenient to more users.
>> But I believe the case of needing real thread-private variables that
>> preserves per-thread values across iterations (and thus also can possibly
>> benefit from firstprivate) is seldomly enough used that an explicit
>> declaration is OK, in particular when it buys us so much in safety in the
>> common case.
>>
>> To be very precise,
>>
>> cdef double x, z
>> for i in prange(n):
>> ? ?x = f(x)
>> ? ?z = f(i)
>> ? ?...
>>
>> goes to
>>
>> cdef double x, z
>> for i in prange(n):
>> ? ?x = z = nan
>> ? ?x = f(x)
>> ? ?z = f(i)
>> ? ?...
>>
>> and we leave it to the C compiler to (trivially) optimize away "z = nan".
>> And, yes, it is a stopgap solution until we've got control flow analysis so
>> that we can outright disallow such uses of x (without threadprivate
>> declaration, which also gives firstprivate behaviour).
>>
> Ah, I see, sure, that sounds sensible. I'm currently working on fused
> types, so when I finish that up I'll return to that.
>>
>>>
>>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>>> We also probably want some way of explicitly marking a variable as
>>> shared and still be able to assign to/flush/sync it. Perhaps the
>>> parallel context could be used for these declarations, i.e.
>>>
>>> ? ? with parallel(threadlocal=a, shared=(b,c)):
>>> ? ? ? ? ...
>>>
>>> which would be considered an "expert" usecase.
>>
>> I'm not set on the syntax for threadlocal variables; although your proposal
>> feels funny/very unpythonic, almost like a C macro. For some inspiration,
>> here's the Python solution (with no obvious place to put the type):
>>
>> import threading
>> mydata = threading.local()
>> mydata.myvar = ... # value is threadprivate
>>
>>> For all the discussion of threadsavailable/threadid, the most common
>>> usecase I see is for allocating a large shared buffer and partitioning
>>> it. This seems better handled by allocating separate thread-local
>>> buffers, no? I still like the context idea, but everything in a
>>> parallel block before and after the loop(s) also seems like a natural
>>> place to put any setup/teardown code (though the context has the
>>> advantage that __exit__ is always called, even if exceptions are
>>> raised, which makes cleanup a lot easier to handle).
>>
>> I'd *really* like to have try/finally available in cython.parallel block for
>> this, although I realize that may have to wait for a while. A big part of
>> our discussions at the workshop were about how to handle exceptions; I guess
>> there'll be a "phase 2" of this where break/continue/raise is dealt with.
>
> I'll leave that until I finish fused types and the typed memory views.
> Before I'd start on that I'd first review the with gil block and
> ensure the tests pass in all python versions, and perhaps that should
> be merged first before I pull it into the parallel branch? Otherwise
> you're kind of forced to review both branches.

Yes, that makes sense.

- Robert

From robertwb at math.washington.edu  Tue Apr 26 20:05:48 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 26 Apr 2011 11:05:48 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
Message-ID: <BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>

On Tue, Apr 26, 2011 at 8:18 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 26 April 2011 16:43, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 26.04.2011 16:23:
>>>
>>> I've been working a bit on fused types
>>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>>> generate code for every permutation of specific types. Currently it
>>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>>> PyrexTypes.best_match() to find the function with the best signature,
>>> but it doesn't seem to take care of coercions, e.g. it calls
>>> 'assignable_from' on the dst_type (e.g. char *), but for
>>> BuiltinObjectType (a str) this returns false.
>>
>> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
>> "bytes", though.
>>
>> http://wiki.cython.org/enhancements/stringliterals
>
> Right, I see, so the thing is that I was using string literals which
> should be inferred as the bytes type when they are assigned to a char
> *. The thing is that because the type is fused, the argument may be
> anything, so I was hoping best_match would figure it out for me.
> Apparently it doesn't, and indeed, this example doesn't work:
>
> cdef extern from "Foo.h":
> ? ?cdef cppclass Foo:
> ? ? ? ?Foo(char *)
> ? ? ? ?Foo(int)
>
> cdef char *foo = "foo"
> cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
> method found")
> cdef Foo* bar = new Foo(foo) ? # <- this works
>
> So that's pretty lame, I think I should fix that.

Agreed.

>>> Why is this the case,
>>> don't calls to overloaded C++ methods need to dispatch properly here
>>> also?
>>
>> If this doesn't work, I assume it just isn't implemented.
>>
>>
>>> Other issues are public and api declarations. So should we support
>>> that at all? We could define a macro that calls a function that does
>>> the dispatch. So e.g. for this
>>>
>>> ctypedef cython.fused_type(typeA, typeB) dtype
>>>
>>> cdef func(dtype x):
>>> ? ? ...
>>>
>>> we would get two generated functions, say, __pyx_typeA_func and
>>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>>> something that then substitutes __pyx_get_func(#dtype), where
>>> __pyx_get_func returns the pointer to the right function based on the
>>> type names. I'm not sure we should support it, right now I just put
>>> the mangled names in the header. At least the cdef functions will be
>>> sharable between Cython implementation files.
>>
>> I'm fine with explicitly forbidding this for now. It may eventually work for
>> Python object types where we can properly dispatch, but it won't easily work
>> for C types. It may work in C++, though.
>>
>
> Ok, will do.

For the moment, putting mangled names in the header should be fine. A
macro might make sense in the long term.

Somewhat orthogonal, it could makes sense to do some dispatching on
type for cpdef functions.

>>> I also noticed that for cdef functions with optional argument it gets
>>> a struct as argument, but this struct doesn't seem to end up in the
>>> header file when the function is declared public. I believe that also
>>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>>> there when used to type a cdef function's arguments. Should that be
>>> fixed?
>>
>> No, I think this should also become a compiler error. These functions are
>> not meant to be called from C code. It's a Cython convenience feature. As
>> long as it's not correctly supported on both ends of the publicly exported
>> C-API, it's best to keep users from using it at all.
>
> Ok, I'll try to make Cython issue an error for these cases.

+1

- Robert

From stefan_ml at behnel.de  Tue Apr 26 22:02:24 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 26 Apr 2011 22:02:24 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
Message-ID: <4DB724D0.8090607@behnel.de>

mark florisson, 26.04.2011 17:18:
> On 26 April 2011 16:43, Stefan Behnel wrote:
>> mark florisson, 26.04.2011 16:23:
>>>
>>> I've been working a bit on fused types
>>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>>> generate code for every permutation of specific types. Currently it
>>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>>> PyrexTypes.best_match() to find the function with the best signature,
>>> but it doesn't seem to take care of coercions, e.g. it calls
>>> 'assignable_from' on the dst_type (e.g. char *), but for
>>> BuiltinObjectType (a str) this returns false.
>>
>> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
>> "bytes", though.
>>
>> http://wiki.cython.org/enhancements/stringliterals
>
> Right, I see, so the thing is that I was using string literals which
> should be inferred as the bytes type when they are assigned to a char
> *. The thing is that because the type is fused, the argument may be
> anything, so I was hoping best_match would figure it out for me.
> Apparently it doesn't, and indeed, this example doesn't work:
>
> cdef extern from "Foo.h":
>      cdef cppclass Foo:
>          Foo(char *)
>          Foo(int)
>
> cdef char *foo = "foo"
> cdef Foo* foo = new Foo("foo") #<- this doesn't work ("no suitable
> method found")
> cdef Foo* bar = new Foo(foo)   #<- this works
>
> So that's pretty lame, I think I should fix that.

Well, I assume it's less lame when you use b"foo". But given the current 
auto-coerce semantics for unprefixed string literals in Cython, I agree 
that it should also auto-coerce here.

Stefan

From stefan_ml at behnel.de  Wed Apr 27 09:45:56 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 27 Apr 2011 09:45:56 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>
References: <4DA2FD5E.5090704@behnel.de>
	<4DA8A8A3.6080600@behnel.de>	<4DB6DBCF.2020301@behnel.de>
	<BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>
Message-ID: <4DB7C9B4.9040004@behnel.de>

Robert Bradshaw, 26.04.2011 19:52:
> On Tue, Apr 26, 2011 at 7:50 AM, Stefan Behnel wrote:
>> Stefan Behnel, 15.04.2011 22:20:
>>>
>>> Stefan Behnel, 11.04.2011 15:08:
>>>>
>>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
>>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>>>> works out well, we may have it up in a couple of days.
>>>
>>> ... or maybe not. It may take a little longer due to lack of time on his
>>> side.
>>>
>>>
>>>> I would expect that Cython won't be a big winner in this game, given that
>>>> it will only compile plain untyped Python code. It's also going to fail
>>>> entirely in some of the benchmarks. But I think it's worth having it up
>>>> there, simply as a way for us to see where we are performance-wise and to
>>>> get quick (nightly) feed-back about optimisations we try. The benchmark
>>>> suite is also a nice set of real-world Python code that will allow us to
>>>> find compliance issues.
>>>
>>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got
>>> at least some of the benchmarks running. Note that they are actually
>>> simple
>>> ones, only a single module. Basically all complex benchmarks fail due to
>>> known bugs, such as Cython def functions not accepting attribute
>>> assignments (e.g. on wrapping). There's also a problem with code that uses
>>> platform specific names conditionally, such as WindowsError when running
>>> on
>>> Windows. Cython complains about non-builtin names here. I'm considering to
>>> turn that into a visible warning instead of an error, so that the name
>>> would instead be looked up dynamically to let the code fail at runtime
>>> *iff* it reaches the name lookup.
>>>
>>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled,
>>> although that doesn't even seem to make that a big difference. The
>>> baseline is a self-compiled Python 2.7.1+ (about a month old).
>>
>> [numbers stripped]
>>
>> And here's the shiny graph:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html
>>
>> It gets automatically rebuilt by this Hudson job:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/
>
> Cool. Any history stored/displayed?

No. Also, the variances are rather large depending on the load of the 
machine. Hudson/Jenkins and all its subprocesses run with a high CPU 
niceness and I/O niceness, so don't expect reproducible results.

Actually, if we want a proper history, I'd suggest a separate codespeed 
installation somewhere.

Stefan

From mantihor at gmail.com  Wed Apr 27 17:45:30 2011
From: mantihor at gmail.com (Bogdan Opanchuk)
Date: Thu, 28 Apr 2011 01:45:30 +1000
Subject: [Cython] Compiler error when @staticmethod is used in cdef class
Message-ID: <BANLkTinjhxJOjxyiFLc4dVFL+m6GJ5pjvQ@mail.gmail.com>

Hello,

I am using Cython 0.14.1 with Python 2.7.1 on OSX 10.6.7

The following code:
---
cdef class Test:
	@staticmethod
	def func():
		print "Static method"
---
gives error when being cythoned: "Method func has wrong number of
arguments (0 declared, 1 or more expected)"

And if one adds some parameters to this method (like "def func(a,
b):"), cython crashes with exception:
---
... long stack trace ...
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Cython-0.14-py2.7-macosx-10.6-intel.egg/Cython/Compiler/TypeSlots.py",
line 100, in fixed_arg_type
    return self.format_map[self.fixed_arg_format[i]]
KeyError: 'T'
---

So, is there a way to create static methods in cdef classes? This
workaround seems to work, but it is a bit ugly:
---
def func(a, b):
	print "Static method"

cdef class Test:
	func = func
---

Best regards,
Bogdan

From dalcinl at gmail.com  Wed Apr 27 18:51:42 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 Apr 2011 13:51:42 -0300
Subject: [Cython] Compiler error when @staticmethod is used in cdef class
In-Reply-To: <BANLkTinjhxJOjxyiFLc4dVFL+m6GJ5pjvQ@mail.gmail.com>
References: <BANLkTinjhxJOjxyiFLc4dVFL+m6GJ5pjvQ@mail.gmail.com>
Message-ID: <BANLkTi=RjDS-O=-Zk1q0eVjSKy6L+Syo1Q@mail.gmail.com>

On 27 April 2011 12:45, Bogdan Opanchuk <mantihor at gmail.com> wrote:
> Hello,
>
> I am using Cython 0.14.1 with Python 2.7.1 on OSX 10.6.7
>
> The following code:
> ---
> cdef class Test:
> ? ? ? ?@staticmethod
> ? ? ? ?def func():
> ? ? ? ? ? ? ? ?print "Static method"
> ---
> gives error when being cythoned: "Method func has wrong number of
> arguments (0 declared, 1 or more expected)"
>
> And if one adds some parameters to this method (like "def func(a,
> b):"), cython crashes with exception:
> ---
> ... long stack trace ...
> ?File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Cython-0.14-py2.7-macosx-10.6-intel.egg/Cython/Compiler/TypeSlots.py",
> line 100, in fixed_arg_type
> ? ?return self.format_map[self.fixed_arg_format[i]]
> KeyError: 'T'
> ---
>
> So, is there a way to create static methods in cdef classes? This
> workaround seems to work, but it is a bit ugly:
> ---
> def func(a, b):
> ? ? ? ?print "Static method"
>
> cdef class Test:
> ? ? ? ?func = func
> ---
>
> Best regards,
> Bogdan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

This issue has bugged my from the fist day I've started to use Cython.
However, never got time to look for a fix. Moreover, I think using
@classmethod is better than @staticmethod. @classmethod gives you some
extra context (the class) that could be useful in the case of
inheritance, specially if your method is a factory function.

So, as a workaround, try to use @classmethod, from an API point of
view they are very similar, and you can take advantage of @classmethod
extra context in the future.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Wed Apr 27 19:08:44 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 27 Apr 2011 10:08:44 -0700
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DB7C9B4.9040004@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
	<4DB6DBCF.2020301@behnel.de>
	<BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>
	<4DB7C9B4.9040004@behnel.de>
Message-ID: <BANLkTim6L4WQDPzLNrNodnrWXKm9FMbq=g@mail.gmail.com>

On Wed, Apr 27, 2011 at 12:45 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 26.04.2011 19:52:
>>
>> On Tue, Apr 26, 2011 at 7:50 AM, Stefan Behnel wrote:
>>>
>>> Stefan Behnel, 15.04.2011 22:20:
>>>>
>>>> Stefan Behnel, 11.04.2011 15:08:
>>>>>
>>>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get
>>>>> Cython
>>>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it
>>>>> works out well, we may have it up in a couple of days.
>>>>
>>>> ... or maybe not. It may take a little longer due to lack of time on his
>>>> side.
>>>>
>>>>
>>>>> I would expect that Cython won't be a big winner in this game, given
>>>>> that
>>>>> it will only compile plain untyped Python code. It's also going to fail
>>>>> entirely in some of the benchmarks. But I think it's worth having it up
>>>>> there, simply as a way for us to see where we are performance-wise and
>>>>> to
>>>>> get quick (nightly) feed-back about optimisations we try. The benchmark
>>>>> suite is also a nice set of real-world Python code that will allow us
>>>>> to
>>>>> find compliance issues.
>>>>
>>>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and
>>>> got
>>>> at least some of the benchmarks running. Note that they are actually
>>>> simple
>>>> ones, only a single module. Basically all complex benchmarks fail due to
>>>> known bugs, such as Cython def functions not accepting attribute
>>>> assignments (e.g. on wrapping). There's also a problem with code that
>>>> uses
>>>> platform specific names conditionally, such as WindowsError when running
>>>> on
>>>> Windows. Cython complains about non-builtin names here. I'm considering
>>>> to
>>>> turn that into a visible warning instead of an error, so that the name
>>>> would instead be looked up dynamically to let the code fail at runtime
>>>> *iff* it reaches the name lookup.
>>>>
>>>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled,
>>>> although that doesn't even seem to make that a big difference. The
>>>> baseline is a self-compiled Python 2.7.1+ (about a month old).
>>>
>>> [numbers stripped]
>>>
>>> And here's the shiny graph:
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html
>>>
>>> It gets automatically rebuilt by this Hudson job:
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/
>>
>> Cool. Any history stored/displayed?
>
> No. Also, the variances are rather large depending on the load of the
> machine.

Of course, that would make the history rather than a snapshot all the
more useful.

> Hudson/Jenkins and all its subprocesses run with a high CPU
> niceness and I/O niceness, so don't expect reproducible results.
>
> Actually, if we want a proper history, I'd suggest a separate codespeed
> installation somewhere.

Makes sense. How many CPU-hours does it take? If it's not to
intensive, we could probably run it, say, daily as a normal-priority
job.

- Robert

From stefan_ml at behnel.de  Wed Apr 27 21:32:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 27 Apr 2011 21:32:45 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <BANLkTim6L4WQDPzLNrNodnrWXKm9FMbq=g@mail.gmail.com>
References: <4DA2FD5E.5090704@behnel.de>
	<4DA8A8A3.6080600@behnel.de>	<4DB6DBCF.2020301@behnel.de>	<BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>	<4DB7C9B4.9040004@behnel.de>
	<BANLkTim6L4WQDPzLNrNodnrWXKm9FMbq=g@mail.gmail.com>
Message-ID: <4DB86F5D.2080603@behnel.de>

Robert Bradshaw, 27.04.2011 19:08:
> On Wed, Apr 27, 2011 at 12:45 AM, Stefan Behnel wrote:
>> Actually, if we want a proper history, I'd suggest a separate codespeed
>> installation somewhere.
>
> Makes sense. How many CPU-hours does it take?

Including the Cython build, it's more like 25 minutes currently, given that 
we only support a smaller part of the benchmark suite. It will obviously 
take longer when we start supporting the larger benchmarks, such as Django 
templates or Twisted.


> If it's not to
> intensive, we could probably run it, say, daily as a normal-priority
> job.

We could certainly do that for now, and check again when we see that it 
starts running substantially longer.

Stefan

From stefan_ml at behnel.de  Thu Apr 28 15:08:08 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Apr 2011 15:08:08 +0200
Subject: [Cython] HTML Coverage reports
Message-ID: <4DB966B8.2030009@behnel.de>

Hi,

I enabled HTML coverage report generation for the test suite. It's now 
available from the "HTML Coverage Reports" link of the coverage job:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-coverage/doclinks/1/

 From a quick look, the test coverage seems not too bad and we are mostly 
lacking error tests. However, especially the branch coverage analysis hints 
at a couple of cases that are worth looking into. Compile time value 
calculation is an obvious area that is worth improving, and there are 
clearly some important untested code sections in ExprNodes.py:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-coverage/doclinks/1/Cython_Compiler_ExprNodes.html

Stefan

From markflorisson88 at gmail.com  Thu Apr 28 21:48:13 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 28 Apr 2011 21:48:13 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
Message-ID: <BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>

On 26 April 2011 20:05, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Apr 26, 2011 at 8:18 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 26 April 2011 16:43, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 26.04.2011 16:23:
>>>>
>>>> I've been working a bit on fused types
>>>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>>>> generate code for every permutation of specific types. Currently it
>>>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>>>> PyrexTypes.best_match() to find the function with the best signature,
>>>> but it doesn't seem to take care of coercions, e.g. it calls
>>>> 'assignable_from' on the dst_type (e.g. char *), but for
>>>> BuiltinObjectType (a str) this returns false.
>>>
>>> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
>>> "bytes", though.
>>>
>>> http://wiki.cython.org/enhancements/stringliterals
>>
>> Right, I see, so the thing is that I was using string literals which
>> should be inferred as the bytes type when they are assigned to a char
>> *. The thing is that because the type is fused, the argument may be
>> anything, so I was hoping best_match would figure it out for me.
>> Apparently it doesn't, and indeed, this example doesn't work:
>>
>> cdef extern from "Foo.h":
>> ? ?cdef cppclass Foo:
>> ? ? ? ?Foo(char *)
>> ? ? ? ?Foo(int)
>>
>> cdef char *foo = "foo"
>> cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
>> method found")
>> cdef Foo* bar = new Foo(foo) ? # <- this works
>>
>> So that's pretty lame, I think I should fix that.
>
> Agreed.
>
>>>> Why is this the case,
>>>> don't calls to overloaded C++ methods need to dispatch properly here
>>>> also?
>>>
>>> If this doesn't work, I assume it just isn't implemented.
>>>
>>>
>>>> Other issues are public and api declarations. So should we support
>>>> that at all? We could define a macro that calls a function that does
>>>> the dispatch. So e.g. for this
>>>>
>>>> ctypedef cython.fused_type(typeA, typeB) dtype
>>>>
>>>> cdef func(dtype x):
>>>> ? ? ...
>>>>
>>>> we would get two generated functions, say, __pyx_typeA_func and
>>>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>>>> something that then substitutes __pyx_get_func(#dtype), where
>>>> __pyx_get_func returns the pointer to the right function based on the
>>>> type names. I'm not sure we should support it, right now I just put
>>>> the mangled names in the header. At least the cdef functions will be
>>>> sharable between Cython implementation files.
>>>
>>> I'm fine with explicitly forbidding this for now. It may eventually work for
>>> Python object types where we can properly dispatch, but it won't easily work
>>> for C types. It may work in C++, though.
>>>
>>
>> Ok, will do.
>
> For the moment, putting mangled names in the header should be fine. A
> macro might make sense in the long term.
>
> Somewhat orthogonal, it could makes sense to do some dispatching on
> type for cpdef functions.
>
>>>> I also noticed that for cdef functions with optional argument it gets
>>>> a struct as argument, but this struct doesn't seem to end up in the
>>>> header file when the function is declared public. I believe that also
>>>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>>>> there when used to type a cdef function's arguments. Should that be
>>>> fixed?
>>>
>>> No, I think this should also become a compiler error. These functions are
>>> not meant to be called from C code. It's a Cython convenience feature. As
>>> long as it's not correctly supported on both ends of the publicly exported
>>> C-API, it's best to keep users from using it at all.
>>
>> Ok, I'll try to make Cython issue an error for these cases.
>
> +1
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

So I fixed all that, but I'm currently wondering about the proposed
cython.typeof(). I believe it currently returns a string with the type
name, and not the type itself. So I think it would be inconsistent to
suddenly start allowing comparison with 'is' and 'isinstance' and
such.

I'm also wondering if it would be useful to allow actual type
retrieval, which could be used in declarations and casts. For instance
consider fusing two structs with the same attribute name but different
attribute types. Perhaps in your code you want to introduce a variable
compatible with such a type, e.g. consider this:

ctypdef struct A:
    int attrib

ctypedef struct B:
    char *attrib

ctypedef cython.fused_type(A, B) struct_t

cdef func(struct_t mystruct, int i):
    cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
    ...

What do you think?

From stefan_ml at behnel.de  Thu Apr 28 21:58:14 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Apr 2011 21:58:14 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
Message-ID: <4DB9C6D6.6010401@behnel.de>

mark florisson, 28.04.2011 21:48:
> I'm currently wondering about the proposed
> cython.typeof(). I believe it currently returns a string with the type
> name, and not the type itself. So I think it would be inconsistent to
> suddenly start allowing comparison with 'is' and 'isinstance' and
> such.
>
> I'm also wondering if it would be useful to allow actual type
> retrieval, which could be used in declarations and casts. For instance
> consider fusing two structs with the same attribute name but different
> attribute types. Perhaps in your code you want to introduce a variable
> compatible with such a type, e.g. consider this:
>
> ctypdef struct A:
>      int attrib
>
> ctypedef struct B:
>      char *attrib
>
> ctypedef cython.fused_type(A, B) struct_t
>
> cdef func(struct_t mystruct, int i):
>      cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i

What's wrong with type() ?

Stefan

From markflorisson88 at gmail.com  Thu Apr 28 22:10:06 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 28 Apr 2011 22:10:06 +0200
Subject: [Cython] Fused Types
In-Reply-To: <4DB9C6D6.6010401@behnel.de>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<4DB9C6D6.6010401@behnel.de>
Message-ID: <BANLkTi=EZBoSNAEQ_EZxijyfQusM0hWeiA@mail.gmail.com>

On 28 April 2011 21:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.04.2011 21:48:
>>
>> I'm currently wondering about the proposed
>> cython.typeof(). I believe it currently returns a string with the type
>> name, and not the type itself. So I think it would be inconsistent to
>> suddenly start allowing comparison with 'is' and 'isinstance' and
>> such.
>>
>> I'm also wondering if it would be useful to allow actual type
>> retrieval, which could be used in declarations and casts. For instance
>> consider fusing two structs with the same attribute name but different
>> attribute types. Perhaps in your code you want to introduce a variable
>> compatible with such a type, e.g. consider this:
>>
>> ctypdef struct A:
>> ? ? int attrib
>>
>> ctypedef struct B:
>> ? ? char *attrib
>>
>> ctypedef cython.fused_type(A, B) struct_t
>>
>> cdef func(struct_t mystruct, int i):
>> ? ? cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>
> What's wrong with type() ?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

When you call type you kind of expect it to call the Python builtin
type(), which means it will coerce your C type to a Python object, on
which it will call type(). So if you change those semantics suddenly
for C types, I think we'd break people's code.

From markflorisson88 at gmail.com  Thu Apr 28 22:11:29 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 28 Apr 2011 22:11:29 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=EZBoSNAEQ_EZxijyfQusM0hWeiA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<4DB9C6D6.6010401@behnel.de>
	<BANLkTi=EZBoSNAEQ_EZxijyfQusM0hWeiA@mail.gmail.com>
Message-ID: <BANLkTi=JSFCnDgpH=2fF-MyzJpTcSfwSaQ@mail.gmail.com>

On 28 April 2011 22:10, mark florisson <markflorisson88 at gmail.com> wrote:
> On 28 April 2011 21:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 28.04.2011 21:48:
>>>
>>> I'm currently wondering about the proposed
>>> cython.typeof(). I believe it currently returns a string with the type
>>> name, and not the type itself. So I think it would be inconsistent to
>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>> such.
>>>
>>> I'm also wondering if it would be useful to allow actual type
>>> retrieval, which could be used in declarations and casts. For instance
>>> consider fusing two structs with the same attribute name but different
>>> attribute types. Perhaps in your code you want to introduce a variable
>>> compatible with such a type, e.g. consider this:
>>>
>>> ctypdef struct A:
>>> ? ? int attrib
>>>
>>> ctypedef struct B:
>>> ? ? char *attrib
>>>
>>> ctypedef cython.fused_type(A, B) struct_t
>>>
>>> cdef func(struct_t mystruct, int i):
>>> ? ? cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>
>> What's wrong with type() ?
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> When you call type you kind of expect it to call the Python builtin
> type(), which means it will coerce your C type to a Python object, on
> which it will call type(). So if you change those semantics suddenly
> for C types, I think we'd break people's code.
>

Also, you'd want this to work for Python types too, for instance for
two differently typed extension type attributes

From robertwb at math.washington.edu  Thu Apr 28 22:12:08 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 Apr 2011 13:12:08 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
Message-ID: <BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>

On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
<markflorisson88 at gmail.com> wrote:

> So I fixed all that, but I'm currently wondering about the proposed
> cython.typeof(). I believe it currently returns a string with the type
> name, and not the type itself.

Yes. This is just because there's not really anything better to return
at this point. We should "fix" this at some point in the future.

> So I think it would be inconsistent to
> suddenly start allowing comparison with 'is' and 'isinstance' and
> such.

I'm open to other suggestions, but would like an expression that
resolves at compile time to true/false (and we need to do branch
pruning on it). Note that type() is not good enough, because it has
different semantics, i.e.

    cdef object o = []
    typeof(o), type(o)

so lets not touch that one.

> I'm also wondering if it would be useful to allow actual type
> retrieval, which could be used in declarations and casts. For instance
> consider fusing two structs with the same attribute name but different
> attribute types. Perhaps in your code you want to introduce a variable
> compatible with such a type, e.g. consider this:
>
> ctypdef struct A:
> ? ?int attrib
>
> ctypedef struct B:
> ? ?char *attrib
>
> ctypedef cython.fused_type(A, B) struct_t
>
> cdef func(struct_t mystruct, int i):
> ? ?cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
> ? ?...
>
> What do you think?

Yes, I was thinking that would be supported as well.

- Robert

From markflorisson88 at gmail.com  Thu Apr 28 22:31:02 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 28 Apr 2011 22:31:02 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
Message-ID: <BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>

On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>
>> So I fixed all that, but I'm currently wondering about the proposed
>> cython.typeof(). I believe it currently returns a string with the type
>> name, and not the type itself.
>
> Yes. This is just because there's not really anything better to return
> at this point. We should "fix" this at some point in the future.
>
>> So I think it would be inconsistent to
>> suddenly start allowing comparison with 'is' and 'isinstance' and
>> such.
>
> I'm open to other suggestions, but would like an expression that
> resolves at compile time to true/false (and we need to do branch
> pruning on it). Note that type() is not good enough, because it has
> different semantics, i.e.
>
> ? ?cdef object o = []
> ? ?typeof(o), type(o)
>
> so lets not touch that one.

Right, so for fused types I don't mind string comparison with
cython.typeof(), but retrieval of the actual type for casts and
declaration remains open. I'd be fine with something like
cython.gettype().

On the other hand, the user could already do this thing by manually
declaring another fused type that would perhaps be resolved based on
its usage. e.g. for my original example:

ctypedef char *string_t
ctypedef cython.fused_type(int, string_t) attrib_t

cdef func(struct_t mystruct, int i):
    cdef attrib_t var = mystruct.attrib + i

Is this something that should be supported anyway? Currently fused
types can only be used when appearing as argument types (in the
function body and as return value).

>> I'm also wondering if it would be useful to allow actual type
>> retrieval, which could be used in declarations and casts. For instance
>> consider fusing two structs with the same attribute name but different
>> attribute types. Perhaps in your code you want to introduce a variable
>> compatible with such a type, e.g. consider this:
>>
>> ctypdef struct A:
>> ? ?int attrib
>>
>> ctypedef struct B:
>> ? ?char *attrib
>>
>> ctypedef cython.fused_type(A, B) struct_t
>>
>> cdef func(struct_t mystruct, int i):
>> ? ?cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>> ? ?...
>>
>> What do you think?
>
> Yes, I was thinking that would be supported as well.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Thu Apr 28 22:32:17 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 Apr 2011 13:32:17 -0700
Subject: [Cython] [cython-users] using generators
In-Reply-To: <4DB90C5B.4030607@behnel.de>
References: <3fcf23f2-2b27-4b5b-ad15-ee419ac9d4a8@v10g2000yqn.googlegroups.com>
	<BANLkTim3UMoJfQPgcMbY=CQPCBwwSjZoLw@mail.gmail.com>
	<4DB90C5B.4030607@behnel.de>
Message-ID: <BANLkTi=CatVwFGfvhF=CBD-XHbvaUGtWEw@mail.gmail.com>

On Wed, Apr 27, 2011 at 11:42 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 27.04.2011 22:41:
>>
>> On Wed, Apr 27, 2011 at 1:19 PM, Mad wrote:
>>>
>>> Hi list,
>>> i just read, that generators are supported in cython 0.15 - great news
>>> and brw, manny thanks. but is it somehow possible to use them with
>>> cython 0.14 too?
>>> It would be nice, if there's a patch (or the patch of ?0.14 -> ?0.15
>>> could be applied to the 0.14 source) to get them working, ?If this is
>>> not possible,
>>
>> No, that's not possible--there's a lot that went on between here and
>> there (it's not something you can just patch in). Any reason you're
>> trying to stick with 0.14?
>>
>>> is 0.15 in a usable state? and how could i get it?
>>
>> It hasn't been released yet, and the development branch (on github) is
>> relatively stable but not quite ready yet. As for the state of
>> generators, Vitja and Stefan can comment more than I can, but there
>> are still some corner cases with exception throwing that don't quite
>> work like Python, and there's also a regression with respect to inline
>> generators. I'm not sure if either of these is enough to be a blocker
>> for a release.
>
> Don't think they are. As you said, the exception handling is for corner
> cases (which exceptions should always be anyway) and I consider the
> behaviour "close enough" now to not be a major issue. Vitja can comment on
> this, too. The only case where I know that it currently diverges is
> exception chaining in Python 3. And that should be fixable.
>
> Inline generators will require some work to become usable again.
> Specifically, the expression scope that I had previously implemented for
> comprehensions (and reused for generator expressions) doesn't work with the
> then already created function scope of the generator. This needs to be fixed
> somehow, either by moving the declarations over to the expression scope or
> by directly using the function scope instead. In the latter case, it would
> also be nice to merge this with the scoped comprehensions as well.
>
> So, agreed that these are regressions, but certainly less important than
> getting generators and coroutines out in the wild.

OK, I've added the inline generator tests to the bug list so we can
see where we stand. I'd like to get Sage back up and compiling as
well. Any thoughts on a release timeline? Any other known
instabilities?

- Robert

From markflorisson88 at gmail.com  Thu Apr 28 23:29:06 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 28 Apr 2011 23:29:06 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
Message-ID: <BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>

On 28 April 2011 22:31, mark florisson <markflorisson88 at gmail.com> wrote:
> On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>
>>> So I fixed all that, but I'm currently wondering about the proposed
>>> cython.typeof(). I believe it currently returns a string with the type
>>> name, and not the type itself.
>>
>> Yes. This is just because there's not really anything better to return
>> at this point. We should "fix" this at some point in the future.
>>
>>> So I think it would be inconsistent to
>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>> such.
>>
>> I'm open to other suggestions, but would like an expression that
>> resolves at compile time to true/false (and we need to do branch
>> pruning on it). Note that type() is not good enough, because it has
>> different semantics, i.e.
>>
>> ? ?cdef object o = []
>> ? ?typeof(o), type(o)
>>
>> so lets not touch that one.
>
> Right, so for fused types I don't mind string comparison with
> cython.typeof(), but retrieval of the actual type for casts and
> declaration remains open. I'd be fine with something like
> cython.gettype().

It seems that this isn't optimized yet, but it looks to me like it
wouldn't be very hard to do so. At least == and != could be resolved
at compile time if the other operand is a string literal.

> On the other hand, the user could already do this thing by manually
> declaring another fused type that would perhaps be resolved based on
> its usage. e.g. for my original example:
>
> ctypedef char *string_t
> ctypedef cython.fused_type(int, string_t) attrib_t
>
> cdef func(struct_t mystruct, int i):
> ? ?cdef attrib_t var = mystruct.attrib + i
>
> Is this something that should be supported anyway? Currently fused
> types can only be used when appearing as argument types (in the
> function body and as return value).
>
>>> I'm also wondering if it would be useful to allow actual type
>>> retrieval, which could be used in declarations and casts. For instance
>>> consider fusing two structs with the same attribute name but different
>>> attribute types. Perhaps in your code you want to introduce a variable
>>> compatible with such a type, e.g. consider this:
>>>
>>> ctypdef struct A:
>>> ? ?int attrib
>>>
>>> ctypedef struct B:
>>> ? ?char *attrib
>>>
>>> ctypedef cython.fused_type(A, B) struct_t
>>>
>>> cdef func(struct_t mystruct, int i):
>>> ? ?cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>> ? ?...
>>>
>>> What do you think?
>>
>> Yes, I was thinking that would be supported as well.
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

From robertwb at math.washington.edu  Thu Apr 28 23:30:52 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 Apr 2011 14:30:52 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
Message-ID: <BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>

On Thu, Apr 28, 2011 at 1:31 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>
>>> So I fixed all that, but I'm currently wondering about the proposed
>>> cython.typeof(). I believe it currently returns a string with the type
>>> name, and not the type itself.
>>
>> Yes. This is just because there's not really anything better to return
>> at this point. We should "fix" this at some point in the future.
>>
>>> So I think it would be inconsistent to
>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>> such.
>>
>> I'm open to other suggestions, but would like an expression that
>> resolves at compile time to true/false (and we need to do branch
>> pruning on it). Note that type() is not good enough, because it has
>> different semantics, i.e.
>>
>> ? ?cdef object o = []
>> ? ?typeof(o), type(o)
>>
>> so lets not touch that one.
>
> Right, so for fused types I don't mind string comparison with
> cython.typeof(), but retrieval of the actual type for casts and
> declaration remains open. I'd be fine with something like
> cython.gettype().

I'd rather re-use typeof here than introduce yet another special
method. On the other hand, allowing parentheses in a type declaration
requires complicating the syntax even more.

I'm not sure this is needed, couldn't one do

    cdef foo(fused_type a):
        cdef fused_type b = a + <fused_type>func()

and all instances of fused_type get specialized in the body.

> On the other hand, the user could already do this thing by manually
> declaring another fused type that would perhaps be resolved based on
> its usage. e.g. for my original example:
>
> ctypedef char *string_t
> ctypedef cython.fused_type(int, string_t) attrib_t
>
> cdef func(struct_t mystruct, int i):
> ? ?cdef attrib_t var = mystruct.attrib + i
>
> Is this something that should be supported anyway?

OK, I take back what I said, I was looking at the RHS, not the LHS. If
one needs to specialize in this manner, explicitly creating two
branches should typically be enough. The same for casting. The one
exception (perhaps) is "my_fused_type complex." Otherwise it's
starting to feel too much like C++ template magic and complexity for
little additional benefit.

> Currently fused
> types can only be used when appearing as argument types (in the
> function body and as return value).

>>> I'm also wondering if it would be useful to allow actual type
>>> retrieval, which could be used in declarations and casts. For instance
>>> consider fusing two structs with the same attribute name but different
>>> attribute types. Perhaps in your code you want to introduce a variable
>>> compatible with such a type, e.g. consider this:
>>>
>>> ctypdef struct A:
>>> ? ?int attrib
>>>
>>> ctypedef struct B:
>>> ? ?char *attrib
>>>
>>> ctypedef cython.fused_type(A, B) struct_t
>>>
>>> cdef func(struct_t mystruct, int i):
>>> ? ?cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>> ? ?...
>>>
>>> What do you think?
>>
>> Yes, I was thinking that would be supported as well.
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Thu Apr 28 23:59:13 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 Apr 2011 14:59:13 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
Message-ID: <BANLkTi=TPJB2-3H58FXxocvJSMgeMR9iSA@mail.gmail.com>

On Thu, Apr 28, 2011 at 2:29 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 28 April 2011 22:31, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>
>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>> cython.typeof(). I believe it currently returns a string with the type
>>>> name, and not the type itself.
>>>
>>> Yes. This is just because there's not really anything better to return
>>> at this point. We should "fix" this at some point in the future.
>>>
>>>> So I think it would be inconsistent to
>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>> such.
>>>
>>> I'm open to other suggestions, but would like an expression that
>>> resolves at compile time to true/false (and we need to do branch
>>> pruning on it). Note that type() is not good enough, because it has
>>> different semantics, i.e.
>>>
>>> ? ?cdef object o = []
>>> ? ?typeof(o), type(o)
>>>
>>> so lets not touch that one.
>>
>> Right, so for fused types I don't mind string comparison with
>> cython.typeof(), but retrieval of the actual type for casts and
>> declaration remains open. I'd be fine with something like
>> cython.gettype().
>
> It seems that this isn't optimized yet, but it looks to me like it
> wouldn't be very hard to do so. At least == and != could be resolved
> at compile time if the other operand is a string literal.

Yes. We could consider supporting "typeof(x) is typeof(double)" or
even "typeof(int*)" etc. as well to not tie ourselves to strings. Or
perhaps some other syntax we haven't thought of. Alternatively, it
would be nice if it involved the literal fused_type as that's what
we're really branching on, e.g.

ctypedef cython.fused_type(A, B) AorB

cdef foo(AorB x):
    if AorB[A]:   # or .A or something
        ...

it's hard to come up with something that plays nicely with pointer types.

- Robert

From stefan_ml at behnel.de  Fri Apr 29 04:09:56 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 29 Apr 2011 04:09:56 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
Message-ID: <4DBA1DF4.6010001@behnel.de>

mark florisson, 28.04.2011 23:29:
> On 28 April 2011 22:31, mark florisson wrote:
>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>
>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>> cython.typeof(). I believe it currently returns a string with the type
>>>> name, and not the type itself.
>>>
>>> Yes. This is just because there's not really anything better to return
>>> at this point. We should "fix" this at some point in the future.
>>>
>>>> So I think it would be inconsistent to
>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>> such.
>>>
>>> I'm open to other suggestions, but would like an expression that
>>> resolves at compile time to true/false (and we need to do branch
>>> pruning on it). Note that type() is not good enough, because it has
>>> different semantics, i.e.
>>>
>>>     cdef object o = []
>>>     typeof(o), type(o)
>>>
>>> so lets not touch that one.
>>
>> Right, so for fused types I don't mind string comparison with
>> cython.typeof(), but retrieval of the actual type for casts and
>> declaration remains open. I'd be fine with something like
>> cython.gettype().
>
> It seems that this isn't optimized yet, but it looks to me like it
> wouldn't be very hard to do so. At least == and != could be resolved
> at compile time if the other operand is a string literal.

Well, the obvious place where this would happen for free would be constant 
folding. But that runs *way* before type analysis, so all it sees is the 
typeof() call, not the string.

Optimising this would thus basically require a second (simpler?) constant 
folding step, or at least an additional hook during (or after) type 
analysis that does the comparison. I wouldn't mind too much, given that 
type analysis can end up injecting some rather handy information into the 
tree. So a second (even complete) constant folding step *after* type 
analysis may still introduce some nice optimisations.

We may actually consider splitting constant folding into two steps - one 
that calculates the results as early as possible (and maybe does only 
safe&obvious tree replacements, e.g. boolean values, strings, etc.) and a 
second step that replaces subtrees by constant nodes once it knows the 
final type of the result.

Stefan

From robertwb at math.washington.edu  Fri Apr 29 06:32:33 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 Apr 2011 21:32:33 -0700
Subject: [Cython] Fused Types
In-Reply-To: <4DBA1DF4.6010001@behnel.de>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<4DBA1DF4.6010001@behnel.de>
Message-ID: <BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>

On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.04.2011 23:29:
>>
>> On 28 April 2011 22:31, mark florisson wrote:
>>>
>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>
>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>
>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>> name, and not the type itself.
>>>>
>>>> Yes. This is just because there's not really anything better to return
>>>> at this point. We should "fix" this at some point in the future.
>>>>
>>>>> So I think it would be inconsistent to
>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>> such.
>>>>
>>>> I'm open to other suggestions, but would like an expression that
>>>> resolves at compile time to true/false (and we need to do branch
>>>> pruning on it). Note that type() is not good enough, because it has
>>>> different semantics, i.e.
>>>>
>>>> ? ?cdef object o = []
>>>> ? ?typeof(o), type(o)
>>>>
>>>> so lets not touch that one.
>>>
>>> Right, so for fused types I don't mind string comparison with
>>> cython.typeof(), but retrieval of the actual type for casts and
>>> declaration remains open. I'd be fine with something like
>>> cython.gettype().
>>
>> It seems that this isn't optimized yet, but it looks to me like it
>> wouldn't be very hard to do so. At least == and != could be resolved
>> at compile time if the other operand is a string literal.
>
> Well, the obvious place where this would happen for free would be constant
> folding. But that runs *way* before type analysis, so all it sees is the
> typeof() call, not the string.

Actually, to support code like

ctypedef cython.fused_type(object, double) my_fused_type

cdef foo(my_fused_type x):
    if my_fused_type is object:
        print x.attr
    else:
        cdef my_fused_type *ptr = &x

we need to resolve this branch and prune "dead code" before type
analysis, so I'm thinking it may need to be a special phase, perhaps
part of the fused-type-specialization phase. Here's another idea:

cdef foo(numeric x):
    if numeric in floating:
        ...
    elif numeric is long:
        ...
    else:
        ...
    print numeric is double   # after the specialization pass, this
would be a literal True/False node.

Again, this would all be resolved before type analysis. In terms of
syntactically supporting pointers, we could either support "fused_type
is typeof(void*)" or require typedefs for types not representable by
an ExprNode. (This would make declaring fused types syntactically
simpler as well...) I prefer the latter.

- Robert


> Optimising this would thus basically require a second (simpler?) constant
> folding step, or at least an additional hook during (or after) type analysis
> that does the comparison. I wouldn't mind too much, given that type analysis
> can end up injecting some rather handy information into the tree. So a
> second (even complete) constant folding step *after* type analysis may still
> introduce some nice optimisations.
>
> We may actually consider splitting constant folding into two steps - one
> that calculates the results as early as possible (and maybe does only
> safe&obvious tree replacements, e.g. boolean values, strings, etc.) and a
> second step that replaces subtrees by constant nodes once it knows the final
> type of the result.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Fri Apr 29 10:23:55 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 10:23:55 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
Message-ID: <BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>

On 28 April 2011 23:30, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 28, 2011 at 1:31 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>
>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>> cython.typeof(). I believe it currently returns a string with the type
>>>> name, and not the type itself.
>>>
>>> Yes. This is just because there's not really anything better to return
>>> at this point. We should "fix" this at some point in the future.
>>>
>>>> So I think it would be inconsistent to
>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>> such.
>>>
>>> I'm open to other suggestions, but would like an expression that
>>> resolves at compile time to true/false (and we need to do branch
>>> pruning on it). Note that type() is not good enough, because it has
>>> different semantics, i.e.
>>>
>>> ? ?cdef object o = []
>>> ? ?typeof(o), type(o)
>>>
>>> so lets not touch that one.
>>
>> Right, so for fused types I don't mind string comparison with
>> cython.typeof(), but retrieval of the actual type for casts and
>> declaration remains open. I'd be fine with something like
>> cython.gettype().
>
> I'd rather re-use typeof here than introduce yet another special
> method. On the other hand, allowing parentheses in a type declaration
> requires complicating the syntax even more.
>
> I'm not sure this is needed, couldn't one do
>
> ? ?cdef foo(fused_type a):
> ? ? ? ?cdef fused_type b = a + <fused_type>func()
>
> and all instances of fused_type get specialized in the body.

Right, that is supported already.

>> On the other hand, the user could already do this thing by manually
>> declaring another fused type that would perhaps be resolved based on
>> its usage. e.g. for my original example:
>>
>> ctypedef char *string_t
>> ctypedef cython.fused_type(int, string_t) attrib_t
>>
>> cdef func(struct_t mystruct, int i):
>> ? ?cdef attrib_t var = mystruct.attrib + i
>>
>> Is this something that should be supported anyway?
>
> OK, I take back what I said, I was looking at the RHS, not the LHS. If
> one needs to specialize in this manner, explicitly creating two
> branches should typically be enough. The same for casting. The one
> exception (perhaps) is "my_fused_type complex." Otherwise it's
> starting to feel too much like C++ template magic and complexity for
> little additional benefit.

Ok, branching on the type sounds fine to me. It brings one problem
though: because you cannot declare the variables of your variable type
(the type of say, mystruct.attrib), you will need to do multiple
declarations outside of your branches. So in my example:

cdef func(struct_t mystruct, int i):
 ? ?cdef string_t string_var
    cdef int int_var

    if typeof(mystruct) is typeof(int):
        int_var = mystruct.attrib + i
        ...
    else:
       string_var = mystruct.attrib + i
       ...

But this is probably not a common case, so likely not an issue.

>> Currently fused
>> types can only be used when appearing as argument types (in the
>> function body and as return value).
>
>>>> I'm also wondering if it would be useful to allow actual type
>>>> retrieval, which could be used in declarations and casts. For instance
>>>> consider fusing two structs with the same attribute name but different
>>>> attribute types. Perhaps in your code you want to introduce a variable
>>>> compatible with such a type, e.g. consider this:
>>>>
>>>> ctypdef struct A:
>>>> ? ?int attrib
>>>>
>>>> ctypedef struct B:
>>>> ? ?char *attrib
>>>>
>>>> ctypedef cython.fused_type(A, B) struct_t
>>>>
>>>> cdef func(struct_t mystruct, int i):
>>>> ? ?cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>>> ? ?...
>>>>
>>>> What do you think?
>>>
>>> Yes, I was thinking that would be supported as well.
>>>
>>> - Robert
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Fri Apr 29 10:24:13 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 10:24:13 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=TPJB2-3H58FXxocvJSMgeMR9iSA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<BANLkTi=TPJB2-3H58FXxocvJSMgeMR9iSA@mail.gmail.com>
Message-ID: <BANLkTimUez26B4u53h0RoD4KyVm20PpcpA@mail.gmail.com>

On 28 April 2011 23:59, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 28, 2011 at 2:29 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 28 April 2011 22:31, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 28 April 2011 22:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>
>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>> name, and not the type itself.
>>>>
>>>> Yes. This is just because there's not really anything better to return
>>>> at this point. We should "fix" this at some point in the future.
>>>>
>>>>> So I think it would be inconsistent to
>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>> such.
>>>>
>>>> I'm open to other suggestions, but would like an expression that
>>>> resolves at compile time to true/false (and we need to do branch
>>>> pruning on it). Note that type() is not good enough, because it has
>>>> different semantics, i.e.
>>>>
>>>> ? ?cdef object o = []
>>>> ? ?typeof(o), type(o)
>>>>
>>>> so lets not touch that one.
>>>
>>> Right, so for fused types I don't mind string comparison with
>>> cython.typeof(), but retrieval of the actual type for casts and
>>> declaration remains open. I'd be fine with something like
>>> cython.gettype().
>>
>> It seems that this isn't optimized yet, but it looks to me like it
>> wouldn't be very hard to do so. At least == and != could be resolved
>> at compile time if the other operand is a string literal.
>
> Yes. We could consider supporting "typeof(x) is typeof(double)" or
> even "typeof(int*)" etc. as well to not tie ourselves to strings. Or
> perhaps some other syntax we haven't thought of. Alternatively, it
> would be nice if it involved the literal fused_type as that's what
> we're really branching on, e.g.
>
> ctypedef cython.fused_type(A, B) AorB
>
> cdef foo(AorB x):
> ? ?if AorB[A]: ? # or .A or something
> ? ? ? ?...
>
> it's hard to come up with something that plays nicely with pointer types.

I like the typeof(my_type_here). So if we support listing types in
typeof(), then you could also call typeof() on AorB to specialize.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Fri Apr 29 10:24:24 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 10:24:24 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<4DBA1DF4.6010001@behnel.de>
	<BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
Message-ID: <BANLkTimp3+iH7hSFZPH4RWHe8aNs0-rhZA@mail.gmail.com>

On 29 April 2011 06:32, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 28.04.2011 23:29:
>>>
>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>
>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>
>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>
>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>> name, and not the type itself.
>>>>>
>>>>> Yes. This is just because there's not really anything better to return
>>>>> at this point. We should "fix" this at some point in the future.
>>>>>
>>>>>> So I think it would be inconsistent to
>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>> such.
>>>>>
>>>>> I'm open to other suggestions, but would like an expression that
>>>>> resolves at compile time to true/false (and we need to do branch
>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>> different semantics, i.e.
>>>>>
>>>>> ? ?cdef object o = []
>>>>> ? ?typeof(o), type(o)
>>>>>
>>>>> so lets not touch that one.
>>>>
>>>> Right, so for fused types I don't mind string comparison with
>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>> declaration remains open. I'd be fine with something like
>>>> cython.gettype().
>>>
>>> It seems that this isn't optimized yet, but it looks to me like it
>>> wouldn't be very hard to do so. At least == and != could be resolved
>>> at compile time if the other operand is a string literal.
>>
>> Well, the obvious place where this would happen for free would be constant
>> folding. But that runs *way* before type analysis, so all it sees is the
>> typeof() call, not the string.
>
> Actually, to support code like
>
> ctypedef cython.fused_type(object, double) my_fused_type
>
> cdef foo(my_fused_type x):
> ? ?if my_fused_type is object:
> ? ? ? ?print x.attr
> ? ?else:
> ? ? ? ?cdef my_fused_type *ptr = &x
>
> we need to resolve this branch and prune "dead code" before type
> analysis, so I'm thinking it may need to be a special phase, perhaps
> part of the fused-type-specialization phase. Here's another idea:
>
> cdef foo(numeric x):
> ? ?if numeric in floating:
> ? ? ? ?...
> ? ?elif numeric is long:
> ? ? ? ?...
> ? ?else:
> ? ? ? ?...
> ? ?print numeric is double ? # after the specialization pass, this
> would be a literal True/False node.

Hmm, that one looks pretty ok. We could also do typeof(numeric) is
typeof(floating), and if any of the fused types are unresolved it
works like 'in' instead of 'is'. Although in that case people could
also write typeof(floating) is typeof(numeric) which would behave like
'in' with reversed operands :)

> Again, this would all be resolved before type analysis. In terms of
> syntactically supporting pointers, we could either support "fused_type
> is typeof(void*)" or require typedefs for types not representable by
> an ExprNode. (This would make declaring fused types syntactically
> simpler as well...) I prefer the latter.

Me too, cython.fused_type() currently just parses identifiers only.

> - Robert
>
>
>> Optimising this would thus basically require a second (simpler?) constant
>> folding step, or at least an additional hook during (or after) type analysis
>> that does the comparison. I wouldn't mind too much, given that type analysis
>> can end up injecting some rather handy information into the tree. So a
>> second (even complete) constant folding step *after* type analysis may still
>> introduce some nice optimisations.
>>
>> We may actually consider splitting constant folding into two steps - one
>> that calculates the results as early as possible (and maybe does only
>> safe&obvious tree replacements, e.g. boolean values, strings, etc.) and a
>> second step that replaces subtrees by constant nodes once it knows the final
>> type of the result.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From pav at iki.fi  Fri Apr 29 11:03:57 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 29 Apr 2011 09:03:57 +0000 (UTC)
Subject: [Cython] Fused Types
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
Message-ID: <ipduts$h1n$1@dough.gmane.org>

Fri, 29 Apr 2011 10:23:55 +0200, mark florisson wrote:
[clip]
> Ok, branching on the type sounds fine to me. It brings one problem
> though: because you cannot declare the variables of your variable type
> (the type of say, mystruct.attrib), you will need to do multiple
> declarations outside of your branches. So in my example:
> 
> cdef func(struct_t mystruct, int i):
>  ? ?cdef string_t string_var
>     cdef int int_var
> 
>     if typeof(mystruct) is typeof(int):
>         int_var = mystruct.attrib + i
>         ...
>     else:
>        string_var = mystruct.attrib + i
>        ...
> 
> But this is probably not a common case, so likely not an issue.

Are you planning to special-case the "real_t complex" syntax? Shooting 
from the sidelines, one more generic solution might be, e.g.,

	ctypedef cython.fused_type(A, B) struct_t
	ctypedef cython.fused_type(float, double, paired=struct_t) real_t
	ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t

and just restrict the specialization to cases that make sense.

-- 
Pauli Virtanen


From markflorisson88 at gmail.com  Fri Apr 29 11:30:19 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 11:30:19 +0200
Subject: [Cython] Fused Types
In-Reply-To: <ipduts$h1n$1@dough.gmane.org>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
Message-ID: <BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>

On 29 April 2011 11:03, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 29 Apr 2011 10:23:55 +0200, mark florisson wrote:
> [clip]
>> Ok, branching on the type sounds fine to me. It brings one problem
>> though: because you cannot declare the variables of your variable type
>> (the type of say, mystruct.attrib), you will need to do multiple
>> declarations outside of your branches. So in my example:
>>
>> cdef func(struct_t mystruct, int i):
>> ?? ?cdef string_t string_var
>> ? ? cdef int int_var
>>
>> ? ? if typeof(mystruct) is typeof(int):
>> ? ? ? ? int_var = mystruct.attrib + i
>> ? ? ? ? ...
>> ? ? else:
>> ? ? ? ?string_var = mystruct.attrib + i
>> ? ? ? ?...
>>
>> But this is probably not a common case, so likely not an issue.
>
> Are you planning to special-case the "real_t complex" syntax? Shooting
> from the sidelines, one more generic solution might be, e.g.,

I'm sorry, I'm not sure what syntax you are referring to. Are you
talking about actual complex numbers?

> ? ? ? ?ctypedef cython.fused_type(A, B) struct_t
> ? ? ? ?ctypedef cython.fused_type(float, double, paired=struct_t) real_t
> ? ? ? ?ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>
> and just restrict the specialization to cases that make sense.

The paired means you're declaring types of attributes?

> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From pav at iki.fi  Fri Apr 29 12:28:01 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 29 Apr 2011 10:28:01 +0000 (UTC)
Subject: [Cython] Fused Types
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
Message-ID: <ipe3rh$efa$1@dough.gmane.org>

Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote:
> On 29 April 2011 11:03, Pauli Virtanen <pav at iki.fi> wrote:
[clip]
>> Are you planning to special-case the "real_t complex" syntax? Shooting
>> from the sidelines, one more generic solution might be, e.g.,
> 
> I'm sorry, I'm not sure what syntax you are referring to. Are you
> talking about actual complex numbers?

This: 

On 28 April 2011 23:30, Robert Bradshaw <robertwb at math.washington.edu> 
wrote:
> OK, I take back what I said, I was looking at the RHS, not the LHS. If
> one needs to specialize in this manner, explicitly creating two
> branches should typically be enough. The same for casting. The one
> exception (perhaps) is "my_fused_type complex." Otherwise it's
> starting to feel too much like C++ template magic and complexity for
> little additional benefit.

That is, declaring a complex type matching a real one.

>> ? ? ? ?ctypedef cython.fused_type(A, B) struct_t 
>>        ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>> ? ? ? ?ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>>
>> and just restrict the specialization to cases that make sense.
> 
> The paired means you're declaring types of attributes?

No, just that real_t is specialized to float whenever struct_t is specialized
to A and to double when B. Or a more realistic example,

	ctypedef cython.fused_type(float, double) real_t
	ctypedef cython.fused_type(float complex, double complex) complex_t

	cdef real_plus_one(complex_t a):
	    real_t b = a.real
            return b + 1

which I suppose would not be a very unusual thing in numerical codes.
This would also allow writing the case you had earlier as

	cdef cython.fused_type(string_t, int, paired=struct_t) attr_t

	cdef func(struct_t mystruct, int i):
	 ? ?cdef attr_t var

	    if typeof(mystruct) is typeof(int):
	        var = mystruct.attrib + i
	        ...
	    else:
	        var = mystruct.attrib + i
	        ...

Things would need to be done explicitly instead of implicitly, though, 
but it would remove the need for any special handling of
the "complex" keyword.

-- 
Pauli Virtanen


From markflorisson88 at gmail.com  Fri Apr 29 12:53:06 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 12:53:06 +0200
Subject: [Cython] Fused Types
In-Reply-To: <ipe3rh$efa$1@dough.gmane.org>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
Message-ID: <BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>

On 29 April 2011 12:28, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote:
>> On 29 April 2011 11:03, Pauli Virtanen <pav at iki.fi> wrote:
> [clip]
>>> Are you planning to special-case the "real_t complex" syntax? Shooting
>>> from the sidelines, one more generic solution might be, e.g.,
>>
>> I'm sorry, I'm not sure what syntax you are referring to. Are you
>> talking about actual complex numbers?
>
> This:
>
> On 28 April 2011 23:30, Robert Bradshaw <robertwb at math.washington.edu>
> wrote:
>> OK, I take back what I said, I was looking at the RHS, not the LHS. If
>> one needs to specialize in this manner, explicitly creating two
>> branches should typically be enough. The same for casting. The one
>> exception (perhaps) is "my_fused_type complex." Otherwise it's
>> starting to feel too much like C++ template magic and complexity for
>> little additional benefit.
>
> That is, declaring a complex type matching a real one.

Ah, I see what you mean now.

>>> ? ? ? ?ctypedef cython.fused_type(A, B) struct_t
>>> ? ? ? ?ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>>> ? ? ? ?ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>>>
>>> and just restrict the specialization to cases that make sense.
>>
>> The paired means you're declaring types of attributes?
>
> No, just that real_t is specialized to float whenever struct_t is specialized
> to A and to double when B. Or a more realistic example,
>
> ? ? ? ?ctypedef cython.fused_type(float, double) real_t
> ? ? ? ?ctypedef cython.fused_type(float complex, double complex) complex_t
>
> ? ? ? ?cdef real_plus_one(complex_t a):
> ? ? ? ? ? ?real_t b = a.real
> ? ? ? ? ? ?return b + 1
>
> which I suppose would not be a very unusual thing in numerical codes.
> This would also allow writing the case you had earlier as
>
> ? ? ? ?cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>
> ? ? ? ?cdef func(struct_t mystruct, int i):
> ? ? ? ? ? ?cdef attr_t var
>
> ? ? ? ? ? ?if typeof(mystruct) is typeof(int):
> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
> ? ? ? ? ? ? ? ?...
> ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
> ? ? ? ? ? ? ? ?...
>
> Things would need to be done explicitly instead of implicitly, though,
> but it would remove the need for any special handling of
> the "complex" keyword.

I see, so it's like a mapping. So, I didn't realize that you can't do this:

def func(arbitrary_type complex x):
    ...

But if we just allow that for fused types, then couldn't we simply do

ctypedef cython.fused_type(float, double) real_t

cdef real_plus_one(real_t complex a):
    real_t b = a.real
    return b + 1

? Then you don't need to pair anything. Perhaps we could introduce
real_t as a type, just like numeric and floating. So I guess
special-casing complex sounds fine with me. Perhaps real_t should be
builtin (not as an attribute of the Cython module), so the parser can
just recognize it immediately?

> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From d.s.seljebotn at astro.uio.no  Fri Apr 29 13:15:23 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 29 Apr 2011 13:15:23 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>	<ipduts$h1n$1@dough.gmane.org>	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
Message-ID: <4DBA9DCB.5090509@astro.uio.no>

On 04/29/2011 12:53 PM, mark florisson wrote:
> On 29 April 2011 12:28, Pauli Virtanen<pav at iki.fi>  wrote:
>> Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote:
>>> On 29 April 2011 11:03, Pauli Virtanen<pav at iki.fi>  wrote:
>> [clip]
>>>> Are you planning to special-case the "real_t complex" syntax? Shooting
>>>> from the sidelines, one more generic solution might be, e.g.,
>>>
>>> I'm sorry, I'm not sure what syntax you are referring to. Are you
>>> talking about actual complex numbers?
>>
>> This:
>>
>> On 28 April 2011 23:30, Robert Bradshaw<robertwb at math.washington.edu>
>> wrote:
>>> OK, I take back what I said, I was looking at the RHS, not the LHS. If
>>> one needs to specialize in this manner, explicitly creating two
>>> branches should typically be enough. The same for casting. The one
>>> exception (perhaps) is "my_fused_type complex." Otherwise it's
>>> starting to feel too much like C++ template magic and complexity for
>>> little additional benefit.
>>
>> That is, declaring a complex type matching a real one.
>
> Ah, I see what you mean now.
>
>>>>         ctypedef cython.fused_type(A, B) struct_t
>>>>         ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>>>>         ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>>>>
>>>> and just restrict the specialization to cases that make sense.
>>>
>>> The paired means you're declaring types of attributes?
>>
>> No, just that real_t is specialized to float whenever struct_t is specialized
>> to A and to double when B. Or a more realistic example,
>>
>>         ctypedef cython.fused_type(float, double) real_t
>>         ctypedef cython.fused_type(float complex, double complex) complex_t
>>
>>         cdef real_plus_one(complex_t a):
>>             real_t b = a.real
>>             return b + 1
>>
>> which I suppose would not be a very unusual thing in numerical codes.

Did you mean

ctypedef cython.fused_type(float complex, double complex,
                            paired=real_t) complex_t

?

>> This would also allow writing the case you had earlier as
>>
>>         cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>
>>         cdef func(struct_t mystruct, int i):
>>             cdef attr_t var
>>
>>             if typeof(mystruct) is typeof(int):
>>                 var = mystruct.attrib + i
>>                 ...
>>             else:
>>                 var = mystruct.attrib + i
>>                 ...
>>
>> Things would need to be done explicitly instead of implicitly, though,
>> but it would remove the need for any special handling of
>> the "complex" keyword.
>
> I see, so it's like a mapping. So, I didn't realize that you can't do this:
>
> def func(arbitrary_type complex x):
>      ...
>

We could support this, but I don't think it is powerful enough. I see 
some code that require pairings like this:

ctypedef cython.fused_type(float, double, float complex, \
  double complex) complex_or_float_t

ctypedef cython.fused_type(float, double, float, \
  double) only_float_t


cdef func(complex_or_float_t x):
     cdef only_float_t y
     ...

So IIUC, one could here add "paired=complex_or_float_t" to say that 
only_float_t links positionally to corresponding types in 
complex_or_float_t.

Perhaps "pair_up_with="? "given_by="?

Anyway, I'm wondering if the special case of complex could be handled by 
having magical built-in fused types for the floating point for these 
purposes, and that these would suffice *shrug*.

Dag Sverre

From pav at iki.fi  Fri Apr 29 13:37:54 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 29 Apr 2011 11:37:54 +0000 (UTC)
Subject: [Cython] Fused Types
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
Message-ID: <ipe7ui$efa$2@dough.gmane.org>

Fri, 29 Apr 2011 12:53:06 +0200, mark florisson wrote:
[clip]
> But if we just allow that for fused types, then couldn't we simply do
> 
> ctypedef cython.fused_type(float, double) real_t
> 
> cdef real_plus_one(real_t complex a):
>     real_t b = a.real
>     return b + 1
> 
> ? Then you don't need to pair anything. Perhaps we could introduce
> real_t as a type, just like numeric and floating. So I guess
> special-casing complex sounds fine with me.

Special-casing for complex sounds ugly to me; "complex float" 
is a type name in the same way as "long int" is, and writing 
"real_t complex" feels a bit like writing "int_t long".

But of course, if this feature is really only needed for declaring
complex variants of real types, then how it is done doesn't matter
so much. But, IMHO, such a special casing would be a somewhat weird
language feature.

-- 
Pauli Virtanen


From markflorisson88 at gmail.com  Fri Apr 29 16:59:22 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 16:59:22 +0200
Subject: [Cython] Fused Types
In-Reply-To: <ipe7ui$efa$2@dough.gmane.org>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<ipe7ui$efa$2@dough.gmane.org>
Message-ID: <BANLkTi=7PLN2aJKxT-x0SD4RhfKm1p=vmg@mail.gmail.com>

On 29 April 2011 13:37, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 29 Apr 2011 12:53:06 +0200, mark florisson wrote:
> [clip]
>> But if we just allow that for fused types, then couldn't we simply do
>>
>> ctypedef cython.fused_type(float, double) real_t
>>
>> cdef real_plus_one(real_t complex a):
>> ? ? real_t b = a.real
>> ? ? return b + 1
>>
>> ? Then you don't need to pair anything. Perhaps we could introduce
>> real_t as a type, just like numeric and floating. So I guess
>> special-casing complex sounds fine with me.
>
> Special-casing for complex sounds ugly to me; "complex float"
> is a type name in the same way as "long int" is, and writing
> "real_t complex" feels a bit like writing "int_t long".
>
> But of course, if this feature is really only needed for declaring
> complex variants of real types, then how it is done doesn't matter
> so much. But, IMHO, such a special casing would be a somewhat weird
> language feature.

Hmm, indeed, it's pretty weird. I'm fine with the pairing also,
although I'm still not sure how common this case is, and if we really
want to support it. Wouldn't good old C promotion work for this? e.g.
if the type is either float or double, just declare your variable
double?

> --
> Pauli Virtanen
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Fri Apr 29 17:04:28 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 29 Apr 2011 17:04:28 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<4DBA1DF4.6010001@behnel.de>
	<BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
Message-ID: <BANLkTi=e2+it5jRYxVN7R0GMFTn-B5N0sg@mail.gmail.com>

On 29 April 2011 06:32, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 28.04.2011 23:29:
>>>
>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>
>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>
>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>
>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>> name, and not the type itself.
>>>>>
>>>>> Yes. This is just because there's not really anything better to return
>>>>> at this point. We should "fix" this at some point in the future.
>>>>>
>>>>>> So I think it would be inconsistent to
>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>> such.
>>>>>
>>>>> I'm open to other suggestions, but would like an expression that
>>>>> resolves at compile time to true/false (and we need to do branch
>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>> different semantics, i.e.
>>>>>
>>>>> ? ?cdef object o = []
>>>>> ? ?typeof(o), type(o)
>>>>>
>>>>> so lets not touch that one.
>>>>
>>>> Right, so for fused types I don't mind string comparison with
>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>> declaration remains open. I'd be fine with something like
>>>> cython.gettype().
>>>
>>> It seems that this isn't optimized yet, but it looks to me like it
>>> wouldn't be very hard to do so. At least == and != could be resolved
>>> at compile time if the other operand is a string literal.
>>
>> Well, the obvious place where this would happen for free would be constant
>> folding. But that runs *way* before type analysis, so all it sees is the
>> typeof() call, not the string.
>
> Actually, to support code like
>
> ctypedef cython.fused_type(object, double) my_fused_type
>
> cdef foo(my_fused_type x):
> ? ?if my_fused_type is object:
> ? ? ? ?print x.attr
> ? ?else:
> ? ? ? ?cdef my_fused_type *ptr = &x
>
> we need to resolve this branch and prune "dead code" before type
> analysis, so I'm thinking it may need to be a special phase, perhaps
> part of the fused-type-specialization phase. Here's another idea:
>
> cdef foo(numeric x):
> ? ?if numeric in floating:
> ? ? ? ?...
> ? ?elif numeric is long:
> ? ? ? ?...
> ? ?else:
> ? ? ? ?...
> ? ?print numeric is double ? # after the specialization pass, this
> would be a literal True/False node.
>
> Again, this would all be resolved before type analysis. In terms of
> syntactically supporting pointers, we could either support "fused_type
> is typeof(void*)" or require typedefs for types not representable by
> an ExprNode. (This would make declaring fused types syntactically
> simpler as well...) I prefer the latter.
>
> - Robert
>

I made both things work, you can use both typeof() on expressions and
you can compare types using 'is' and 'in'. However, with typeof() it
doesn't remove branches, so if you're doing incompatible stuff you
need to do the type matching. It could be supported, but currently it
isn't because TransformBuiltinMethods runs after
AnalyseDeclarationsTransform (for which the reason is not immediately
apparent).

With the type matching it matches on exactly 'if src_type is
dst_type:' so you can't use 'and' and such... perhaps I should turn
these expression into a node with the constant value first and then
see if the result of the entire expression is known at compile time?

From pav at iki.fi  Fri Apr 29 17:57:53 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 29 Apr 2011 15:57:53 +0000 (UTC)
Subject: [Cython] Fused Types
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<ipe7ui$efa$2@dough.gmane.org>
	<BANLkTi=7PLN2aJKxT-x0SD4RhfKm1p=vmg@mail.gmail.com>
Message-ID: <ipen61$2pm$1@dough.gmane.org>

Fri, 29 Apr 2011 16:59:22 +0200, mark florisson wrote:
[clip]
> Hmm, indeed, it's pretty weird. I'm fine with the pairing also, although
> I'm still not sure how common this case is, and if we really want to
> support it. Wouldn't good old C promotion work for this? e.g. if the
> type is either float or double, just declare your variable double?

In practice mostly yes I guess; IIRC single-precision arithmetic is not
any faster than double precision nowadays anyway. However, (playing the 
devil's advocate :)

(i) Theoretically, someone might also want to write code that works
both for "double" and for "long double complex". You probably wouldn't
want to use "long double" for the double-precision specialization.

(ii) One reason to use floats could be the 2x size advantage. In principle
someone might want to deal with huge float and float complex arrays. (Sounds
like a somewhat rare use case though.)

(iii) If you want to wrap a library that already provides complex float
functions in both precisions, having a specialized real type could come
handy sometimes.


But in practice, I guess the "fused_type(double, double complex)" would be
the most common use case. Maybe it's best to wait until enough concrete
examples accumulate before implementing anything more --- I guess
e.g. the pairing feature wouldn't be too difficult to add if it turns out
something like that is really needed.

	Pauli


From greg.ewing at canterbury.ac.nz  Sat Apr 30 01:59:55 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 30 Apr 2011 11:59:55 +1200
Subject: [Cython] What *is* a fused type?
In-Reply-To: <ipe7ui$efa$2@dough.gmane.org>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<ipe7ui$efa$2@dough.gmane.org>
Message-ID: <4DBB50FB.4070205@canterbury.ac.nz>

I seem to have missed the beginning of the discussion about this
fused type business. Is there a document somewhere describing
what a "fused type" is and what it's meant to be used for?

-- 
Greg

From robertwb at math.washington.edu  Sat Apr 30 06:09:13 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Apr 2011 21:09:13 -0700
Subject: [Cython] What *is* a fused type?
In-Reply-To: <4DBB50FB.4070205@canterbury.ac.nz>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<ipe7ui$efa$2@dough.gmane.org> <4DBB50FB.4070205@canterbury.ac.nz>
Message-ID: <BANLkTikO3jZo7MwuzYxfYwrQ83SRkueUgw@mail.gmail.com>

On Fri, Apr 29, 2011 at 4:59 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I seem to have missed the beginning of the discussion about this
> fused type business. Is there a document somewhere describing
> what a "fused type" is and what it's meant to be used for?

http://wiki.cython.org/enhancements/fusedtypes

In short, it's a very basic type parameterization.

- Robert

From vitja.makarov at gmail.com  Sat Apr 30 08:08:57 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 30 Apr 2011 10:08:57 +0400
Subject: [Cython] speed.pypy.org
In-Reply-To: <4DB86F5D.2080603@behnel.de>
References: <4DA2FD5E.5090704@behnel.de> <4DA8A8A3.6080600@behnel.de>
	<4DB6DBCF.2020301@behnel.de>
	<BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>
	<4DB7C9B4.9040004@behnel.de>
	<BANLkTim6L4WQDPzLNrNodnrWXKm9FMbq=g@mail.gmail.com>
	<4DB86F5D.2080603@behnel.de>
Message-ID: <BANLkTiniJVJ1CQjpf++V1OYOekJyrUGEag@mail.gmail.com>

2011/4/27 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 27.04.2011 19:08:
>>
>> On Wed, Apr 27, 2011 at 12:45 AM, Stefan Behnel wrote:
>>>
>>> Actually, if we want a proper history, I'd suggest a separate codespeed
>>> installation somewhere.
>>
>> Makes sense. How many CPU-hours does it take?
>
> Including the Cython build, it's more like 25 minutes currently, given that
> we only support a smaller part of the benchmark suite. It will obviously
> take longer when we start supporting the larger benchmarks, such as Django
> templates or Twisted.
>
>
>> If it's not to
>> intensive, we could probably run it, say, daily as a normal-priority
>> job.
>
> We could certainly do that for now, and check again when we see that it
> starts running substantially longer.
>

Nice thing! Can I run this tests at home?
-- 
vitja.

From robertwb at math.washington.edu  Sat Apr 30 08:16:13 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Apr 2011 23:16:13 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=e2+it5jRYxVN7R0GMFTn-B5N0sg@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<4DBA1DF4.6010001@behnel.de>
	<BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
	<BANLkTi=e2+it5jRYxVN7R0GMFTn-B5N0sg@mail.gmail.com>
Message-ID: <BANLkTi=zdL9DoO8GkDCnvpusJDOLYZW2Ow@mail.gmail.com>

On Fri, Apr 29, 2011 at 8:04 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 29 April 2011 06:32, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 28.04.2011 23:29:
>>>>
>>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>>
>>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>>
>>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>>
>>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>>> name, and not the type itself.
>>>>>>
>>>>>> Yes. This is just because there's not really anything better to return
>>>>>> at this point. We should "fix" this at some point in the future.
>>>>>>
>>>>>>> So I think it would be inconsistent to
>>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>>> such.
>>>>>>
>>>>>> I'm open to other suggestions, but would like an expression that
>>>>>> resolves at compile time to true/false (and we need to do branch
>>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>>> different semantics, i.e.
>>>>>>
>>>>>> ? ?cdef object o = []
>>>>>> ? ?typeof(o), type(o)
>>>>>>
>>>>>> so lets not touch that one.
>>>>>
>>>>> Right, so for fused types I don't mind string comparison with
>>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>>> declaration remains open. I'd be fine with something like
>>>>> cython.gettype().
>>>>
>>>> It seems that this isn't optimized yet, but it looks to me like it
>>>> wouldn't be very hard to do so. At least == and != could be resolved
>>>> at compile time if the other operand is a string literal.
>>>
>>> Well, the obvious place where this would happen for free would be constant
>>> folding. But that runs *way* before type analysis, so all it sees is the
>>> typeof() call, not the string.
>>
>> Actually, to support code like
>>
>> ctypedef cython.fused_type(object, double) my_fused_type
>>
>> cdef foo(my_fused_type x):
>> ? ?if my_fused_type is object:
>> ? ? ? ?print x.attr
>> ? ?else:
>> ? ? ? ?cdef my_fused_type *ptr = &x
>>
>> we need to resolve this branch and prune "dead code" before type
>> analysis, so I'm thinking it may need to be a special phase, perhaps
>> part of the fused-type-specialization phase. Here's another idea:
>>
>> cdef foo(numeric x):
>> ? ?if numeric in floating:
>> ? ? ? ?...
>> ? ?elif numeric is long:
>> ? ? ? ?...
>> ? ?else:
>> ? ? ? ?...
>> ? ?print numeric is double ? # after the specialization pass, this
>> would be a literal True/False node.
>>
>> Again, this would all be resolved before type analysis. In terms of
>> syntactically supporting pointers, we could either support "fused_type
>> is typeof(void*)" or require typedefs for types not representable by
>> an ExprNode. (This would make declaring fused types syntactically
>> simpler as well...) I prefer the latter.
>>
>> - Robert
>>
>
> I made both things work, you can use both typeof() on expressions and
> you can compare types using 'is' and 'in'.

Cool.

> However, with typeof() it
> doesn't remove branches, so if you're doing incompatible stuff you
> need to do the type matching. It could be supported, but currently it
> isn't because TransformBuiltinMethods runs after
> AnalyseDeclarationsTransform (for which the reason is not immediately
> apparent).

For one thing, it's the declaration analysis that allows you to figure
out what name nodes are built-ins.

> With the type matching it matches on exactly 'if src_type is
> dst_type:' so you can't use 'and' and such... perhaps I should turn
> these expression into a node with the constant value first and then
> see if the result of the entire expression is known at compile time?

Yes, that's exactly what I was thinking you should do.

- Robert

From robertwb at math.washington.edu  Sat Apr 30 08:39:58 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Apr 2011 23:39:58 -0700
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
Message-ID: <BANLkTi=LkkMiUJP2fLhyGVQ7rCx4F8ORrg@mail.gmail.com>

On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 29 April 2011 12:28, Pauli Virtanen <pav at iki.fi> wrote:
>> Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote:
>>> On 29 April 2011 11:03, Pauli Virtanen <pav at iki.fi> wrote:
>> [clip]
>>>> Are you planning to special-case the "real_t complex" syntax? Shooting
>>>> from the sidelines, one more generic solution might be, e.g.,
>>>
>>> I'm sorry, I'm not sure what syntax you are referring to. Are you
>>> talking about actual complex numbers?
>>
>> This:
>>
>> On 28 April 2011 23:30, Robert Bradshaw <robertwb at math.washington.edu>
>> wrote:
>>> OK, I take back what I said, I was looking at the RHS, not the LHS. If
>>> one needs to specialize in this manner, explicitly creating two
>>> branches should typically be enough. The same for casting. The one
>>> exception (perhaps) is "my_fused_type complex." Otherwise it's
>>> starting to feel too much like C++ template magic and complexity for
>>> little additional benefit.
>>
>> That is, declaring a complex type matching a real one.
>
> Ah, I see what you mean now.
>
>>>> ? ? ? ?ctypedef cython.fused_type(A, B) struct_t
>>>> ? ? ? ?ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>>>> ? ? ? ?ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>>>>
>>>> and just restrict the specialization to cases that make sense.
>>>
>>> The paired means you're declaring types of attributes?
>>
>> No, just that real_t is specialized to float whenever struct_t is specialized
>> to A and to double when B. Or a more realistic example,
>>
>> ? ? ? ?ctypedef cython.fused_type(float, double) real_t
>> ? ? ? ?ctypedef cython.fused_type(float complex, double complex) complex_t
>>
>> ? ? ? ?cdef real_plus_one(complex_t a):
>> ? ? ? ? ? ?real_t b = a.real
>> ? ? ? ? ? ?return b + 1
>>
>> which I suppose would not be a very unusual thing in numerical codes.
>> This would also allow writing the case you had earlier as
>>
>> ? ? ? ?cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>
>> ? ? ? ?cdef func(struct_t mystruct, int i):
>> ? ? ? ? ? ?cdef attr_t var
>>
>> ? ? ? ? ? ?if typeof(mystruct) is typeof(int):
>> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
>> ? ? ? ? ? ? ? ?...
>> ? ? ? ? ? ?else:
>> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
>> ? ? ? ? ? ? ? ?...
>>
>> Things would need to be done explicitly instead of implicitly, though,
>> but it would remove the need for any special handling of
>> the "complex" keyword.

If we're going to introduce pairing, another option would be

    ctypedef fused_type((double complex, double), (float complex,
float)) (complex_t, real_t)

though I'm not sure I like that either. We're not trying to create the
all-powerful templating system here, and anything that can be done
with pairing can be done (though less elegantly) via branching on the
types, or, as Pauli mentions, using a wider type is often (but not
always) a viable option.

We talked of supporting fused types in classes, one could do the same
for structs, thus

    cdef fused_type(int, double) int_or_double
    cdef my_struct:
        int_or_double attr

    cdef func1(int_or_double x):
        cdef my_struct
        my_struct.attr = x

and

    cdef func2(my_struct s)
        cdef int_or_double x = s.attr

could work. Maybe we'd require an explicit "my_struct[int_or_double]".
This would solve a large number of the remaining cases, and complex is
like a struct + native arithmetic/promotion.

> I see, so it's like a mapping. So, I didn't realize that you can't do this:
>
> def func(arbitrary_type complex x):
> ? ?...
>
> But if we just allow that for fused types, then couldn't we simply do
>
> ctypedef cython.fused_type(float, double) real_t
>
> cdef real_plus_one(real_t complex a):
> ? ?real_t b = a.real
> ? ?return b + 1
>
> ? Then you don't need to pair anything.

That's what I was thinking. It's a bit special, but on the other hand
a very natural pairing that one would want to be able to do with a
very natural syntax (no one reading that code would wonder what it
means).

> Perhaps we could introduce
> real_t as a type, just like numeric and floating. So I guess
> special-casing complex sounds fine with me. Perhaps real_t should be
> builtin (not as an attribute of the Cython module), so the parser can
> just recognize it immediately?

I'd rather not add new keywords to the language, and I don't see what
advantage letting real_t be known to the parser woud be, we can ship a
pxd file with the common fused types.

- Robert

From stefan_ml at behnel.de  Sat Apr 30 08:49:52 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Apr 2011 08:49:52 +0200
Subject: [Cython] Fwd: Proposal for a common benchmark suite
Message-ID: <4DBBB110.6000501@behnel.de>

[Seems like this was meant to go to this list but didn't make it there. 
Forwarding from python-dev]

-------- Original-Message --------
Subject: Proposal for a common benchmark suite
Date: Thu, 28 Apr 2011 20:55:19 +0200
From: DasIch

Hello,
As announced in my GSoC proposal I'd like to announce which benchmarks
I'll use for the benchmark suite I will work on this summer.

As of now there are two benchmark suites (that I know of) which
receive some sort of attention, those are the ones developed as part
of the PyPy project[1] which is used for http://speed.pypy.org and the
one initially developed for Unladen Swallow which has been continued
by CPython[2]. The PyPy benchmarks contain a lot of interesting
benchmarks some explicitly developed for that suite, the CPython
benchmarks have an extensive set of microbenchmarks in the pybench
package as well as the previously mentioned modifications made to the
Unladen Swallow benchmarks.

I'd like to "simply" merge both suites so that no changes are lost.
However I'd like to leave out the waf benchmark which is part of the
PyPy suite, the removal was proposed on pypy-dev for obvious
deficits[3]. It will be easier to add a better benchmark later than
replacing it at a later point.

Unless there is a major issue with this plan I'd like to go forward with this.

.. [1]: https://bitbucket.org/pypy/benchmarks
.. [2]: http://hg.python.org/benchmarks
.. [3]: http://mailrepository.com/pypy-dev.codespeak.net/msg/3627509/


From stefan_ml at behnel.de  Sat Apr 30 09:09:24 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Apr 2011 09:09:24 +0200
Subject: [Cython] speed.pypy.org
In-Reply-To: <BANLkTiniJVJ1CQjpf++V1OYOekJyrUGEag@mail.gmail.com>
References: <4DA2FD5E.5090704@behnel.de>
	<4DA8A8A3.6080600@behnel.de>	<4DB6DBCF.2020301@behnel.de>	<BANLkTimUqW0jK7SOZ+Jx69_P4iW2G7APAw@mail.gmail.com>	<4DB7C9B4.9040004@behnel.de>	<BANLkTim6L4WQDPzLNrNodnrWXKm9FMbq=g@mail.gmail.com>	<4DB86F5D.2080603@behnel.de>
	<BANLkTiniJVJ1CQjpf++V1OYOekJyrUGEag@mail.gmail.com>
Message-ID: <4DBBB5A4.30902@behnel.de>

Vitja Makarov, 30.04.2011 08:08:
> Can I run this tests at home?

Don't expect it to come shrink wrapped. It took me a while to get the setup 
running on Hudson/Jenkins, and the scripts that do it aren't completely 
trivial. You can take a look at the config of the job that runs them:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/

I'm currently using the PyPy test suite at

https://bitbucket.org/pypy/benchmarks/

instead of the future CPython one at

http://hg.python.org/benchmarks/

simply because the PyPy suite has more benchmarks (that work for us ;).

The major problem with the test suite is that the unladen swallow people 
who wrote the test runner somehow got the paranoid idea that they had to 
control which environment variables were set during the run, so they chose 
to drop the entire environment and use their own one. They clearly never 
tried to run distutils in a benchmark, which requires some intimate 
knowledge about the environment it runs in.

So you basically need a wrapper around cythonrun that properly sets up the 
environment for you. Take a look at what I did in the build job, it 
generates a script that sets up the environment, and then passes that into 
the benchmark runner.

You also need a "usercustomize.py" script that installs pyximport (but only 
for the Cython runs, not for plain CPython!), so that it's not only the 
benchmark script that gets compiled but also its dependencies. I'm 
currently blacklisting the stdlib modules ['threading', 'socket', 
'gettext', 'locale'] in pyximport.PyxImporter.blocked_modules when I run it 
locally, simply because they don't work but have a substantial impact on 
the running code.

The basic command I'm using to run the suite is:

python runner.py -a ",-Xauto_cpdef=True" \
     -p ./cythonrun.sh --baseline=/path/to/python \
     --fast -o results.json

The CPython test runner is a tad better here, because it allows you to 
provide at least a whitelist of environment variables that are being passed 
through. I'm currently trying to set up a second benchmark job to run the 
(smaller) CPython benchmark suite in Py3 mode. The command to run that 
suite is more like

python perf.py -a ',-Xauto_cpdef=true' \
     --inherit_env=PYTHONPATH,PYTHONHOME,PYTHON,CFLAGS,OPT,PATH \
     /path/to/python ./cythonrun.sh \
    --fast

and it doesn't seem to have json output. (Seriously, I can't believe they 
even broke "PATH" in their runner script!).

There is also a GSoC project that aims to fix up the benchmark suite for 
the different Python implementations in order to set up a 
"speed.python.org" site that compares them. We should stay involved in that.

Stefan

From d.s.seljebotn at astro.uio.no  Sat Apr 30 09:51:20 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 30 Apr 2011 09:51:20 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTi=LkkMiUJP2fLhyGVQ7rCx4F8ORrg@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>	<ipduts$h1n$1@dough.gmane.org>	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>	<ipe3rh$efa$1@dough.gmane.org>	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<BANLkTi=LkkMiUJP2fLhyGVQ7rCx4F8ORrg@mail.gmail.com>
Message-ID: <4DBBBF78.4030104@astro.uio.no>

On 04/30/2011 08:39 AM, Robert Bradshaw wrote:
> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> On 29 April 2011 12:28, Pauli Virtanen<pav at iki.fi>  wrote:
>>> No, just that real_t is specialized to float whenever struct_t is specialized
>>> to A and to double when B. Or a more realistic example,
>>>
>>>         ctypedef cython.fused_type(float, double) real_t
>>>         ctypedef cython.fused_type(float complex, double complex) complex_t
>>>
>>>         cdef real_plus_one(complex_t a):
>>>             real_t b = a.real
>>>             return b + 1
>>>
>>> which I suppose would not be a very unusual thing in numerical codes.
>>> This would also allow writing the case you had earlier as
>>>
>>>         cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>>
>>>         cdef func(struct_t mystruct, int i):
>>>             cdef attr_t var
>>>
>>>             if typeof(mystruct) is typeof(int):
>>>                 var = mystruct.attrib + i
>>>                 ...
>>>             else:
>>>                 var = mystruct.attrib + i
>>>                 ...
>>>
>>> Things would need to be done explicitly instead of implicitly, though,
>>> but it would remove the need for any special handling of
>>> the "complex" keyword.
>
> If we're going to introduce pairing, another option would be
>
>      ctypedef fused_type((double complex, double), (float complex,
> float)) (complex_t, real_t)
>
> though I'm not sure I like that either. We're not trying to create the
> all-powerful templating system here, and anything that can be done
> with pairing can be done (though less elegantly) via branching on the
> types, or, as Pauli mentions, using a wider type is often (but not
> always) a viable option.

Keeping the right balance is difficult. But, at least there's some cases 
of needing this in various codebases when interfacing with LAPACK.

Most uses of templating with Cython code I've seen so far does a similar 
kind of "zip" as what you have above (as we discussed on the workshop). 
So at least the usage pattern you write above is very common.

float32 is not about to disappear, it really is twice as fast when 
you're memory IO bound.

Using a wider type is actually quite often not possible; any time the 
type is involved as the base type of an array it is not possible, and 
that's a pretty common case. (With LAPACK you take the address of the 
variable and pass it to Fortran, so using a wider type is not possible 
there either, although I'll agree that's a more remote case.)

My proposal: Don't support either "real_t complex" or paired fused types 
for the time being. Then see.

But my vote is for paired fused types instead of "real_t complex".

Dag Sverre

From stefan_ml at behnel.de  Sat Apr 30 16:24:38 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Apr 2011 16:24:38 +0200
Subject: [Cython] Fused Types
In-Reply-To: <BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>	<4DB6D9FA.9030405@behnel.de>	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>	<4DBA1DF4.6010001@behnel.de>
	<BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
Message-ID: <4DBC1BA6.3030806@behnel.de>

Robert Bradshaw, 29.04.2011 06:32:
> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> mark florisson, 28.04.2011 23:29:
>>>
>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>
>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>
>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>
>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>> name, and not the type itself.
>>>>>
>>>>> Yes. This is just because there's not really anything better to return
>>>>> at this point. We should "fix" this at some point in the future.
>>>>>
>>>>>> So I think it would be inconsistent to
>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>> such.
>>>>>
>>>>> I'm open to other suggestions, but would like an expression that
>>>>> resolves at compile time to true/false (and we need to do branch
>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>> different semantics, i.e.
>>>>>
>>>>>     cdef object o = []
>>>>>     typeof(o), type(o)
>>>>>
>>>>> so lets not touch that one.
>>>>
>>>> Right, so for fused types I don't mind string comparison with
>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>> declaration remains open. I'd be fine with something like
>>>> cython.gettype().
>>>
>>> It seems that this isn't optimized yet, but it looks to me like it
>>> wouldn't be very hard to do so. At least == and != could be resolved
>>> at compile time if the other operand is a string literal.
>>
>> Well, the obvious place where this would happen for free would be constant
>> folding. But that runs *way* before type analysis, so all it sees is the
>> typeof() call, not the string.
>
> Actually, to support code like
>
> ctypedef cython.fused_type(object, double) my_fused_type
>
> cdef foo(my_fused_type x):
>      if my_fused_type is object:
>          print x.attr
>      else:
>          cdef my_fused_type *ptr =&x
>
> we need to resolve this branch and prune "dead code" before type
> analysis, so I'm thinking it may need to be a special phase, perhaps
> part of the fused-type-specialization phase. Here's another idea:
>
> cdef foo(numeric x):
>      if numeric in floating:
>          ...
>      elif numeric is long:
>          ...
>      else:
>          ...
>      print numeric is double   # after the specialization pass, this
> would be a literal True/False node.
>
> Again, this would all be resolved before type analysis.

Hmm, I wonder, if we ever want to start with whole program analysis, and we 
find code like this:

     def func(arg):
         ...

     a = [1,2,3]
     func(a)

we could infer that "arg" is actually a "fused_type(object, list)". So, I 
think, what we eventually want is an intermingling of type analysis and 
specialisation. Meaning, we'd find a new type during type analysis or 
inference, split off a new version of the function and then analyse it. But 
maybe that's just a pie in the sky for now.

Stefan

From robertwb at math.washington.edu  Sat Apr 30 19:05:09 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 Apr 2011 10:05:09 -0700
Subject: [Cython] Fused Types
In-Reply-To: <4DBBBF78.4030104@astro.uio.no>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTimbzNc_aTbe=oP2JHcXR_7d07CdTw@mail.gmail.com>
	<BANLkTi=RaDiRFzksaEdt6Vj7+1P-4JGt5g@mail.gmail.com>
	<ipduts$h1n$1@dough.gmane.org>
	<BANLkTimUvfnc_OudRZNP1VBc6PwoMh9z_A@mail.gmail.com>
	<ipe3rh$efa$1@dough.gmane.org>
	<BANLkTimNELYBCZsPOuSBhZj+_1GUA+WoZA@mail.gmail.com>
	<BANLkTi=LkkMiUJP2fLhyGVQ7rCx4F8ORrg@mail.gmail.com>
	<4DBBBF78.4030104@astro.uio.no>
Message-ID: <BANLkTi=5AjZLeSH2EeuQzkN5B6p2kKsKyQ@mail.gmail.com>

On Sat, Apr 30, 2011 at 12:51 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 04/30/2011 08:39 AM, Robert Bradshaw wrote:
>>
>> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
>> <markflorisson88 at gmail.com> ?wrote:
>>>
>>> On 29 April 2011 12:28, Pauli Virtanen<pav at iki.fi> ?wrote:
>>>>
>>>> No, just that real_t is specialized to float whenever struct_t is
>>>> specialized
>>>> to A and to double when B. Or a more realistic example,
>>>>
>>>> ? ? ? ?ctypedef cython.fused_type(float, double) real_t
>>>> ? ? ? ?ctypedef cython.fused_type(float complex, double complex)
>>>> complex_t
>>>>
>>>> ? ? ? ?cdef real_plus_one(complex_t a):
>>>> ? ? ? ? ? ?real_t b = a.real
>>>> ? ? ? ? ? ?return b + 1
>>>>
>>>> which I suppose would not be a very unusual thing in numerical codes.
>>>> This would also allow writing the case you had earlier as
>>>>
>>>> ? ? ? ?cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>>>
>>>> ? ? ? ?cdef func(struct_t mystruct, int i):
>>>> ? ? ? ? ? ?cdef attr_t var
>>>>
>>>> ? ? ? ? ? ?if typeof(mystruct) is typeof(int):
>>>> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
>>>> ? ? ? ? ? ? ? ?...
>>>> ? ? ? ? ? ?else:
>>>> ? ? ? ? ? ? ? ?var = mystruct.attrib + i
>>>> ? ? ? ? ? ? ? ?...
>>>>
>>>> Things would need to be done explicitly instead of implicitly, though,
>>>> but it would remove the need for any special handling of
>>>> the "complex" keyword.
>>
>> If we're going to introduce pairing, another option would be
>>
>> ? ? ctypedef fused_type((double complex, double), (float complex,
>> float)) (complex_t, real_t)
>>
>> though I'm not sure I like that either. We're not trying to create the
>> all-powerful templating system here, and anything that can be done
>> with pairing can be done (though less elegantly) via branching on the
>> types, or, as Pauli mentions, using a wider type is often (but not
>> always) a viable option.
>
> Keeping the right balance is difficult. But, at least there's some cases of
> needing this in various codebases when interfacing with LAPACK.
>
> Most uses of templating with Cython code I've seen so far does a similar
> kind of "zip" as what you have above (as we discussed on the workshop). So
> at least the usage pattern you write above is very common.
>
> float32 is not about to disappear, it really is twice as fast when you're
> memory IO bound.
>
> Using a wider type is actually quite often not possible; any time the type
> is involved as the base type of an array it is not possible, and that's a
> pretty common case. (With LAPACK you take the address of the variable and
> pass it to Fortran, so using a wider type is not possible there either,
> although I'll agree that's a more remote case.)
>
> My proposal: Don't support either "real_t complex" or paired fused types for
> the time being. Then see.

+1, then we can address real need.

> But my vote is for paired fused types instead of "real_t complex".
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Sat Apr 30 19:06:38 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 Apr 2011 10:06:38 -0700
Subject: [Cython] Fused Types
In-Reply-To: <4DBC1BA6.3030806@behnel.de>
References: <BANLkTinJFVgg4CPkdHT2PzFzEk08No9M_g@mail.gmail.com>
	<4DB6D9FA.9030405@behnel.de>
	<BANLkTimhWxsyBiprnO1bpaZnNR-Ox_Vc8A@mail.gmail.com>
	<BANLkTinae3VzU=YksWCSuKdeZQizJgsrqQ@mail.gmail.com>
	<BANLkTi=U7+TfcGOn6yf0nVxffcX7WS0FwA@mail.gmail.com>
	<BANLkTimUyidRCii+VMVP2VYswDLpwtfWCA@mail.gmail.com>
	<BANLkTikf5hYzZzsHNgVmEx_fH7gaDQY13Q@mail.gmail.com>
	<BANLkTingEewxkQ-PMUCtqNWZ=NKWMv_h9g@mail.gmail.com>
	<4DBA1DF4.6010001@behnel.de>
	<BANLkTinALWLxx-f6Arn6f+bc28zKC2KiVw@mail.gmail.com>
	<4DBC1BA6.3030806@behnel.de>
Message-ID: <BANLkTikJ4Psrv0MorfoXw6sKkKJTrbJTJg@mail.gmail.com>

On Sat, Apr 30, 2011 at 7:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 29.04.2011 06:32:
>>
>> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel<stefan_ml at behnel.de>
>> ?wrote:
>>>
>>> mark florisson, 28.04.2011 23:29:
>>>>
>>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>>
>>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>>
>>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>>
>>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>>> cython.typeof(). I believe it currently returns a string with the
>>>>>>> type
>>>>>>> name, and not the type itself.
>>>>>>
>>>>>> Yes. This is just because there's not really anything better to return
>>>>>> at this point. We should "fix" this at some point in the future.
>>>>>>
>>>>>>> So I think it would be inconsistent to
>>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>>> such.
>>>>>>
>>>>>> I'm open to other suggestions, but would like an expression that
>>>>>> resolves at compile time to true/false (and we need to do branch
>>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>>> different semantics, i.e.
>>>>>>
>>>>>> ? ?cdef object o = []
>>>>>> ? ?typeof(o), type(o)
>>>>>>
>>>>>> so lets not touch that one.
>>>>>
>>>>> Right, so for fused types I don't mind string comparison with
>>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>>> declaration remains open. I'd be fine with something like
>>>>> cython.gettype().
>>>>
>>>> It seems that this isn't optimized yet, but it looks to me like it
>>>> wouldn't be very hard to do so. At least == and != could be resolved
>>>> at compile time if the other operand is a string literal.
>>>
>>> Well, the obvious place where this would happen for free would be
>>> constant
>>> folding. But that runs *way* before type analysis, so all it sees is the
>>> typeof() call, not the string.
>>
>> Actually, to support code like
>>
>> ctypedef cython.fused_type(object, double) my_fused_type
>>
>> cdef foo(my_fused_type x):
>> ? ? if my_fused_type is object:
>> ? ? ? ? print x.attr
>> ? ? else:
>> ? ? ? ? cdef my_fused_type *ptr =&x
>>
>> we need to resolve this branch and prune "dead code" before type
>> analysis, so I'm thinking it may need to be a special phase, perhaps
>> part of the fused-type-specialization phase. Here's another idea:
>>
>> cdef foo(numeric x):
>> ? ? if numeric in floating:
>> ? ? ? ? ...
>> ? ? elif numeric is long:
>> ? ? ? ? ...
>> ? ? else:
>> ? ? ? ? ...
>> ? ? print numeric is double ? # after the specialization pass, this
>> would be a literal True/False node.
>>
>> Again, this would all be resolved before type analysis.
>
> Hmm, I wonder, if we ever want to start with whole program analysis, and we
> find code like this:
>
> ? ?def func(arg):
> ? ? ? ?...
>
> ? ?a = [1,2,3]
> ? ?func(a)
>
> we could infer that "arg" is actually a "fused_type(object, list)". So, I
> think, what we eventually want is an intermingling of type analysis and
> specialisation. Meaning, we'd find a new type during type analysis or
> inference, split off a new version of the function and then analyse it. But
> maybe that's just a pie in the sky for now.

This fits into the idea of creating two (or possibly more, but
avoiding combinatorial explosion) branches, a fast one and a safe one,
and bailing from one to the other if guards fail (all within the same
function).

From d.s.seljebotn at astro.uio.no  Sat Apr 30 23:35:59 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 30 Apr 2011 23:35:59 +0200
Subject: [Cython] Pull request emails
Message-ID: <4DBC80BF.4090707@astro.uio.no>

Finally think I figured out how to get pull request emails (thanks to 
Gael V). From https://github.com/organizations/cython/teams/24445:

"""
Owners do not receive notifications for the organization's repos by 
default. To receive notifications, create a team and add the owners and 
repos for which notifications are desired.
"""

I created Reviewers and added me, Stefan & Robert for now.

https://github.com/organizations/cython/teams/54516

Dag Sverre

From robertwb at math.washington.edu  Sat Apr 30 23:47:29 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 Apr 2011 14:47:29 -0700
Subject: [Cython] Pull request emails
In-Reply-To: <4DBC80BF.4090707@astro.uio.no>
References: <4DBC80BF.4090707@astro.uio.no>
Message-ID: <BANLkTimKaG0=9h+MEnFJDB2a12ndH0O2_Q@mail.gmail.com>

Excellent, thanks.

On Sat, Apr 30, 2011 at 2:35 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> Finally think I figured out how to get pull request emails (thanks to Gael
> V). From https://github.com/organizations/cython/teams/24445:
>
> """
> Owners do not receive notifications for the organization's repos by default.
> To receive notifications, create a team and add the owners and repos for
> which notifications are desired.
> """
>
> I created Reviewers and added me, Stefan & Robert for now.
>
> https://github.com/organizations/cython/teams/54516
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>