From robertwb at math.washington.edu  Wed Feb  1 19:50:45 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Feb 2012 10:50:45 -0800
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
	<CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
	<CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
Message-ID: <CADiQ+QBKRnad6c1nq_wk7N3roUHuVZqcR9z_yByk3PjtEMieLA@mail.gmail.com>

On Tue, Jan 31, 2012 at 8:30 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 31 January 2012 02:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>>>> mark florisson, 27.01.2012 17:30:
>>>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com> ?wrote:
>>>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>>>
>>>>>>>> How can I patch Cython such that some more information on the location
>>>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>>>> message
>>>>>>>> ? ?Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>>>> object has no attribute '_modulus' in ?ignored
>>>>>>>>
>>>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>>>> responsible for printing the warning?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Simon
>>>>>>>
>>>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>>>> typically two reasons for unraisable exceptions:
>>>>>>>
>>>>>>> ? ? 1) as Robert mentioned, a function that does not allow propagation
>>>>>>> of exceptions, e.g.
>>>>>>>
>>>>>>> ? ? ? ? cdef int func():
>>>>>>> ? ? ? ? ? ? raise Exception
>>>>>>>
>>>>>>> ? ? ? ? Here there is no way to propagate the raised exception, so
>>>>>>> instead one should write something like
>>>>>>>
>>>>>>> ? ? ? ? ? ? cdef int func() except -1: ...
>>>>>>>
>>>>>>> ? ? ? ? Alternatively one may use 'except *' in case there is no error
>>>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>>>> "-1 may or may not indicate an error".
>>>>>>>
>>>>>>> ? ? 2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>>>
>>>>>>> For functions the right thing is to add an except clause, for
>>>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>>>
>>>>>>> ? ? try:
>>>>>>> ? ? ? ? ...
>>>>>>> ? ? except:
>>>>>>> ? ? ? ? traceback.print_exc()
>>>>>>>
>>>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>>>
>>>>>> Actually, I don't see why the default is to write unraisable
>>>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>>>> and have callers do the check (i.e. make it implicitly "except *").
>>>>
>>>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>>>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>>>
>>>> And for floating point, we could make our own NaN -- that's obscure enough
>>>> that it could probably be made "except cython.cython_exception_nan" by
>>>> default, not "except? cython.cython_exception_nan".
>>>
>>> The problem with that is that we can't be sure that Cython will be the only
>>> caller. So exceptions may still not propagate in cases, and users will have
>>> to know about these "obscure" values and that they must deal with them
>>> manually then.
>>>
>>> You could add that we'd just have to disable this when user code takes a
>>> pointer from a function, but then, how many rules are there that users will
>>> have to learn and remember after such a change? And what's that for a
>>> language that changes the calling semantics of a function because way down
>>> in the code someone happens to take a pointer to it?
>>>
>>>
>>>>>> Was this not implemented because Cython only knows whether functions
>>>>>> may propagate exceptions at code generation time by looking at the
>>>>>> presence of an error label?
>>>>>> Maybe it could keep code insertion points around for every call to
>>>>>> such a potential function and if the function uses the error label
>>>>>> have the caller perform the check? Although I do forsee problems for
>>>>>> external such functions... maybe Cython could have it's own
>>>>>> threadstate regardless of the GIL which would indicate whether an
>>>>>> error has occurred? e.g. CyErr_Occurred()?
>>>>>
>>>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>>>> default.
>>>>
>>>> Still,
>>>
>>> I wasn't really advocating this behaviour, just indicating that it's hard
>>> to do "better", because this "better" isn't all that clear. It's also not
>>> "better" for all code, which means that we get from one trade-off to
>>> another, while breaking existing code at the same time. Not exactly
>>> paradise on either side of the tunnel.
>>
>> I still feel like we're stuck in the wrong default. I'd rather require
>> more work to interact with C libraries than require more work to
>> convert innocent-looking Python to Cython.
>>
>>> One example that keeps popping up in my mind is callback functions that
>>> cannot propagate errors, at least not the CPython way. I have a couple of
>>> those in lxml, even some returning void. So I wrap their code in a bare
>>> try-except and when an exception strikes, I set a C error flag to tell the
>>> C library that something went wrong and return normally. No Python code
>>> outside of the try block. But Cython still generates code for unraisable
>>> errors. Why? Because the internal code that handles the bare except clause
>>> may fail and raise an exception. How about that?
>>>
>>>
>>>> the need to explicitly declare "except *" keeps coming up again and
>>>> again, and is really a blemish on the usability of Cython. When teaching
>>>> people Cython, then it's really irritating to have to follow "all you need
>>>> to do is add some 'cdef' and some types" with "and then you need to
>>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>>> looks very elegant until that point...
>>>
>>> I know what this feels like. The problem is that these things *are* complex.
>>
>> Yes. We've been wrestling with this issue almost since Cython's inception...
>>
>> I like Mark's two-function idea, with the caveat that f(bad_argument)
>> now behaves quite differently than (&f)[0](bad_argument) for even more
>> obscure reasons. But it may be the way to go.
>>
>> The other option is to embed the error behavior into the signature and
>> require casts to explicitly go from one to the other. This would
>> probably require a notation for never raising an exception (e.g.
>> "except -"). Cdef public or api functions could require an except
>> declaration (positive or negative), ordinary cdef functions would be
>> "except *" by default, and cdef extern functions would be "except -"
>> by default.
>
> Only except * and except ? have ever made some sense to me. Except +
> is the most mysterious syntax ever, imho it should have been 'except
> cpperror' or something. And when you try to search for "except +" or
> "except *" etc on docs.cython.org it doesn't find anything, which
> makes it hard for people reading the code and unfamiliar with the
> syntax to figure out what it means. In general I also think decorators
> would have been clearer when defining such functions. Let's please not
> introduce more weird syntax.

"except IDENTIFIER" already has a meaning, and it's nice to have as
few keywords as possible (though I agree for searching). The problem
with decorators is that they don't lend themselves being declarable
part of a type declaration. What would the syntax be for a function
pointer (or extern function) that does propagate Python exceptions?
What about one that throws a C++ exception?

> In any event I don't see why we'd want 'except -', as we're trying to
> get rid of the except clause.

Is it possible to get rid of it entirely? I was thinking we could
provide more natural defaults, but the user might still need to
declare when things are different.

> So you can still get your old behaviour
> for function pointers by not using the except clause and having it
> write unraisable exceptions in the function, but in Cython space you'd
> simply get better semantics (that is, propagating exceptions).

I agree with Dag that this is very similar to the GIL issue, and
should probably be tackled similarly.

- Robert

From dtcaciuc at gmail.com  Thu Feb  2 01:53:02 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Wed, 1 Feb 2012 16:53:02 -0800
Subject: [Cython] distutils extension pxd problem
Message-ID: <CAL41sfRzKTM2d8aqLM4NOaoWHu45VpS-AF4h+vbFP5q29WvcGQ@mail.gmail.com>

Hey everyone,

I bumped into an issue where my .pyx file doesn't see its matching
.pxd file. Here's a build test to show the problem If I change my
target package from `b.a` to just `a`, it works as expected. Running
`cython src/a.pyx` works as expected as well, but not the Extension.

----

PYTHON setup.py build_ext --inplace
PYTHON -c "from b import a"

######## setup.py ########

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension("b.a", ["src/a.pyx"])
]

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules
)

######## b/__init__.py ########

######## src/a.pxd ########

cdef class X:
    cdef object foo

######## src/a.pyx ########

cdef class X:

    def __cinit__(self):
        self.foo = 1

x = X()

----

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.pyx", line 7, in init b.a (src/a.c:793)
  File "a.pyx", line 5, in b.a.X.__cinit__ (src/a.c:488)
AttributeError: 'b.a.X' object has no attribute 'foo'

----

Any idea what's going on here?

Thanks,


Dimitri.

From stefan_ml at behnel.de  Thu Feb  2 08:11:16 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Feb 2012 08:11:16 +0100
Subject: [Cython] distutils extension pxd problem
In-Reply-To: <CAL41sfRzKTM2d8aqLM4NOaoWHu45VpS-AF4h+vbFP5q29WvcGQ@mail.gmail.com>
References: <CAL41sfRzKTM2d8aqLM4NOaoWHu45VpS-AF4h+vbFP5q29WvcGQ@mail.gmail.com>
Message-ID: <4F2A3714.7000704@behnel.de>

Dimitri Tcaciuc, 02.02.2012 01:53:
> I bumped into an issue where my .pyx file doesn't see its matching
> .pxd file. Here's a build test to show the problem If I change my
> target package from `b.a` to just `a`, it works as expected. Running
> `cython src/a.pyx` works as expected as well, but not the Extension.
> 
> ----
> 
> PYTHON setup.py build_ext --inplace
> PYTHON -c "from b import a"
> 
> ######## setup.py ########
> 
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
> 
> ext_modules = [
>     Extension("b.a", ["src/a.pyx"])
> ]
> 
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = ext_modules
> )
> 
> ######## b/__init__.py ########
> 
> ######## src/a.pxd ########
> 
> cdef class X:
>     cdef object foo
> 
> ######## src/a.pyx ########
> 
> cdef class X:
> 
>     def __cinit__(self):
>         self.foo = 1
> 
> x = X()
> 
> ----
> 
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "a.pyx", line 7, in init b.a (src/a.c:793)
>   File "a.pyx", line 5, in b.a.X.__cinit__ (src/a.c:488)
> AttributeError: 'b.a.X' object has no attribute 'foo'
> 
> ----

Any reason you cannot rename "src" to "b"? Because that would fix your
problem. Cython basically uses the same algorithm for finding package
content as Python, i.e. it will look inside the package "b" when looking
for "b.a". And "a.pxd" is not in "b" in your setup.

Stefan

From dtcaciuc at gmail.com  Thu Feb  2 08:12:23 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Wed, 1 Feb 2012 23:12:23 -0800
Subject: [Cython] distutils extension pxd problem
In-Reply-To: <CAL41sfRzKTM2d8aqLM4NOaoWHu45VpS-AF4h+vbFP5q29WvcGQ@mail.gmail.com>
References: <CAL41sfRzKTM2d8aqLM4NOaoWHu45VpS-AF4h+vbFP5q29WvcGQ@mail.gmail.com>
Message-ID: <CAL41sfRgh+c+EexNz_oM_OpF9Q47g22Vy6HnDC1L9Dt1Y00X4Q@mail.gmail.com>

Ok, so I narrowed the problem down to
https://github.com/cython/cython/blob/master/Cython/Compiler/Main.py#L223.
At this point, it looks like if target extension name is `x.y.z`, the
pxd must either be called `x.y.z.pxd` and be located in project root
(I believe this is Pyrex convention?) or be in the exact x/y/z.pxd
directory structure, and each of the parents have to be a package (ie.
contain __init__.[py,pyx,pyd]).

Again, this looks like a problem only if module name is nested. If
this is along the right lines, I'd be happy to make some
clarifications to
http://docs.cython.org/src/userguide/sharing_declarations.html#search-paths-for-definition-files.

It looks like there's a conflict between the Extension name parameter
(which also says where the module gets installed in package tree) and
the name of the actual .so file, which sometimes one needs to
customize (eg. I need to compile x/y/z.pyx to x/y/_z.so since I'd like
to have z.py with some extra bits in them. In this case `cythonize`
seems to ignore the extension name, goes ahead and names the output
`z.so`)

I recon you guys probably had plenty of discussions on this topic. Is
there a general direction where you're taking the whole include/pxd
discovery system or is it staying where it is right now?


Dimitri.


On Wed, Feb 1, 2012 at 4:53 PM, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> Hey everyone,
>
> I bumped into an issue where my .pyx file doesn't see its matching
> .pxd file. Here's a build test to show the problem If I change my
> target package from `b.a` to just `a`, it works as expected. Running
> `cython src/a.pyx` works as expected as well, but not the Extension.
>
> ----
>
> PYTHON setup.py build_ext --inplace
> PYTHON -c "from b import a"
>
> ######## setup.py ########
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> ext_modules = [
> ? ?Extension("b.a", ["src/a.pyx"])
> ]
>
> setup(
> ? ?cmdclass = {'build_ext': build_ext},
> ? ?ext_modules = ext_modules
> )
>
> ######## b/__init__.py ########
>
> ######## src/a.pxd ########
>
> cdef class X:
> ? ?cdef object foo
>
> ######## src/a.pyx ########
>
> cdef class X:
>
> ? ?def __cinit__(self):
> ? ? ? ?self.foo = 1
>
> x = X()
>
> ----
>
> Traceback (most recent call last):
> ?File "<string>", line 1, in <module>
> ?File "a.pyx", line 7, in init b.a (src/a.c:793)
> ?File "a.pyx", line 5, in b.a.X.__cinit__ (src/a.c:488)
> AttributeError: 'b.a.X' object has no attribute 'foo'
>
> ----
>
> Any idea what's going on here?
>
> Thanks,
>
>
> Dimitri.

From dtcaciuc at gmail.com  Thu Feb  2 08:24:43 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Wed, 1 Feb 2012 23:24:43 -0800
Subject: [Cython] distutils extension pxd problem
Message-ID: <CAL41sfRcp2CZcG3qDKJqRHmOYx5+=YuqcFLCXJdoXyozNtTBUg@mail.gmail.com>

> Date: Thu, 02 Feb 2012 08:11:16 +0100
> From: Stefan Behnel <stefan_ml at behnel.de>
> To: Core developer mailing list of the Cython compiler
> ? ? ? ?<cython-devel at python.org>
> Subject: Re: [Cython] distutils extension pxd problem
> Message-ID: <4F2A3714.7000704 at behnel.de>
> Content-Type: text/plain; charset=UTF-8
>
> Dimitri Tcaciuc, 02.02.2012 01:53:
>> I bumped into an issue where my .pyx file doesn't see its matching
>> .pxd file. Here's a build test to show the problem If I change my
>> target package from `b.a` to just `a`, it works as expected. Running
>> `cython src/a.pyx` works as expected as well, but not the Extension.
>>
>> ----
>>
>> PYTHON setup.py build_ext --inplace
>> PYTHON -c "from b import a"
>>
>> ######## setup.py ########
>>
>> from distutils.core import setup
>> from distutils.extension import Extension
>> from Cython.Distutils import build_ext
>>
>> ext_modules = [
>> ? ? Extension("b.a", ["src/a.pyx"])
>> ]
>>
>> setup(
>> ? ? cmdclass = {'build_ext': build_ext},
>> ? ? ext_modules = ext_modules
>> )
>>
>> ######## b/__init__.py ########
>>
>> ######## src/a.pxd ########
>>
>> cdef class X:
>> ? ? cdef object foo
>>
>> ######## src/a.pyx ########
>>
>> cdef class X:
>>
>> ? ? def __cinit__(self):
>> ? ? ? ? self.foo = 1
>>
>> x = X()
>>
>> ----
>>
>> Traceback (most recent call last):
>> ? File "<string>", line 1, in <module>
>> ? File "a.pyx", line 7, in init b.a (src/a.c:793)
>> ? File "a.pyx", line 5, in b.a.X.__cinit__ (src/a.c:488)
>> AttributeError: 'b.a.X' object has no attribute 'foo'
>>
>> ----
>
> Any reason you cannot rename "src" to "b"? Because that would fix your
> problem. Cython basically uses the same algorithm for finding package
> content as Python, i.e. it will look inside the package "b" when looking
> for "b.a". And "a.pxd" is not in "b" in your setup.
>
> Stefan
>

This certainly looks like the cleanest solution now. I have some mixed
C++ bits in src/ so I thought to keep compiled/interpreted bits
separate, if possible.


Dimitri.

From stefan_ml at behnel.de  Thu Feb  2 09:21:11 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Feb 2012 09:21:11 +0100
Subject: [Cython] Cython's view on a common benchmark suite (was: Re:
 [Speed] Buildbot Status)
In-Reply-To: <CAP1=2W5MK_Df_y5a95C4uuQnYvVS-5T18533LzLKU4Qt7Oxu8g@mail.gmail.com>
References: <4F21B5AE.2080304@rehfisch.de>
	<CAP1=2W74MADTQMK=YkRzGXd32sFessBQXmSjauzXiaWpY4Ej=w@mail.gmail.com>
	<CAK5idxQkZ2X100dEBDw5_p3OVUTAJh8XiW3M_xcXtmVoswp9OQ@mail.gmail.com>
	<CAP1=2W49khY=nFRHLf52ztwEUhUpt-ap0xNFTLs0mAiTnCT3gA@mail.gmail.com>
	<4F281DA4.7010300@paulgraydon.co.uk>
	<CAP1=2W6ncefUJOYbUc-C4fSdwLWFEZV7BOseBO1mx1P8-xK4jA@mail.gmail.com>
	<CAK5idxSSuboEt+4N6zZ-dgXV-57Giw=4HBY+0jHFSxRtBCgz6w@mail.gmail.com>
	<CAP1=2W6nijY1x1tK+8+t1rMby8LRE1cXUT9q1fsBF+HJYyFrwg@mail.gmail.com>
	<CAK5idxTLNHXvPhB7PQmZYc0mvBQxL4WUjjL70Lh=L3rKiNouiA@mail.gmail.com>
	<CAP1=2W5f=Pktbpf3rTCF8OjUVenOYYP2chDW8u-RTuaAA=NPBA@mail.gmail.com>
	<CAK5idxRdO2kfKxGwoxwoPO9VCz1cpLFb61tBK79v5pq4RF4ZOA@mail.gmail.com>
	<2403B6B2-6077-4061-B85E-C32B8268B6FC@gmail.com>
	<CAK5idxTs4sEZUQkJavV6-DHapvSBNAE=fJ91h8Z2N+VcOydbXw@mail.gmail.com>
	<CAP1=2W5MK_Df_y5a95C4uuQnYvVS-5T18533LzLKU4Qt7Oxu8g@mail.gmail.com>
Message-ID: <4F2A4777.2080705@behnel.de>

Brett Cannon, 01.02.2012 18:25:
> to prevent this from either ending up in a dead-end because of this, we
> need to first decide where the canonical set of Python VM benchmarks are
> going to live. I say hg.python.org/benchmarks for two reasons. One is that
> Antoine has already done work there to port some of the benchmarks so there
> is at least some there that are ready to be  run under Python 3 (and the
> tooling is in place to create separate Python 2 and Python 3 benchmark
> suites). Two, this can be a test of having the various VM contributors work
> out of hg.python.org if we are ever going to break the stdlib out for
> shared development. At worst we can simply take the changes made at
> pypy/benchmarks that apply to just the unladen benchmarks that exists, and
> at best merge the two sets (manually) into one benchmark suite so PyPy
> doesn't lose anything for Python 2 measurements that they have written and
> CPython doesn't lose any of its Python 3 benchmarks that it has created.
> 
> How does that sound?

+1

FWIW, Cython currently uses both benchmark suites, that of PyPy (in Py2.7)
and that of hg.python.org (in Py2.7 and 3.3), but without codespeed
integration and also without a dedicated server for benchmark runs. So the
results are unfortunately not accurate enough to spot minor changes even
over time.

https://sage.math.washington.edu:8091/hudson/view/bench/

We would like to join in on speed.python.org, once it's clear how the
benchmarks will be run and how the data uploads work and all that. It
already proved a bit tricky to get Cython integrated with the benchmark
runner on our side, and I'm planning to rewrite that integration at some
point, but it should already be doable to get "something" to work now.

I should also note that we don't currently support the whole benchmark
suite, so there must be a way to record individual benchmark results even
in the face of failures in other benchmarks. Basically, speed.python.org
would be useless for us if a failure in a single benchmark left us without
any performance data at all, because it will still take us some time to get
to 100% compliance and we would like to know if anything on that road has a
performance impact. Currently, we apply a short patch that adds a
try-except to the benchmark runner's main loop before starting the
measurements, because otherwise it would just bail out completely on a
single failure. Oh, and we also patch the benchmarks to remove references
to __file__ because of CPython issue 13429, although we may be able to work
around that at some point, specifically when doing on-the-fly compilation
during imports.

http://bugs.python.org/issue13429

Also note that benchmarks that only test C implemented stdlib modules (re,
pickle, json) are useless for Cython because they would only end up timing
the exact same code as for plain CPython.

Another test that is useless for us is the "mako" benchmark, because most
of what it does is to run generated code. There is currently no way for
Cython to hook into that, so we're out of the game here.

We also don't care about program startup tests, obviously, because we know
that Cython's compiler overhead plus an optimising gcc run will render them
meaningless anyway. I like the fact that there's still an old hg_startup
timing result lingering around from the time before I disabled that test,
telling us that Cython runs it 99.68% slower than CPython. Got to beat
that. 8-)

Stefan

From d.s.seljebotn at astro.uio.no  Thu Feb  2 13:19:13 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Feb 2012 13:19:13 +0100
Subject: [Cython] memoryview slices can't be None?
Message-ID: <4F2A7F41.6010303@astro.uio.no>

I just realized that

cdef int[:] a = None

raises an exception; even though I'd argue that 'a' is of the 
"reference" kind of type where Cython usually allow None (i.e., "cdef 
MyClass b = None" is allowed even if type(None) is NoneType). Is this a 
bug or not, and is it possible to do something about it?

Dag Sverre

From markflorisson88 at gmail.com  Thu Feb  2 22:10:49 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Feb 2012 21:10:49 +0000
Subject: [Cython] [cython-users] Re: How to find out where an
 AttributeError is ignored
In-Reply-To: <CADiQ+QBKRnad6c1nq_wk7N3roUHuVZqcR9z_yByk3PjtEMieLA@mail.gmail.com>
References: <2bdc0373-c865-4c88-9764-b520e7dcf707@t16g2000vba.googlegroups.com>
	<c1e78ed5-8b25-4344-a280-7b8220c200c3@k29g2000vbl.googlegroups.com>
	<CADiQ+QBRJxgV+qMpXifGzx-_ME71bnpZEZJ9EL2TTEiqPKAU6A@mail.gmail.com>
	<a772805d-3079-4149-83df-0ab048f7e07a@r16g2000yqi.googlegroups.com>
	<0c7296f3-085d-4edd-8aaa-4062bb75d175@h6g2000yqk.googlegroups.com>
	<CANg26EXRESwqyoEw3jr5jmHbADs0DXRAB-qum22ckea5L_8rvQ@mail.gmail.com>
	<CANg26EUgf5YHKM889tXdwT-3LnfT9vLGdUOxMWuqxZO0rA8d=A@mail.gmail.com>
	<4F22D7A2.1050806@behnel.de> <4F230312.9050506@astro.uio.no>
	<4F23109E.3030203@behnel.de>
	<CADiQ+QB0gOFsBPistJsTs4MaOwsrkok_QFAY9wfMpjU0sohs+g@mail.gmail.com>
	<CANg26EV72kuYwiFRJPxbygoa-w8qHcvYfejzEZrKhR7CtdFYDQ@mail.gmail.com>
	<CADiQ+QBKRnad6c1nq_wk7N3roUHuVZqcR9z_yByk3PjtEMieLA@mail.gmail.com>
Message-ID: <CANg26EUhdqGj0NrUqYTqB-eASGboEZF0jTpnLvK6XRJD+3bxrQ@mail.gmail.com>

On 1 February 2012 18:50, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Jan 31, 2012 at 8:30 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 31 January 2012 02:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>>>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>>>>> mark florisson, 27.01.2012 17:30:
>>>>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com> ?wrote:
>>>>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de> ?wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>>>>
>>>>>>>>> How can I patch Cython such that some more information on the location
>>>>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>>>>> message
>>>>>>>>> ? ?Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>>>>> object has no attribute '_modulus' in ?ignored
>>>>>>>>>
>>>>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>>>>> responsible for printing the warning?
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Simon
>>>>>>>>
>>>>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>>>>> typically two reasons for unraisable exceptions:
>>>>>>>>
>>>>>>>> ? ? 1) as Robert mentioned, a function that does not allow propagation
>>>>>>>> of exceptions, e.g.
>>>>>>>>
>>>>>>>> ? ? ? ? cdef int func():
>>>>>>>> ? ? ? ? ? ? raise Exception
>>>>>>>>
>>>>>>>> ? ? ? ? Here there is no way to propagate the raised exception, so
>>>>>>>> instead one should write something like
>>>>>>>>
>>>>>>>> ? ? ? ? ? ? cdef int func() except -1: ...
>>>>>>>>
>>>>>>>> ? ? ? ? Alternatively one may use 'except *' in case there is no error
>>>>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>>>>> "-1 may or may not indicate an error".
>>>>>>>>
>>>>>>>> ? ? 2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>>>>
>>>>>>>> For functions the right thing is to add an except clause, for
>>>>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>>>>
>>>>>>>> ? ? try:
>>>>>>>> ? ? ? ? ...
>>>>>>>> ? ? except:
>>>>>>>> ? ? ? ? traceback.print_exc()
>>>>>>>>
>>>>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>>>>
>>>>>>> Actually, I don't see why the default is to write unraisable
>>>>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>>>>> and have callers do the check (i.e. make it implicitly "except *").
>>>>>
>>>>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>>>>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>>>>
>>>>> And for floating point, we could make our own NaN -- that's obscure enough
>>>>> that it could probably be made "except cython.cython_exception_nan" by
>>>>> default, not "except? cython.cython_exception_nan".
>>>>
>>>> The problem with that is that we can't be sure that Cython will be the only
>>>> caller. So exceptions may still not propagate in cases, and users will have
>>>> to know about these "obscure" values and that they must deal with them
>>>> manually then.
>>>>
>>>> You could add that we'd just have to disable this when user code takes a
>>>> pointer from a function, but then, how many rules are there that users will
>>>> have to learn and remember after such a change? And what's that for a
>>>> language that changes the calling semantics of a function because way down
>>>> in the code someone happens to take a pointer to it?
>>>>
>>>>
>>>>>>> Was this not implemented because Cython only knows whether functions
>>>>>>> may propagate exceptions at code generation time by looking at the
>>>>>>> presence of an error label?
>>>>>>> Maybe it could keep code insertion points around for every call to
>>>>>>> such a potential function and if the function uses the error label
>>>>>>> have the caller perform the check? Although I do forsee problems for
>>>>>>> external such functions... maybe Cython could have it's own
>>>>>>> threadstate regardless of the GIL which would indicate whether an
>>>>>>> error has occurred? e.g. CyErr_Occurred()?
>>>>>>
>>>>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>>>>> default.
>>>>>
>>>>> Still,
>>>>
>>>> I wasn't really advocating this behaviour, just indicating that it's hard
>>>> to do "better", because this "better" isn't all that clear. It's also not
>>>> "better" for all code, which means that we get from one trade-off to
>>>> another, while breaking existing code at the same time. Not exactly
>>>> paradise on either side of the tunnel.
>>>
>>> I still feel like we're stuck in the wrong default. I'd rather require
>>> more work to interact with C libraries than require more work to
>>> convert innocent-looking Python to Cython.
>>>
>>>> One example that keeps popping up in my mind is callback functions that
>>>> cannot propagate errors, at least not the CPython way. I have a couple of
>>>> those in lxml, even some returning void. So I wrap their code in a bare
>>>> try-except and when an exception strikes, I set a C error flag to tell the
>>>> C library that something went wrong and return normally. No Python code
>>>> outside of the try block. But Cython still generates code for unraisable
>>>> errors. Why? Because the internal code that handles the bare except clause
>>>> may fail and raise an exception. How about that?
>>>>
>>>>
>>>>> the need to explicitly declare "except *" keeps coming up again and
>>>>> again, and is really a blemish on the usability of Cython. When teaching
>>>>> people Cython, then it's really irritating to have to follow "all you need
>>>>> to do is add some 'cdef' and some types" with "and then you need to
>>>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>>>> looks very elegant until that point...
>>>>
>>>> I know what this feels like. The problem is that these things *are* complex.
>>>
>>> Yes. We've been wrestling with this issue almost since Cython's inception...
>>>
>>> I like Mark's two-function idea, with the caveat that f(bad_argument)
>>> now behaves quite differently than (&f)[0](bad_argument) for even more
>>> obscure reasons. But it may be the way to go.
>>>
>>> The other option is to embed the error behavior into the signature and
>>> require casts to explicitly go from one to the other. This would
>>> probably require a notation for never raising an exception (e.g.
>>> "except -"). Cdef public or api functions could require an except
>>> declaration (positive or negative), ordinary cdef functions would be
>>> "except *" by default, and cdef extern functions would be "except -"
>>> by default.
>>
>> Only except * and except ? have ever made some sense to me. Except +
>> is the most mysterious syntax ever, imho it should have been 'except
>> cpperror' or something. And when you try to search for "except +" or
>> "except *" etc on docs.cython.org it doesn't find anything, which
>> makes it hard for people reading the code and unfamiliar with the
>> syntax to figure out what it means. In general I also think decorators
>> would have been clearer when defining such functions. Let's please not
>> introduce more weird syntax.
>
> "except IDENTIFIER" already has a meaning, and it's nice to have as
> few keywords as possible (though I agree for searching). The problem
> with decorators is that they don't lend themselves being declarable
> part of a type declaration. What would the syntax be for a function
> pointer (or extern function) that does propagate Python exceptions?
> What about one that throws a C++ exception?

Right, that's why I said 'when defining the function' :) The except
clause would still be used for declarations (and may also be used in
definitions, if wanted).
But it's best to provide the best possible defaults, in which case you
only need except in rare cases.

>> In any event I don't see why we'd want 'except -', as we're trying to
>> get rid of the except clause.
>
> Is it possible to get rid of it entirely? I was thinking we could
> provide more natural defaults, but the user might still need to
> declare when things are different.
>
>> So you can still get your old behaviour
>> for function pointers by not using the except clause and having it
>> write unraisable exceptions in the function, but in Cython space you'd
>> simply get better semantics (that is, propagating exceptions).
>
> I agree with Dag that this is very similar to the GIL issue, and
> should probably be tackled similarly.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Thu Feb  2 22:16:29 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Feb 2012 21:16:29 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2A7F41.6010303@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
Message-ID: <CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>

On 2 February 2012 12:19, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> I just realized that
>
> cdef int[:] a = None
>
> raises an exception; even though I'd argue that 'a' is of the "reference"
> kind of type where Cython usually allow None (i.e., "cdef MyClass b = None"
> is allowed even if type(None) is NoneType). Is this a bug or not, and is it
> possible to do something about it?
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Yeah I disabled that quite early. It was supposed to be working but
gave a lot of trouble in cases (segfaults, mainly). At the time I was
trying to get rid of all the segfaults and get the basic functionality
working, so I disabled it. Personally, I have never liked how things
can be None unchecked. I personally prefer to write

cdef foo(obj=None):
    cdef int[:] a
    if obj is None:
        obj = ...
    a = obj

Often you forget to write 'not None' when declaring the parameter (and
apparently that it only allowed for 'def' functions).

As such, I never bothered to re-enable it. However, it does support
control flow with uninitialized slices, and will raise an error if it
is uninitialized. Do we want this behaviour (e.g. for consistency)?

From d.s.seljebotn at astro.uio.no  Thu Feb  2 22:38:15 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Feb 2012 22:38:15 +0100
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
Message-ID: <4F2B0247.8040008@astro.uio.no>

On 02/02/2012 10:16 PM, mark florisson wrote:
> On 2 February 2012 12:19, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> I just realized that
>>
>> cdef int[:] a = None
>>
>> raises an exception; even though I'd argue that 'a' is of the "reference"
>> kind of type where Cython usually allow None (i.e., "cdef MyClass b = None"
>> is allowed even if type(None) is NoneType). Is this a bug or not, and is it
>> possible to do something about it?
>>
>> Dag Sverre
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Yeah I disabled that quite early. It was supposed to be working but
> gave a lot of trouble in cases (segfaults, mainly). At the time I was
> trying to get rid of all the segfaults and get the basic functionality
> working, so I disabled it. Personally, I have never liked how things

Well, you can segfault quite easily with

cdef MyClass a = None
print a.field

so it doesn't make sense to slices different from cdef classes IMO.

> can be None unchecked. I personally prefer to write
>
> cdef foo(obj=None):
>      cdef int[:] a
>      if obj is None:
>          obj = ...
>      a = obj
>
> Often you forget to write 'not None' when declaring the parameter (and
> apparently that it only allowed for 'def' functions).
>
> As such, I never bothered to re-enable it. However, it does support
> control flow with uninitialized slices, and will raise an error if it
> is uninitialized. Do we want this behaviour (e.g. for consistency)?

When in doubt, go for consistency. So +1 for that reason. I do believe 
that setting stuff to None is rather vital in Python.

What I typically do is more like this:

def f(double[:] input, double[:] out=None):
     if out is None:
         out = np.empty_like(input)
     ...

Having to use another variable name is a bit of a pain. (Come on -- do 
you use "a" in real code? What do you actually call "the other obj"? I 
sometimes end up with "out_" and so on, but it creates smelly code quite 
quickly.)

It's easy to segfault with cdef classes anyway, so decent nonechecking 
should be implemented at some point, and then memoryviews would use the 
same mechanisms. Java has decent null-checking...

Dag Sverre

From markflorisson88 at gmail.com  Fri Feb  3 00:09:09 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Feb 2012 23:09:09 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2B0247.8040008@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
Message-ID: <CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>

On 2 February 2012 21:38, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/02/2012 10:16 PM, mark florisson wrote:
>>
>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> I just realized that
>>>
>>> cdef int[:] a = None
>>>
>>> raises an exception; even though I'd argue that 'a' is of the "reference"
>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>> None"
>>> is allowed even if type(None) is NoneType). Is this a bug or not, and is
>>> it
>>> possible to do something about it?
>>>
>>> Dag Sverre
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>> Yeah I disabled that quite early. It was supposed to be working but
>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>> trying to get rid of all the segfaults and get the basic functionality
>> working, so I disabled it. Personally, I have never liked how things
>
>
> Well, you can segfault quite easily with
>
> cdef MyClass a = None
> print a.field
>
> so it doesn't make sense to slices different from cdef classes IMO.
>
>
>> can be None unchecked. I personally prefer to write
>>
>> cdef foo(obj=None):
>> ? ? cdef int[:] a
>> ? ? if obj is None:
>> ? ? ? ? obj = ...
>> ? ? a = obj
>>
>> Often you forget to write 'not None' when declaring the parameter (and
>> apparently that it only allowed for 'def' functions).
>>
>> As such, I never bothered to re-enable it. However, it does support
>> control flow with uninitialized slices, and will raise an error if it
>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>
>
> When in doubt, go for consistency. So +1 for that reason. I do believe that
> setting stuff to None is rather vital in Python.
>
> What I typically do is more like this:
>
> def f(double[:] input, double[:] out=None):
> ? ?if out is None:
> ? ? ? ?out = np.empty_like(input)
> ? ?...
>
> Having to use another variable name is a bit of a pain. (Come on -- do you
> use "a" in real code? What do you actually call "the other obj"? I sometimes
> end up with "out_" and so on, but it creates smelly code quite quickly.)

No, it was just a contrived example.

> It's easy to segfault with cdef classes anyway, so decent nonechecking
> should be implemented at some point, and then memoryviews would use the same
> mechanisms. Java has decent null-checking...
>

The problem with none checking is that it has to occur at every point.
With initialized slices the control flow knows when the slices are
initialized, or when they might not be (and it can raise a
compile-time or runtime error, instead of a segfault if you're lucky).
I'm fine with implementing the behaviour, I just always left it at the
bottom of my todo list.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Fri Feb  3 18:53:02 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Feb 2012 18:53:02 +0100
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
Message-ID: <4F2C1EFE.4050904@astro.uio.no>

On 02/03/2012 12:09 AM, mark florisson wrote:
> On 2 February 2012 21:38, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>
>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no>    wrote:
>>>>
>>>> I just realized that
>>>>
>>>> cdef int[:] a = None
>>>>
>>>> raises an exception; even though I'd argue that 'a' is of the "reference"
>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>> None"
>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and is
>>>> it
>>>> possible to do something about it?
>>>>
>>>> Dag Sverre
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>> Yeah I disabled that quite early. It was supposed to be working but
>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>> trying to get rid of all the segfaults and get the basic functionality
>>> working, so I disabled it. Personally, I have never liked how things
>>
>>
>> Well, you can segfault quite easily with
>>
>> cdef MyClass a = None
>> print a.field
>>
>> so it doesn't make sense to slices different from cdef classes IMO.
>>
>>
>>> can be None unchecked. I personally prefer to write
>>>
>>> cdef foo(obj=None):
>>>      cdef int[:] a
>>>      if obj is None:
>>>          obj = ...
>>>      a = obj
>>>
>>> Often you forget to write 'not None' when declaring the parameter (and
>>> apparently that it only allowed for 'def' functions).
>>>
>>> As such, I never bothered to re-enable it. However, it does support
>>> control flow with uninitialized slices, and will raise an error if it
>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>
>>
>> When in doubt, go for consistency. So +1 for that reason. I do believe that
>> setting stuff to None is rather vital in Python.
>>
>> What I typically do is more like this:
>>
>> def f(double[:] input, double[:] out=None):
>>     if out is None:
>>         out = np.empty_like(input)
>>     ...
>>
>> Having to use another variable name is a bit of a pain. (Come on -- do you
>> use "a" in real code? What do you actually call "the other obj"? I sometimes
>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>
> No, it was just a contrived example.
>
>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>> should be implemented at some point, and then memoryviews would use the same
>> mechanisms. Java has decent null-checking...
>>
>
> The problem with none checking is that it has to occur at every point.

Well, using control flow analysis etc. it doesn't really. E.g.,

for i in range(a.shape[0]):
     print i
     a[i] *= 3

can be unrolled and none-checks inserted as

print 0
if a is None: raise ....
a[0] *= 3
for i in range(1, a.shape[0]):
     print i
     a[i] *= 3 # no need for none-check

It's very similar to what you'd want to do to pull boundschecking out of 
the loop...

> With initialized slices the control flow knows when the slices are
> initialized, or when they might not be (and it can raise a
> compile-time or runtime error, instead of a segfault if you're lucky).
> I'm fine with implementing the behaviour, I just always left it at the
> bottom of my todo list.

Wasn't saying you should do it, just checking.

I'm still not sure about this. I think what I'd really like is

  a) Stop cdef classes from being None as well

  b) Sort-of deprecate cdef in favor of cast/assertion type statements 
that help the type inferences:

def f(arr):
     if arr is None:
         arr = ...
     arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
                       # acts as statement, with a specific point
                       # for the none-check
     ...

or even:

def f(arr):
     if arr is None:
         return 'foo'
     else:
         arr = int[:](arr) # takes effect *here*, does none-check
         ...
     # arr still typed as int[:] here

If we can make this work well enough with control flow analysis I'd 
never cdef declare local vars again :-)

Dag

From d.s.seljebotn at astro.uio.no  Fri Feb  3 18:59:25 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Feb 2012 18:59:25 +0100
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2C1EFE.4050904@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>	<4F2B0247.8040008@astro.uio.no>	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
Message-ID: <4F2C207D.3050100@astro.uio.no>

On 02/03/2012 06:53 PM, Dag Sverre Seljebotn wrote:
> On 02/03/2012 12:09 AM, mark florisson wrote:
>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>
>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>>
>>>>> I just realized that
>>>>>
>>>>> cdef int[:] a = None
>>>>>
>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>> "reference"
>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>> None"
>>>>> is allowed even if type(None) is NoneType). Is this a bug or not,
>>>>> and is
>>>>> it
>>>>> possible to do something about it?
>>>>>
>>>>> Dag Sverre
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>>
>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>> trying to get rid of all the segfaults and get the basic functionality
>>>> working, so I disabled it. Personally, I have never liked how things
>>>
>>>
>>> Well, you can segfault quite easily with
>>>
>>> cdef MyClass a = None
>>> print a.field
>>>
>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>
>>>
>>>> can be None unchecked. I personally prefer to write
>>>>
>>>> cdef foo(obj=None):
>>>> cdef int[:] a
>>>> if obj is None:
>>>> obj = ...
>>>> a = obj
>>>>
>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>> apparently that it only allowed for 'def' functions).
>>>>
>>>> As such, I never bothered to re-enable it. However, it does support
>>>> control flow with uninitialized slices, and will raise an error if it
>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>
>>>
>>> When in doubt, go for consistency. So +1 for that reason. I do
>>> believe that
>>> setting stuff to None is rather vital in Python.
>>>
>>> What I typically do is more like this:
>>>
>>> def f(double[:] input, double[:] out=None):
>>> if out is None:
>>> out = np.empty_like(input)
>>> ...
>>>
>>> Having to use another variable name is a bit of a pain. (Come on --
>>> do you
>>> use "a" in real code? What do you actually call "the other obj"? I
>>> sometimes
>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>
>> No, it was just a contrived example.
>>
>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>> should be implemented at some point, and then memoryviews would use
>>> the same
>>> mechanisms. Java has decent null-checking...
>>>
>>
>> The problem with none checking is that it has to occur at every point.
>
> Well, using control flow analysis etc. it doesn't really. E.g.,
>
> for i in range(a.shape[0]):
> print i
> a[i] *= 3
>
> can be unrolled and none-checks inserted as
>
> print 0
> if a is None: raise ....
> a[0] *= 3
> for i in range(1, a.shape[0]):
> print i
> a[i] *= 3 # no need for none-check
>
> It's very similar to what you'd want to do to pull boundschecking out of
> the loop...
>
>> With initialized slices the control flow knows when the slices are
>> initialized, or when they might not be (and it can raise a
>> compile-time or runtime error, instead of a segfault if you're lucky).
>> I'm fine with implementing the behaviour, I just always left it at the
>> bottom of my todo list.
>
> Wasn't saying you should do it, just checking.
>
> I'm still not sure about this. I think what I'd really like is
>
> a) Stop cdef classes from being None as well

I'm sorry, this was out of place and hardly constructive. I beg you all 
to ignore the above sentence.

The below proposal still works for slices only though, which we can 
still define the semantics for. I rather like it. Is type-inference + 
control flow analysis powerful enough to pull this off with some minor 
changes?

>
> b) Sort-of deprecate cdef in favor of cast/assertion type statements
> that help the type inferences:
>
> def f(arr):
> if arr is None:
> arr = ...
> arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
> # acts as statement, with a specific point
> # for the none-check
> ...
>
> or even:
>
> def f(arr):
> if arr is None:
> return 'foo'
> else:
> arr = int[:](arr) # takes effect *here*, does none-check
> ...
> # arr still typed as int[:] here
>
> If we can make this work well enough with control flow analysis I'd
> never cdef declare local vars again :-)


Dag

From markflorisson88 at gmail.com  Fri Feb  3 19:06:14 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Feb 2012 18:06:14 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2C1EFE.4050904@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
Message-ID: <CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>

On 3 February 2012 17:53, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/03/2012 12:09 AM, mark florisson wrote:
>>
>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>
>>>>
>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>
>>>>>
>>>>> I just realized that
>>>>>
>>>>> cdef int[:] a = None
>>>>>
>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>> "reference"
>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>> None"
>>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and
>>>>> is
>>>>> it
>>>>> possible to do something about it?
>>>>>
>>>>> Dag Sverre
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>>
>>>>
>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>> trying to get rid of all the segfaults and get the basic functionality
>>>> working, so I disabled it. Personally, I have never liked how things
>>>
>>>
>>>
>>> Well, you can segfault quite easily with
>>>
>>> cdef MyClass a = None
>>> print a.field
>>>
>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>
>>>
>>>> can be None unchecked. I personally prefer to write
>>>>
>>>> cdef foo(obj=None):
>>>> ? ? cdef int[:] a
>>>> ? ? if obj is None:
>>>> ? ? ? ? obj = ...
>>>> ? ? a = obj
>>>>
>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>> apparently that it only allowed for 'def' functions).
>>>>
>>>> As such, I never bothered to re-enable it. However, it does support
>>>> control flow with uninitialized slices, and will raise an error if it
>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>
>>>
>>>
>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>> that
>>> setting stuff to None is rather vital in Python.
>>>
>>> What I typically do is more like this:
>>>
>>> def f(double[:] input, double[:] out=None):
>>> ? ?if out is None:
>>> ? ? ? ?out = np.empty_like(input)
>>> ? ?...
>>>
>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>> you
>>> use "a" in real code? What do you actually call "the other obj"? I
>>> sometimes
>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>
>>
>> No, it was just a contrived example.
>>
>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>> should be implemented at some point, and then memoryviews would use the
>>> same
>>> mechanisms. Java has decent null-checking...
>>>
>>
>> The problem with none checking is that it has to occur at every point.
>
>
> Well, using control flow analysis etc. it doesn't really. E.g.,
>
> for i in range(a.shape[0]):
> ? ?print i
> ? ?a[i] *= 3
>
> can be unrolled and none-checks inserted as
>
> print 0
> if a is None: raise ....
> a[0] *= 3
> for i in range(1, a.shape[0]):
> ? ?print i
> ? ?a[i] *= 3 # no need for none-check
>
> It's very similar to what you'd want to do to pull boundschecking out of the
> loop...
>

Oh, definitely. Both optimizations may not always be possible to do,
though. The optimization (for boundschecking) is easier for prange()
than range(), as you can immediately raise an exception as the
exceptional condition may be issued at any iteration.  What do you do
with bounds checking when some accesses are in-bound, and some are
out-of-bound? Do you immediately raise the exception? Are we fine with
aborting (like Fortran compilers do when you ask them for bounds
checking)? And how do you detect that the code doesn't already raise
an exception or break out of the loop itself to prevent the
out-of-bound access? (Unless no exceptions are propagating and no
break/return is used, but exceptions are so very common).

>> With initialized slices the control flow knows when the slices are
>> initialized, or when they might not be (and it can raise a
>> compile-time or runtime error, instead of a segfault if you're lucky).
>> I'm fine with implementing the behaviour, I just always left it at the
>> bottom of my todo list.
>
>
> Wasn't saying you should do it, just checking.
>
> I'm still not sure about this. I think what I'd really like is
>
> ?a) Stop cdef classes from being None as well
>
> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements that
> help the type inferences:
>
> def f(arr):
> ? ?if arr is None:
> ? ? ? ?arr = ...
> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
> ? ?...
>
> or even:
>
> def f(arr):
> ? ?if arr is None:
> ? ? ? ?return 'foo'
> ? ?else:
> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
> ? ? ? ?...
> ? ?# arr still typed as int[:] here
>
> If we can make this work well enough with control flow analysis I'd never
> cdef declare local vars again :-)

Hm, what about the following?

def f(arr):
    if arr is None:
        return 'foo'

    cdef int[:] arr # arr may not be None

> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Fri Feb  3 19:07:09 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Feb 2012 18:07:09 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
Message-ID: <CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>

On 3 February 2012 18:06, mark florisson <markflorisson88 at gmail.com> wrote:
> On 3 February 2012 17:53, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>
>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>
>>>>>
>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>>
>>>>>>
>>>>>> I just realized that
>>>>>>
>>>>>> cdef int[:] a = None
>>>>>>
>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>> "reference"
>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>>> None"
>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and
>>>>>> is
>>>>>> it
>>>>>> possible to do something about it?
>>>>>>
>>>>>> Dag Sverre
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>>
>>>>>
>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>> trying to get rid of all the segfaults and get the basic functionality
>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>
>>>>
>>>>
>>>> Well, you can segfault quite easily with
>>>>
>>>> cdef MyClass a = None
>>>> print a.field
>>>>
>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>
>>>>
>>>>> can be None unchecked. I personally prefer to write
>>>>>
>>>>> cdef foo(obj=None):
>>>>> ? ? cdef int[:] a
>>>>> ? ? if obj is None:
>>>>> ? ? ? ? obj = ...
>>>>> ? ? a = obj
>>>>>
>>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>>> apparently that it only allowed for 'def' functions).
>>>>>
>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>
>>>>
>>>>
>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>> that
>>>> setting stuff to None is rather vital in Python.
>>>>
>>>> What I typically do is more like this:
>>>>
>>>> def f(double[:] input, double[:] out=None):
>>>> ? ?if out is None:
>>>> ? ? ? ?out = np.empty_like(input)
>>>> ? ?...
>>>>
>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>> you
>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>> sometimes
>>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>>
>>>
>>> No, it was just a contrived example.
>>>
>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>> should be implemented at some point, and then memoryviews would use the
>>>> same
>>>> mechanisms. Java has decent null-checking...
>>>>
>>>
>>> The problem with none checking is that it has to occur at every point.
>>
>>
>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>
>> for i in range(a.shape[0]):
>> ? ?print i
>> ? ?a[i] *= 3
>>
>> can be unrolled and none-checks inserted as
>>
>> print 0
>> if a is None: raise ....
>> a[0] *= 3
>> for i in range(1, a.shape[0]):
>> ? ?print i
>> ? ?a[i] *= 3 # no need for none-check
>>
>> It's very similar to what you'd want to do to pull boundschecking out of the
>> loop...
>>
>
> Oh, definitely. Both optimizations may not always be possible to do,
> though. The optimization (for boundschecking) is easier for prange()
> than range(), as you can immediately raise an exception as the
> exceptional condition may be issued at any iteration. ?What do you do
> with bounds checking when some accesses are in-bound, and some are
> out-of-bound? Do you immediately raise the exception? Are we fine with
> aborting (like Fortran compilers do when you ask them for bounds
> checking)? And how do you detect that the code doesn't already raise
> an exception or break out of the loop itself to prevent the
> out-of-bound access? (Unless no exceptions are propagating and no
> break/return is used, but exceptions are so very common).
>
>>> With initialized slices the control flow knows when the slices are
>>> initialized, or when they might not be (and it can raise a
>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>> I'm fine with implementing the behaviour, I just always left it at the
>>> bottom of my todo list.
>>
>>
>> Wasn't saying you should do it, just checking.
>>
>> I'm still not sure about this. I think what I'd really like is
>>
>> ?a) Stop cdef classes from being None as well
>>
>> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements that
>> help the type inferences:
>>
>> def f(arr):
>> ? ?if arr is None:
>> ? ? ? ?arr = ...
>> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
>> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
>> ? ?...
>>
>> or even:
>>
>> def f(arr):
>> ? ?if arr is None:
>> ? ? ? ?return 'foo'
>> ? ?else:
>> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
>> ? ? ? ?...
>> ? ?# arr still typed as int[:] here
>>
>> If we can make this work well enough with control flow analysis I'd never
>> cdef declare local vars again :-)
>
> Hm, what about the following?
>
> def f(arr):
> ? ?if arr is None:
> ? ? ? ?return 'foo'
>
> ? ?cdef int[:] arr # arr may not be None

The above would work in general, until the declaration is lexically
encountered, the object is typed as object.

>> Dag
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Fri Feb  3 19:07:41 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Feb 2012 18:07:41 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
Message-ID: <CANg26EWsrEhhGi62GRvUL_F7u6Gsmi_76BiTwL=e7uCxvfK8Tg@mail.gmail.com>

On 3 February 2012 18:07, mark florisson <markflorisson88 at gmail.com> wrote:
> On 3 February 2012 18:06, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 3 February 2012 17:53, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>>
>>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>>
>>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>>
>>>>>>
>>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>>>
>>>>>>>
>>>>>>> I just realized that
>>>>>>>
>>>>>>> cdef int[:] a = None
>>>>>>>
>>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>>> "reference"
>>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>>>> None"
>>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and
>>>>>>> is
>>>>>>> it
>>>>>>> possible to do something about it?
>>>>>>>
>>>>>>> Dag Sverre
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>>> trying to get rid of all the segfaults and get the basic functionality
>>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>>
>>>>>
>>>>>
>>>>> Well, you can segfault quite easily with
>>>>>
>>>>> cdef MyClass a = None
>>>>> print a.field
>>>>>
>>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>>
>>>>>
>>>>>> can be None unchecked. I personally prefer to write
>>>>>>
>>>>>> cdef foo(obj=None):
>>>>>> ? ? cdef int[:] a
>>>>>> ? ? if obj is None:
>>>>>> ? ? ? ? obj = ...
>>>>>> ? ? a = obj
>>>>>>
>>>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>>>> apparently that it only allowed for 'def' functions).
>>>>>>
>>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>>
>>>>>
>>>>>
>>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>>> that
>>>>> setting stuff to None is rather vital in Python.
>>>>>
>>>>> What I typically do is more like this:
>>>>>
>>>>> def f(double[:] input, double[:] out=None):
>>>>> ? ?if out is None:
>>>>> ? ? ? ?out = np.empty_like(input)
>>>>> ? ?...
>>>>>
>>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>>> you
>>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>>> sometimes
>>>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>>>
>>>>
>>>> No, it was just a contrived example.
>>>>
>>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>>> should be implemented at some point, and then memoryviews would use the
>>>>> same
>>>>> mechanisms. Java has decent null-checking...
>>>>>
>>>>
>>>> The problem with none checking is that it has to occur at every point.
>>>
>>>
>>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>>
>>> for i in range(a.shape[0]):
>>> ? ?print i
>>> ? ?a[i] *= 3
>>>
>>> can be unrolled and none-checks inserted as
>>>
>>> print 0
>>> if a is None: raise ....
>>> a[0] *= 3
>>> for i in range(1, a.shape[0]):
>>> ? ?print i
>>> ? ?a[i] *= 3 # no need for none-check
>>>
>>> It's very similar to what you'd want to do to pull boundschecking out of the
>>> loop...
>>>
>>
>> Oh, definitely. Both optimizations may not always be possible to do,
>> though. The optimization (for boundschecking) is easier for prange()
>> than range(), as you can immediately raise an exception as the
>> exceptional condition may be issued at any iteration. ?What do you do
>> with bounds checking when some accesses are in-bound, and some are
>> out-of-bound? Do you immediately raise the exception? Are we fine with
>> aborting (like Fortran compilers do when you ask them for bounds
>> checking)? And how do you detect that the code doesn't already raise
>> an exception or break out of the loop itself to prevent the
>> out-of-bound access? (Unless no exceptions are propagating and no
>> break/return is used, but exceptions are so very common).
>>
>>>> With initialized slices the control flow knows when the slices are
>>>> initialized, or when they might not be (and it can raise a
>>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>>> I'm fine with implementing the behaviour, I just always left it at the
>>>> bottom of my todo list.
>>>
>>>
>>> Wasn't saying you should do it, just checking.
>>>
>>> I'm still not sure about this. I think what I'd really like is
>>>
>>> ?a) Stop cdef classes from being None as well
>>>
>>> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements that
>>> help the type inferences:
>>>
>>> def f(arr):
>>> ? ?if arr is None:
>>> ? ? ? ?arr = ...
>>> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>>> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
>>> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
>>> ? ?...
>>>
>>> or even:
>>>
>>> def f(arr):
>>> ? ?if arr is None:
>>> ? ? ? ?return 'foo'
>>> ? ?else:
>>> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
>>> ? ? ? ?...
>>> ? ?# arr still typed as int[:] here
>>>
>>> If we can make this work well enough with control flow analysis I'd never
>>> cdef declare local vars again :-)
>>
>> Hm, what about the following?
>>
>> def f(arr):
>> ? ?if arr is None:
>> ? ? ? ?return 'foo'
>>
>> ? ?cdef int[:] arr # arr may not be None
>
> The above would work in general, until the declaration is lexically
> encountered, the object is typed as object.

Hurr, the *variable* is typed as object :)

>>> Dag
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Fri Feb  3 19:15:07 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Feb 2012 19:15:07 +0100
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>	<4F2B0247.8040008@astro.uio.no>	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>	<4F2C1EFE.4050904@astro.uio.no>	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
Message-ID: <4F2C242B.9010309@astro.uio.no>

On 02/03/2012 07:07 PM, mark florisson wrote:
> On 3 February 2012 18:06, mark florisson<markflorisson88 at gmail.com>  wrote:
>> On 3 February 2012 17:53, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no>  wrote:
>>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>>
>>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no>    wrote:
>>>>>
>>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>>
>>>>>>
>>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>>> <d.s.seljebotn at astro.uio.no>      wrote:
>>>>>>>
>>>>>>>
>>>>>>> I just realized that
>>>>>>>
>>>>>>> cdef int[:] a = None
>>>>>>>
>>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>>> "reference"
>>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>>>> None"
>>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and
>>>>>>> is
>>>>>>> it
>>>>>>> possible to do something about it?
>>>>>>>
>>>>>>> Dag Sverre
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>>> trying to get rid of all the segfaults and get the basic functionality
>>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>>
>>>>>
>>>>>
>>>>> Well, you can segfault quite easily with
>>>>>
>>>>> cdef MyClass a = None
>>>>> print a.field
>>>>>
>>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>>
>>>>>
>>>>>> can be None unchecked. I personally prefer to write
>>>>>>
>>>>>> cdef foo(obj=None):
>>>>>>      cdef int[:] a
>>>>>>      if obj is None:
>>>>>>          obj = ...
>>>>>>      a = obj
>>>>>>
>>>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>>>> apparently that it only allowed for 'def' functions).
>>>>>>
>>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>>
>>>>>
>>>>>
>>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>>> that
>>>>> setting stuff to None is rather vital in Python.
>>>>>
>>>>> What I typically do is more like this:
>>>>>
>>>>> def f(double[:] input, double[:] out=None):
>>>>>     if out is None:
>>>>>         out = np.empty_like(input)
>>>>>     ...
>>>>>
>>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>>> you
>>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>>> sometimes
>>>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>>>
>>>>
>>>> No, it was just a contrived example.
>>>>
>>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>>> should be implemented at some point, and then memoryviews would use the
>>>>> same
>>>>> mechanisms. Java has decent null-checking...
>>>>>
>>>>
>>>> The problem with none checking is that it has to occur at every point.
>>>
>>>
>>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>>
>>> for i in range(a.shape[0]):
>>>     print i
>>>     a[i] *= 3
>>>
>>> can be unrolled and none-checks inserted as
>>>
>>> print 0
>>> if a is None: raise ....
>>> a[0] *= 3
>>> for i in range(1, a.shape[0]):
>>>     print i
>>>     a[i] *= 3 # no need for none-check
>>>
>>> It's very similar to what you'd want to do to pull boundschecking out of the
>>> loop...
>>>
>>
>> Oh, definitely. Both optimizations may not always be possible to do,
>> though. The optimization (for boundschecking) is easier for prange()
>> than range(), as you can immediately raise an exception as the
>> exceptional condition may be issued at any iteration.  What do you do
>> with bounds checking when some accesses are in-bound, and some are
>> out-of-bound? Do you immediately raise the exception? Are we fine with
>> aborting (like Fortran compilers do when you ask them for bounds
>> checking)? And how do you detect that the code doesn't already raise
>> an exception or break out of the loop itself to prevent the
>> out-of-bound access? (Unless no exceptions are propagating and no
>> break/return is used, but exceptions are so very common).
>>
>>>> With initialized slices the control flow knows when the slices are
>>>> initialized, or when they might not be (and it can raise a
>>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>>> I'm fine with implementing the behaviour, I just always left it at the
>>>> bottom of my todo list.
>>>
>>>
>>> Wasn't saying you should do it, just checking.
>>>
>>> I'm still not sure about this. I think what I'd really like is
>>>
>>>   a) Stop cdef classes from being None as well
>>>
>>>   b) Sort-of deprecate cdef in favor of cast/assertion type statements that
>>> help the type inferences:
>>>
>>> def f(arr):
>>>     if arr is None:
>>>         arr = ...
>>>     arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>>>                       # acts as statement, with a specific point
>>>                       # for the none-check
>>>     ...
>>>
>>> or even:
>>>
>>> def f(arr):
>>>     if arr is None:
>>>         return 'foo'
>>>     else:
>>>         arr = int[:](arr) # takes effect *here*, does none-check
>>>         ...
>>>     # arr still typed as int[:] here
>>>
>>> If we can make this work well enough with control flow analysis I'd never
>>> cdef declare local vars again :-)
>>
>> Hm, what about the following?
>>
>> def f(arr):
>>     if arr is None:
>>         return 'foo'
>>
>>     cdef int[:] arr # arr may not be None
>
> The above would work in general, until the declaration is lexically
> encountered, the object is typed as object.

This was actually going to be my first proposal :-) That would finally 
define how "cdef" inside of if-statements etc. behave too (simply use 
control flow analysis and treat it like a statement).

But I like int[:] as a way of making it pure Python syntax compatible as 
well. Perhaps the two are orthogonal -- a) make variable declaration a 
statement, b) make cython.int[:](x) do, essentially, a cdef declaration, 
for Python compatability.

Dag

From markflorisson88 at gmail.com  Fri Feb  3 19:26:48 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Feb 2012 18:26:48 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2C242B.9010309@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
	<4F2C242B.9010309@astro.uio.no>
Message-ID: <CANg26EVyGLia1vG2hPy+F7nsnrR5WNGd+0egBftacNZTTaSjNg@mail.gmail.com>

On 3 February 2012 18:15, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/03/2012 07:07 PM, mark florisson wrote:
>>
>> On 3 February 2012 18:06, mark florisson<markflorisson88 at gmail.com>
>> ?wrote:
>>>
>>> On 3 February 2012 17:53, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>>>
>>>>>
>>>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>>
>>>>>>
>>>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>>>> <d.s.seljebotn at astro.uio.no> ? ? ?wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I just realized that
>>>>>>>>
>>>>>>>> cdef int[:] a = None
>>>>>>>>
>>>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>>>> "reference"
>>>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b
>>>>>>>> =
>>>>>>>> None"
>>>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not,
>>>>>>>> and
>>>>>>>> is
>>>>>>>> it
>>>>>>>> possible to do something about it?
>>>>>>>>
>>>>>>>> Dag Sverre
>>>>>>>> _______________________________________________
>>>>>>>> cython-devel mailing list
>>>>>>>> cython-devel at python.org
>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>>>> trying to get rid of all the segfaults and get the basic
>>>>>>> functionality
>>>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Well, you can segfault quite easily with
>>>>>>
>>>>>> cdef MyClass a = None
>>>>>> print a.field
>>>>>>
>>>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>>>
>>>>>>
>>>>>>> can be None unchecked. I personally prefer to write
>>>>>>>
>>>>>>> cdef foo(obj=None):
>>>>>>> ? ? cdef int[:] a
>>>>>>> ? ? if obj is None:
>>>>>>> ? ? ? ? obj = ...
>>>>>>> ? ? a = obj
>>>>>>>
>>>>>>> Often you forget to write 'not None' when declaring the parameter
>>>>>>> (and
>>>>>>> apparently that it only allowed for 'def' functions).
>>>>>>>
>>>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>>>> that
>>>>>> setting stuff to None is rather vital in Python.
>>>>>>
>>>>>> What I typically do is more like this:
>>>>>>
>>>>>> def f(double[:] input, double[:] out=None):
>>>>>> ? ?if out is None:
>>>>>> ? ? ? ?out = np.empty_like(input)
>>>>>> ? ?...
>>>>>>
>>>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>>>> you
>>>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>>>> sometimes
>>>>>> end up with "out_" and so on, but it creates smelly code quite
>>>>>> quickly.)
>>>>>
>>>>>
>>>>>
>>>>> No, it was just a contrived example.
>>>>>
>>>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>>>> should be implemented at some point, and then memoryviews would use
>>>>>> the
>>>>>> same
>>>>>> mechanisms. Java has decent null-checking...
>>>>>>
>>>>>
>>>>> The problem with none checking is that it has to occur at every point.
>>>>
>>>>
>>>>
>>>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>>>
>>>> for i in range(a.shape[0]):
>>>> ? ?print i
>>>> ? ?a[i] *= 3
>>>>
>>>> can be unrolled and none-checks inserted as
>>>>
>>>> print 0
>>>> if a is None: raise ....
>>>> a[0] *= 3
>>>> for i in range(1, a.shape[0]):
>>>> ? ?print i
>>>> ? ?a[i] *= 3 # no need for none-check
>>>>
>>>> It's very similar to what you'd want to do to pull boundschecking out of
>>>> the
>>>> loop...
>>>>
>>>
>>> Oh, definitely. Both optimizations may not always be possible to do,
>>> though. The optimization (for boundschecking) is easier for prange()
>>> than range(), as you can immediately raise an exception as the
>>> exceptional condition may be issued at any iteration. ?What do you do
>>> with bounds checking when some accesses are in-bound, and some are
>>> out-of-bound? Do you immediately raise the exception? Are we fine with
>>> aborting (like Fortran compilers do when you ask them for bounds
>>> checking)? And how do you detect that the code doesn't already raise
>>> an exception or break out of the loop itself to prevent the
>>> out-of-bound access? (Unless no exceptions are propagating and no
>>> break/return is used, but exceptions are so very common).
>>>
>>>>> With initialized slices the control flow knows when the slices are
>>>>> initialized, or when they might not be (and it can raise a
>>>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>>>> I'm fine with implementing the behaviour, I just always left it at the
>>>>> bottom of my todo list.
>>>>
>>>>
>>>>
>>>> Wasn't saying you should do it, just checking.
>>>>
>>>> I'm still not sure about this. I think what I'd really like is
>>>>
>>>> ?a) Stop cdef classes from being None as well
>>>>
>>>> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements
>>>> that
>>>> help the type inferences:
>>>>
>>>> def f(arr):
>>>> ? ?if arr is None:
>>>> ? ? ? ?arr = ...
>>>> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>>>> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
>>>> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
>>>> ? ?...
>>>>
>>>> or even:
>>>>
>>>> def f(arr):
>>>> ? ?if arr is None:
>>>> ? ? ? ?return 'foo'
>>>> ? ?else:
>>>> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
>>>> ? ? ? ?...
>>>> ? ?# arr still typed as int[:] here
>>>>
>>>> If we can make this work well enough with control flow analysis I'd
>>>> never
>>>> cdef declare local vars again :-)
>>>
>>>
>>> Hm, what about the following?
>>>
>>> def f(arr):
>>> ? ?if arr is None:
>>> ? ? ? ?return 'foo'
>>>
>>> ? ?cdef int[:] arr # arr may not be None
>>
>>
>> The above would work in general, until the declaration is lexically
>> encountered, the object is typed as object.
>
>
> This was actually going to be my first proposal :-) That would finally
> define how "cdef" inside of if-statements etc. behave too (simply use
> control flow analysis and treat it like a statement).

Block-local declarations are definitely something we want, although I
think it would require some more (non-trivial) changes to the
compiler.

Maybe the cleanup code from functions, as well as the temp handling
etc could be re-factored to a BlockNode, that all block nodes could
subclass. They'd have to instantiate new symbol table environments as
well. I'm not yet entirely sure what else would be involved in the
implementation of that.

> But I like int[:] as a way of making it pure Python syntax compatible as
> well. Perhaps the two are orthogonal -- a) make variable declaration a
> statement, b) make cython.int[:](x) do, essentially, a cdef declaration, for
> Python compatability.
>

Don't we have cython.declare() for that? e.g.

    arr = cython.declare(cython.int[:])

That would also be treated as a statement like normal declarations (if
and when implemented).

> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Fri Feb  3 21:27:21 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 4 Feb 2012 00:27:21 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
	<4F21A92F.6050401@behnel.de>
	<CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>
Message-ID: <CAKGHGPRMWWMVbpWh8PH6M2csT8w1NAMiv44UHk7rro7V0jJLTw@mail.gmail.com>

2012/1/26 mark florisson <markflorisson88 at gmail.com>:
> On 26 January 2012 19:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 26.01.2012 20:15:
>>> On 26 January 2012 18:53, Stefan Behnel wrote:
>>>> mark florisson, 26.01.2012 16:20:
>>>>> I think this problem can trivially be solved by creating a ProxyNode
>>>>> that should never be replaced by any transform, but it's argument may
>>>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>>>> create your CloneNodes.
>>>>
>>>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
>>>
>>> It wouldn't be a replacement, merely an addition (an extra indirection).
>>
>> What I was trying to say was that a ProxyNode would always be required by a
>> CloneNode, but I don't see where a ProxyNode would be needed outside of a
>> CloneNode. So it seems rather redundant and I don't know if we need a
>> separate node for it.
>
> Yes it would be needed only for that, but I think the only real
> alternative is to not use CloneNode at all, i.e. make the
> transformation Dag mentioned, where you create new rhs (NameNode?)
> references to the temporary result.
>

Now it seems to be the only case when we got problem like this. It
means that clones may be safely created at very late stage.
So transforming CascadeAssignment into SingleAssignments doesn't solve
generic problem.

I tried to implement conditional inlining the same problem may happen
there (ConditionalCallNode owns arguments and replaces
SimpleCallNode's args with clones). Splitting analyse_expressions()
would help. On the other hand moving this optimization after
OptimizeBuiltinCalls() would help too.

-- 
vitja.

From markflorisson88 at gmail.com  Fri Feb  3 22:59:03 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Feb 2012 21:59:03 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
Message-ID: <CANg26EWRU9LAAKUhXm-GE4gR0axcdsrXNNOVPQe2oAaTT6UJQw@mail.gmail.com>

On 3 February 2012 18:06, mark florisson <markflorisson88 at gmail.com> wrote:
> On 3 February 2012 17:53, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>
>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>
>>>>>
>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>>
>>>>>>
>>>>>> I just realized that
>>>>>>
>>>>>> cdef int[:] a = None
>>>>>>
>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>> "reference"
>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>>>> None"
>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and
>>>>>> is
>>>>>> it
>>>>>> possible to do something about it?
>>>>>>
>>>>>> Dag Sverre
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>>
>>>>>
>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>> trying to get rid of all the segfaults and get the basic functionality
>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>
>>>>
>>>>
>>>> Well, you can segfault quite easily with
>>>>
>>>> cdef MyClass a = None
>>>> print a.field
>>>>
>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>
>>>>
>>>>> can be None unchecked. I personally prefer to write
>>>>>
>>>>> cdef foo(obj=None):
>>>>> ? ? cdef int[:] a
>>>>> ? ? if obj is None:
>>>>> ? ? ? ? obj = ...
>>>>> ? ? a = obj
>>>>>
>>>>> Often you forget to write 'not None' when declaring the parameter (and
>>>>> apparently that it only allowed for 'def' functions).
>>>>>
>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>
>>>>
>>>>
>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>> that
>>>> setting stuff to None is rather vital in Python.
>>>>
>>>> What I typically do is more like this:
>>>>
>>>> def f(double[:] input, double[:] out=None):
>>>> ? ?if out is None:
>>>> ? ? ? ?out = np.empty_like(input)
>>>> ? ?...
>>>>
>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>> you
>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>> sometimes
>>>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>>
>>>
>>> No, it was just a contrived example.
>>>
>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>> should be implemented at some point, and then memoryviews would use the
>>>> same
>>>> mechanisms. Java has decent null-checking...
>>>>
>>>
>>> The problem with none checking is that it has to occur at every point.
>>
>>
>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>
>> for i in range(a.shape[0]):
>> ? ?print i
>> ? ?a[i] *= 3
>>
>> can be unrolled and none-checks inserted as
>>
>> print 0
>> if a is None: raise ....
>> a[0] *= 3
>> for i in range(1, a.shape[0]):
>> ? ?print i
>> ? ?a[i] *= 3 # no need for none-check
>>
>> It's very similar to what you'd want to do to pull boundschecking out of the
>> loop...
>>
>
> Oh, definitely. Both optimizations may not always be possible to do,
> though. The optimization (for boundschecking) is easier for prange()
> than range(), as you can immediately raise an exception as the
> exceptional condition may be issued at any iteration. ?What do you do
> with bounds checking when some accesses are in-bound, and some are
> out-of-bound? Do you immediately raise the exception? Are we fine with
> aborting (like Fortran compilers do when you ask them for bounds
> checking)? And how do you detect that the code doesn't already raise
> an exception or break out of the loop itself to prevent the
> out-of-bound access? (Unless no exceptions are propagating and no
> break/return is used, but exceptions are so very common).

I enabled bound checking in nogil contexts:
https://github.com/markflorisson88/cython/commit/73c6b0ea8e7e1c243e87b3966ade834b02664a4f
. It's not optimized yet, but at least it doesn't force users to use
boundscheck(False), it just hints that it would be faster to disable
the bounds checking.

When we actually start optimizing these things (e.g. moving it outside
loops etc), it might also be useful to consider inlining functions at
the Cython level (otherwise optimizations cannot escape the function).

>>> With initialized slices the control flow knows when the slices are
>>> initialized, or when they might not be (and it can raise a
>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>> I'm fine with implementing the behaviour, I just always left it at the
>>> bottom of my todo list.
>>
>>
>> Wasn't saying you should do it, just checking.
>>
>> I'm still not sure about this. I think what I'd really like is
>>
>> ?a) Stop cdef classes from being None as well
>>
>> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements that
>> help the type inferences:
>>
>> def f(arr):
>> ? ?if arr is None:
>> ? ? ? ?arr = ...
>> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
>> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
>> ? ?...
>>
>> or even:
>>
>> def f(arr):
>> ? ?if arr is None:
>> ? ? ? ?return 'foo'
>> ? ?else:
>> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
>> ? ? ? ?...
>> ? ?# arr still typed as int[:] here
>>
>> If we can make this work well enough with control flow analysis I'd never
>> cdef declare local vars again :-)
>
> Hm, what about the following?
>
> def f(arr):
> ? ?if arr is None:
> ? ? ? ?return 'foo'
>
> ? ?cdef int[:] arr # arr may not be None
>
>> Dag
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Sat Feb  4 19:49:26 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 4 Feb 2012 22:49:26 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
Message-ID: <CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>

2012/1/31 Robert Bradshaw <robertwb at math.washington.edu>:
> On Sat, Jan 28, 2012 at 8:05 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/26 Jason Grout <jason-sage at creativetrax.com>:
>>> On 1/25/12 11:39 AM, Robert Bradshaw wrote:
>>>>
>>>> install
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>> by downloading it and running "sage -i cython-devel.spkg"
>>>
>>>
>>>
>>> In fact, you could just do
>>>
>>> sage -i
>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>
>>> and Sage will (at least, should) download it for you, so that's even one
>>> less step!
>>>
>>> Jason
>>>
>>
>> Thanks for detailed instruction! I've successfully built it.
>>
>> "sage -t -gdb ./...." doesn't work, is that a bug?
>>
>> vitja at mchome:~/Downloads/sage-4.8$ ./sage ?-t -gdb
>> devel/sage/sage/combinat/sf/macdonald.py
>> sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
>> ********************************************************************************
>> Type r at the (gdb) prompt to run the doctests.
>> Type bt if there is a crash to see a traceback.
>> ********************************************************************************
>> gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>> starting cmd gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>> ImportError: No module named site
>> ? ? ? ? [0.2 s]
>>
>> ----------------------------------------------------------------------
>> The following tests failed:
>>
>>
>> ? ? ? ?sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"release
>> Total time for all tests: 0.2 seconds
>
> Yes, that's a bug.
>
>> I've found another way to run tests (using sage -sh and then direct
>> python ~/.sage/tmp/...py)
>>
>> So I found one of the problems. Here is minimal cython example:
>>
>> def foo(values):
>> ? ?return (0,)*len(values)
>> foo([1,2,3])
>>
>> len(values) somehow is passed as an integer to PyObject_Multiply()
>
> Yeah, that's a bug too :).

I've fixed tuple mult_factor bug here:

https://github.com/cython/cython/commit/2d4b85dbcef885fbdaf6a3b2daef7a017184a56f

No more segfaults in sage-tests but still 7 errors.

-- 
vitja.

From d.s.seljebotn at astro.uio.no  Sat Feb  4 20:39:29 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 04 Feb 2012 20:39:29 +0100
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EVyGLia1vG2hPy+F7nsnrR5WNGd+0egBftacNZTTaSjNg@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>	<4F2B0247.8040008@astro.uio.no>	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>	<4F2C1EFE.4050904@astro.uio.no>	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>	<4F2C242B.9010309@astro.uio.no>
	<CANg26EVyGLia1vG2hPy+F7nsnrR5WNGd+0egBftacNZTTaSjNg@mail.gmail.com>
Message-ID: <4F2D8971.1000102@astro.uio.no>

On 02/03/2012 07:26 PM, mark florisson wrote:
> On 3 February 2012 18:15, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 02/03/2012 07:07 PM, mark florisson wrote:
>>>
>>> On 3 February 2012 18:06, mark florisson<markflorisson88 at gmail.com>
>>>   wrote:
>>>>
>>>> On 3 February 2012 17:53, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no>    wrote:
>>>>>
>>>>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>>>>
>>>>>>
>>>>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>>>>> <d.s.seljebotn at astro.uio.no>      wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>>>>> <d.s.seljebotn at astro.uio.no>        wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I just realized that
>>>>>>>>>
>>>>>>>>> cdef int[:] a = None
>>>>>>>>>
>>>>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>>>>> "reference"
>>>>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b
>>>>>>>>> =
>>>>>>>>> None"
>>>>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not,
>>>>>>>>> and
>>>>>>>>> is
>>>>>>>>> it
>>>>>>>>> possible to do something about it?
>>>>>>>>>
>>>>>>>>> Dag Sverre
>>>>>>>>> _______________________________________________
>>>>>>>>> cython-devel mailing list
>>>>>>>>> cython-devel at python.org
>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>>>>>>> trying to get rid of all the segfaults and get the basic
>>>>>>>> functionality
>>>>>>>> working, so I disabled it. Personally, I have never liked how things
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Well, you can segfault quite easily with
>>>>>>>
>>>>>>> cdef MyClass a = None
>>>>>>> print a.field
>>>>>>>
>>>>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>>>>
>>>>>>>
>>>>>>>> can be None unchecked. I personally prefer to write
>>>>>>>>
>>>>>>>> cdef foo(obj=None):
>>>>>>>>      cdef int[:] a
>>>>>>>>      if obj is None:
>>>>>>>>          obj = ...
>>>>>>>>      a = obj
>>>>>>>>
>>>>>>>> Often you forget to write 'not None' when declaring the parameter
>>>>>>>> (and
>>>>>>>> apparently that it only allowed for 'def' functions).
>>>>>>>>
>>>>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>>>>> control flow with uninitialized slices, and will raise an error if it
>>>>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> When in doubt, go for consistency. So +1 for that reason. I do believe
>>>>>>> that
>>>>>>> setting stuff to None is rather vital in Python.
>>>>>>>
>>>>>>> What I typically do is more like this:
>>>>>>>
>>>>>>> def f(double[:] input, double[:] out=None):
>>>>>>>     if out is None:
>>>>>>>         out = np.empty_like(input)
>>>>>>>     ...
>>>>>>>
>>>>>>> Having to use another variable name is a bit of a pain. (Come on -- do
>>>>>>> you
>>>>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>>>>> sometimes
>>>>>>> end up with "out_" and so on, but it creates smelly code quite
>>>>>>> quickly.)
>>>>>>
>>>>>>
>>>>>>
>>>>>> No, it was just a contrived example.
>>>>>>
>>>>>>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>>>>>>> should be implemented at some point, and then memoryviews would use
>>>>>>> the
>>>>>>> same
>>>>>>> mechanisms. Java has decent null-checking...
>>>>>>>
>>>>>>
>>>>>> The problem with none checking is that it has to occur at every point.
>>>>>
>>>>>
>>>>>
>>>>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>>>>
>>>>> for i in range(a.shape[0]):
>>>>>     print i
>>>>>     a[i] *= 3
>>>>>
>>>>> can be unrolled and none-checks inserted as
>>>>>
>>>>> print 0
>>>>> if a is None: raise ....
>>>>> a[0] *= 3
>>>>> for i in range(1, a.shape[0]):
>>>>>     print i
>>>>>     a[i] *= 3 # no need for none-check
>>>>>
>>>>> It's very similar to what you'd want to do to pull boundschecking out of
>>>>> the
>>>>> loop...
>>>>>
>>>>
>>>> Oh, definitely. Both optimizations may not always be possible to do,
>>>> though. The optimization (for boundschecking) is easier for prange()
>>>> than range(), as you can immediately raise an exception as the
>>>> exceptional condition may be issued at any iteration.  What do you do
>>>> with bounds checking when some accesses are in-bound, and some are
>>>> out-of-bound? Do you immediately raise the exception? Are we fine with
>>>> aborting (like Fortran compilers do when you ask them for bounds
>>>> checking)? And how do you detect that the code doesn't already raise
>>>> an exception or break out of the loop itself to prevent the
>>>> out-of-bound access? (Unless no exceptions are propagating and no
>>>> break/return is used, but exceptions are so very common).
>>>>
>>>>>> With initialized slices the control flow knows when the slices are
>>>>>> initialized, or when they might not be (and it can raise a
>>>>>> compile-time or runtime error, instead of a segfault if you're lucky).
>>>>>> I'm fine with implementing the behaviour, I just always left it at the
>>>>>> bottom of my todo list.
>>>>>
>>>>>
>>>>>
>>>>> Wasn't saying you should do it, just checking.
>>>>>
>>>>> I'm still not sure about this. I think what I'd really like is
>>>>>
>>>>>   a) Stop cdef classes from being None as well
>>>>>
>>>>>   b) Sort-of deprecate cdef in favor of cast/assertion type statements
>>>>> that
>>>>> help the type inferences:
>>>>>
>>>>> def f(arr):
>>>>>     if arr is None:
>>>>>         arr = ...
>>>>>     arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>>>>>                       # acts as statement, with a specific point
>>>>>                       # for the none-check
>>>>>     ...
>>>>>
>>>>> or even:
>>>>>
>>>>> def f(arr):
>>>>>     if arr is None:
>>>>>         return 'foo'
>>>>>     else:
>>>>>         arr = int[:](arr) # takes effect *here*, does none-check
>>>>>         ...
>>>>>     # arr still typed as int[:] here
>>>>>
>>>>> If we can make this work well enough with control flow analysis I'd
>>>>> never
>>>>> cdef declare local vars again :-)
>>>>
>>>>
>>>> Hm, what about the following?
>>>>
>>>> def f(arr):
>>>>     if arr is None:
>>>>         return 'foo'
>>>>
>>>>     cdef int[:] arr # arr may not be None
>>>
>>>
>>> The above would work in general, until the declaration is lexically
>>> encountered, the object is typed as object.
>>
>>
>> This was actually going to be my first proposal :-) That would finally
>> define how "cdef" inside of if-statements etc. behave too (simply use
>> control flow analysis and treat it like a statement).
>
> Block-local declarations are definitely something we want, although I
> think it would require some more (non-trivial) changes to the
> compiler.

Note that my proposal was actually not about block-local declarations.

Block-local:

{
    int x = 4;
}
/* x not available here */

My idea was much more like hints to control flow analysis. That is, I 
wanted to have this raise an error:

x = 'adf'
if foo():
     cdef int x = y
print x # type of x not known

This is OK:

if foo():
     cdef int x = y
else:
     cdef int x = 4
print x # ok, type the same anyway -- so type "escapes" block

And I would allow

cdef str x = y
if foo:
     cdef int x = int(x)
     return g(x) # x must be int
print x # x must be str at this point


The reason for this madness is simply that control statements do NOT 
create blocks in Python, and making it so in Cython is just confusing. 
It would bring too much of C into the language for my taste.

I think that in my Cython-utopia, Symtab.py is only responsible for 
resolving the scope of *names*, and types of things are not bound to 
blocks, just to the state at control flow points.

Of course, implementing this would be a nightmare.

> Maybe the cleanup code from functions, as well as the temp handling
> etc could be re-factored to a BlockNode, that all block nodes could
> subclass. They'd have to instantiate new symbol table environments as
> well. I'm not yet entirely sure what else would be involved in the
> implementation of that.
>
>> But I like int[:] as a way of making it pure Python syntax compatible as
>> well. Perhaps the two are orthogonal -- a) make variable declaration a
>> statement, b) make cython.int[:](x) do, essentially, a cdef declaration, for
>> Python compatability.
>>
>
> Don't we have cython.declare() for that? e.g.
>
>      arr = cython.declare(cython.int[:])
>
> That would also be treated as a statement like normal declarations (if
> and when implemented).

This was what I said, but it wasn't what I meant. Sorry. I'll try to 
explain better:

1)  There's no way to have the above actually do the right thing in 
Python. With "arr = cython.int[:](arr)" one could actually return a 
NumPy or NumPy-like array that works in Python (since "arr" might not 
have the "shape" attribute before the conversion, all we know is that it 
exports the buffer interface...).

2) I don't like the fact that we overload the assignment operator to 
acquire a view. "cdef np.ndarray[int] x = y" is fine since if you do 
"x.someattr" then a NumPy subclass could provide someattr and it works 
fine. Acquiring a view is just something different.

3) Hence I guess I like "arr = int[:](arr)" better both for Cython and 
Python; at least if "arr" is always type-inferred to be int[:], even if 
arr was an "object" further up in the code (really, if you do "x = f(x)" 
at the top-level of the function, then x can just take the identity of 
another variable from that point on -- I don't know if the current 
control flow analysis and type inferences does this though?)

Dag Sverre

From stefan_ml at behnel.de  Sat Feb  4 21:32:45 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Feb 2012 21:32:45 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
Message-ID: <4F2D95ED.4050908@behnel.de>

Vitja Makarov, 04.02.2012 19:49:
>> On Sat, Jan 28, 2012 at 8:05 AM, Vitja Makarov wrote:
>>> So I found one of the problems. Here is minimal cython example:
>>>
>>> def foo(values):
>>>    return (0,)*len(values)
>>> foo([1,2,3])
>>>
>>> len(values) somehow is passed as an integer to PyObject_Multiply()
> 
> I've fixed tuple mult_factor bug here:
> 
> https://github.com/cython/cython/commit/2d4b85dbcef885fbdaf6a3b2daef7a017184a56f

I didn't have any time to look into this, but your fix seems right.

Thanks!

Stefan

From robertwb at math.washington.edu  Sun Feb  5 11:31:58 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 5 Feb 2012 02:31:58 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
Message-ID: <CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>

On Sat, Feb 4, 2012 at 10:49 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/1/31 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Sat, Jan 28, 2012 at 8:05 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/1/26 Jason Grout <jason-sage at creativetrax.com>:
>>>> On 1/25/12 11:39 AM, Robert Bradshaw wrote:
>>>>>
>>>>> install
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>>> by downloading it and running "sage -i cython-devel.spkg"
>>>>
>>>>
>>>>
>>>> In fact, you could just do
>>>>
>>>> sage -i
>>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>>
>>>> and Sage will (at least, should) download it for you, so that's even one
>>>> less step!
>>>>
>>>> Jason
>>>>
>>>
>>> Thanks for detailed instruction! I've successfully built it.
>>>
>>> "sage -t -gdb ./...." doesn't work, is that a bug?
>>>
>>> vitja at mchome:~/Downloads/sage-4.8$ ./sage ?-t -gdb
>>> devel/sage/sage/combinat/sf/macdonald.py
>>> sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
>>> ********************************************************************************
>>> Type r at the (gdb) prompt to run the doctests.
>>> Type bt if there is a crash to see a traceback.
>>> ********************************************************************************
>>> gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>>> starting cmd gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>>> ImportError: No module named site
>>> ? ? ? ? [0.2 s]
>>>
>>> ----------------------------------------------------------------------
>>> The following tests failed:
>>>
>>>
>>> ? ? ? ?sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"release
>>> Total time for all tests: 0.2 seconds
>>
>> Yes, that's a bug.
>>
>>> I've found another way to run tests (using sage -sh and then direct
>>> python ~/.sage/tmp/...py)
>>>
>>> So I found one of the problems. Here is minimal cython example:
>>>
>>> def foo(values):
>>> ? ?return (0,)*len(values)
>>> foo([1,2,3])
>>>
>>> len(values) somehow is passed as an integer to PyObject_Multiply()
>>
>> Yeah, that's a bug too :).
>
> I've fixed tuple mult_factor bug here:
>
> https://github.com/cython/cython/commit/2d4b85dbcef885fbdaf6a3b2daef7a017184a56f
>
> No more segfaults in sage-tests but still 7 errors.
>

Thanks! I've looked into the other errors and I think it boils down to
our use of --disable-function-redefinition being incompatible with how
decorators work. Of course that was just a hack, so I've fixed sage to
build/startup without using that flag, but there's some strangeness
with import order now that I haven't had time to resolve yet.

From markflorisson88 at gmail.com  Sun Feb  5 22:56:18 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 5 Feb 2012 21:56:18 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2D8971.1000102@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
	<4F2C242B.9010309@astro.uio.no>
	<CANg26EVyGLia1vG2hPy+F7nsnrR5WNGd+0egBftacNZTTaSjNg@mail.gmail.com>
	<4F2D8971.1000102@astro.uio.no>
Message-ID: <CANg26EVicBKTT31eeBtNsM6GZ6NdKbb42W+xf4nfWZ9Ez4aU1Q@mail.gmail.com>

On 4 February 2012 19:39, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/03/2012 07:26 PM, mark florisson wrote:
>>
>> On 3 February 2012 18:15, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> On 02/03/2012 07:07 PM, mark florisson wrote:
>>>>
>>>>
>>>> On 3 February 2012 18:06, mark florisson<markflorisson88 at gmail.com>
>>>> ?wrote:
>>>>>
>>>>>
>>>>> On 3 February 2012 17:53, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> ? ?wrote:
>>>>>>
>>>>>>
>>>>>> On 02/03/2012 12:09 AM, mark florisson wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2 February 2012 21:38, Dag Sverre Seljebotn
>>>>>>> <d.s.seljebotn at astro.uio.no> ? ? ?wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>>>>>>>> <d.s.seljebotn at astro.uio.no> ? ? ? ?wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I just realized that
>>>>>>>>>>
>>>>>>>>>> cdef int[:] a = None
>>>>>>>>>>
>>>>>>>>>> raises an exception; even though I'd argue that 'a' is of the
>>>>>>>>>> "reference"
>>>>>>>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass
>>>>>>>>>> b
>>>>>>>>>> =
>>>>>>>>>> None"
>>>>>>>>>> is allowed even if type(None) is NoneType). Is this a bug or not,
>>>>>>>>>> and
>>>>>>>>>> is
>>>>>>>>>> it
>>>>>>>>>> possible to do something about it?
>>>>>>>>>>
>>>>>>>>>> Dag Sverre
>>>>>>>>>> _______________________________________________
>>>>>>>>>> cython-devel mailing list
>>>>>>>>>> cython-devel at python.org
>>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yeah I disabled that quite early. It was supposed to be working but
>>>>>>>>> gave a lot of trouble in cases (segfaults, mainly). At the time I
>>>>>>>>> was
>>>>>>>>> trying to get rid of all the segfaults and get the basic
>>>>>>>>> functionality
>>>>>>>>> working, so I disabled it. Personally, I have never liked how
>>>>>>>>> things
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Well, you can segfault quite easily with
>>>>>>>>
>>>>>>>> cdef MyClass a = None
>>>>>>>> print a.field
>>>>>>>>
>>>>>>>> so it doesn't make sense to slices different from cdef classes IMO.
>>>>>>>>
>>>>>>>>
>>>>>>>>> can be None unchecked. I personally prefer to write
>>>>>>>>>
>>>>>>>>> cdef foo(obj=None):
>>>>>>>>> ? ? cdef int[:] a
>>>>>>>>> ? ? if obj is None:
>>>>>>>>> ? ? ? ? obj = ...
>>>>>>>>> ? ? a = obj
>>>>>>>>>
>>>>>>>>> Often you forget to write 'not None' when declaring the parameter
>>>>>>>>> (and
>>>>>>>>> apparently that it only allowed for 'def' functions).
>>>>>>>>>
>>>>>>>>> As such, I never bothered to re-enable it. However, it does support
>>>>>>>>> control flow with uninitialized slices, and will raise an error if
>>>>>>>>> it
>>>>>>>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When in doubt, go for consistency. So +1 for that reason. I do
>>>>>>>> believe
>>>>>>>> that
>>>>>>>> setting stuff to None is rather vital in Python.
>>>>>>>>
>>>>>>>> What I typically do is more like this:
>>>>>>>>
>>>>>>>> def f(double[:] input, double[:] out=None):
>>>>>>>> ? ?if out is None:
>>>>>>>> ? ? ? ?out = np.empty_like(input)
>>>>>>>> ? ?...
>>>>>>>>
>>>>>>>> Having to use another variable name is a bit of a pain. (Come on --
>>>>>>>> do
>>>>>>>> you
>>>>>>>> use "a" in real code? What do you actually call "the other obj"? I
>>>>>>>> sometimes
>>>>>>>> end up with "out_" and so on, but it creates smelly code quite
>>>>>>>> quickly.)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> No, it was just a contrived example.
>>>>>>>
>>>>>>>> It's easy to segfault with cdef classes anyway, so decent
>>>>>>>> nonechecking
>>>>>>>> should be implemented at some point, and then memoryviews would use
>>>>>>>> the
>>>>>>>> same
>>>>>>>> mechanisms. Java has decent null-checking...
>>>>>>>>
>>>>>>>
>>>>>>> The problem with none checking is that it has to occur at every
>>>>>>> point.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Well, using control flow analysis etc. it doesn't really. E.g.,
>>>>>>
>>>>>> for i in range(a.shape[0]):
>>>>>> ? ?print i
>>>>>> ? ?a[i] *= 3
>>>>>>
>>>>>> can be unrolled and none-checks inserted as
>>>>>>
>>>>>> print 0
>>>>>> if a is None: raise ....
>>>>>> a[0] *= 3
>>>>>> for i in range(1, a.shape[0]):
>>>>>> ? ?print i
>>>>>> ? ?a[i] *= 3 # no need for none-check
>>>>>>
>>>>>> It's very similar to what you'd want to do to pull boundschecking out
>>>>>> of
>>>>>> the
>>>>>> loop...
>>>>>>
>>>>>
>>>>> Oh, definitely. Both optimizations may not always be possible to do,
>>>>> though. The optimization (for boundschecking) is easier for prange()
>>>>> than range(), as you can immediately raise an exception as the
>>>>> exceptional condition may be issued at any iteration. ?What do you do
>>>>> with bounds checking when some accesses are in-bound, and some are
>>>>> out-of-bound? Do you immediately raise the exception? Are we fine with
>>>>> aborting (like Fortran compilers do when you ask them for bounds
>>>>> checking)? And how do you detect that the code doesn't already raise
>>>>> an exception or break out of the loop itself to prevent the
>>>>> out-of-bound access? (Unless no exceptions are propagating and no
>>>>> break/return is used, but exceptions are so very common).
>>>>>
>>>>>>> With initialized slices the control flow knows when the slices are
>>>>>>> initialized, or when they might not be (and it can raise a
>>>>>>> compile-time or runtime error, instead of a segfault if you're
>>>>>>> lucky).
>>>>>>> I'm fine with implementing the behaviour, I just always left it at
>>>>>>> the
>>>>>>> bottom of my todo list.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Wasn't saying you should do it, just checking.
>>>>>>
>>>>>> I'm still not sure about this. I think what I'd really like is
>>>>>>
>>>>>> ?a) Stop cdef classes from being None as well
>>>>>>
>>>>>> ?b) Sort-of deprecate cdef in favor of cast/assertion type statements
>>>>>> that
>>>>>> help the type inferences:
>>>>>>
>>>>>> def f(arr):
>>>>>> ? ?if arr is None:
>>>>>> ? ? ? ?arr = ...
>>>>>> ? ?arr = int[:](arr) # equivalent to "cdef int[:] arr = arr", but
>>>>>> ? ? ? ? ? ? ? ? ? ? ?# acts as statement, with a specific point
>>>>>> ? ? ? ? ? ? ? ? ? ? ?# for the none-check
>>>>>> ? ?...
>>>>>>
>>>>>> or even:
>>>>>>
>>>>>> def f(arr):
>>>>>> ? ?if arr is None:
>>>>>> ? ? ? ?return 'foo'
>>>>>> ? ?else:
>>>>>> ? ? ? ?arr = int[:](arr) # takes effect *here*, does none-check
>>>>>> ? ? ? ?...
>>>>>> ? ?# arr still typed as int[:] here
>>>>>>
>>>>>> If we can make this work well enough with control flow analysis I'd
>>>>>> never
>>>>>> cdef declare local vars again :-)
>>>>>
>>>>>
>>>>>
>>>>> Hm, what about the following?
>>>>>
>>>>> def f(arr):
>>>>> ? ?if arr is None:
>>>>> ? ? ? ?return 'foo'
>>>>>
>>>>> ? ?cdef int[:] arr # arr may not be None
>>>>
>>>>
>>>>
>>>> The above would work in general, until the declaration is lexically
>>>> encountered, the object is typed as object.
>>>
>>>
>>>
>>> This was actually going to be my first proposal :-) That would finally
>>> define how "cdef" inside of if-statements etc. behave too (simply use
>>> control flow analysis and treat it like a statement).
>>
>>
>> Block-local declarations are definitely something we want, although I
>> think it would require some more (non-trivial) changes to the
>> compiler.
>
>
> Note that my proposal was actually not about block-local declarations.
>
> Block-local:
>
> {
> ? int x = 4;
> }
> /* x not available here */
>
> My idea was much more like hints to control flow analysis. That is, I wanted
> to have this raise an error:
>
> x = 'adf'
> if foo():
> ? ?cdef int x = y
> print x # type of x not known
>
> This is OK:
>
> if foo():
> ? ?cdef int x = y
> else:
> ? ?cdef int x = 4
> print x # ok, type the same anyway -- so type "escapes" block

Seeing that it doesn't work that way in any language with block
scopes, I find that pretty surprising behaviour. Why would you not
simply mandate that the user declares 'x' outside of the blocks?

> And I would allow
>
> cdef str x = y
> if foo:
> ? ?cdef int x = int(x)
> ? ?return g(x) # x must be int
> print x # x must be str at this point
>
>
> The reason for this madness is simply that control statements do NOT create
> blocks in Python, and making it so in Cython is just confusing. It would
> bring too much of C into the language for my taste.

And yet it can be very useful and intuitive in several contexts, just
not for objects (which aren't typed anyway!). Block-local declarations
are useful when a variable is only used in the block and it can be
useful to make variables private in the cython.parallel context
("assignment makes private" is really not as intuitive). It's not a
very important feature though, and it's indeed more a thing from
static languages than Python.

> I think that in my Cython-utopia, Symtab.py is only responsible for
> resolving the scope of *names*, and types of things are not bound to blocks,
> just to the state at control flow points.
>
> Of course, implementing this would be a nightmare.
>
>
>> Maybe the cleanup code from functions, as well as the temp handling
>> etc could be re-factored to a BlockNode, that all block nodes could
>> subclass. They'd have to instantiate new symbol table environments as
>> well. I'm not yet entirely sure what else would be involved in the
>> implementation of that.
>>
>>> But I like int[:] as a way of making it pure Python syntax compatible as
>>> well. Perhaps the two are orthogonal -- a) make variable declaration a
>>> statement, b) make cython.int[:](x) do, essentially, a cdef declaration,
>>> for
>>> Python compatability.
>>>
>>
>> Don't we have cython.declare() for that? e.g.
>>
>> ? ? arr = cython.declare(cython.int[:])
>>
>> That would also be treated as a statement like normal declarations (if
>> and when implemented).
>
>
> This was what I said, but it wasn't what I meant. Sorry. I'll try to explain
> better:
>
> 1) ?There's no way to have the above actually do the right thing in Python.
> With "arr = cython.int[:](arr)" one could actually return a NumPy or
> NumPy-like array that works in Python (since "arr" might not have the
> "shape" attribute before the conversion, all we know is that it exports the
> buffer interface...).

Right, but the same thing goes for other types as well. E.g. I can
type something int with cython.declare() and then use strings instead.

> 2) I don't like the fact that we overload the assignment operator to acquire
> a view. "cdef np.ndarray[int] x = y" is fine since if you do "x.someattr"
> then a NumPy subclass could provide someattr and it works fine. Acquiring a
> view is just something different.

Yeah it's kind of overloaded, but in a good way :) It's the language
that does the overloading, which means it's not very surprising. And
the memoryview slices coerce to numpy-like (although somewhat
incapable) objects and support some of their attributes. I like the
simplicity of assignment here, you don't really care that it takes a
view, you just want to access and operate on the data.

What do you think of allowing the user to register a
conversion-to-object function? And perhaps the default should be that
if a view was never sliced, it just returns the original object
(although that might mean you get back objects with incompatible
interfaces...).

> 3) Hence I guess I like "arr = int[:](arr)" better both for Cython and
> Python; at least if "arr" is always type-inferred to be int[:], even if arr
> was an "object" further up in the code (really, if you do "x = f(x)" at the
> top-level of the function, then x can just take the identity of another
> variable from that point on -- I don't know if the current control flow
> analysis and type inferences does this though?)
>
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sun Feb  5 22:57:51 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 5 Feb 2012 21:57:51 +0000
Subject: [Cython] OpenCL support
Message-ID: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>

Hey,

I created a CEP for opencl support: http://wiki.cython.org/enhancements/opencl
What do you think?

Mark

From markflorisson88 at gmail.com  Sun Feb  5 23:03:36 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 5 Feb 2012 22:03:36 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2B0247.8040008@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
Message-ID: <CANg26EWK91FRgcmV+JZ5x77KPJs2xcB+ffPLHMrF_x_nuF7Ykg@mail.gmail.com>

On 2 February 2012 21:38, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/02/2012 10:16 PM, mark florisson wrote:
>>
>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>
>>> I just realized that
>>>
>>> cdef int[:] a = None
>>>
>>> raises an exception; even though I'd argue that 'a' is of the "reference"
>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>> None"
>>> is allowed even if type(None) is NoneType). Is this a bug or not, and is
>>> it
>>> possible to do something about it?
>>>
>>> Dag Sverre
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>> Yeah I disabled that quite early. It was supposed to be working but
>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>> trying to get rid of all the segfaults and get the basic functionality
>> working, so I disabled it. Personally, I have never liked how things
>
>
> Well, you can segfault quite easily with
>
> cdef MyClass a = None
> print a.field
>
> so it doesn't make sense to slices different from cdef classes IMO.
>
>
>> can be None unchecked. I personally prefer to write
>>
>> cdef foo(obj=None):
>> ? ? cdef int[:] a
>> ? ? if obj is None:
>> ? ? ? ? obj = ...
>> ? ? a = obj
>>
>> Often you forget to write 'not None' when declaring the parameter (and
>> apparently that it only allowed for 'def' functions).
>>
>> As such, I never bothered to re-enable it. However, it does support
>> control flow with uninitialized slices, and will raise an error if it
>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>
>
> When in doubt, go for consistency. So +1 for that reason. I do believe that
> setting stuff to None is rather vital in Python.

Yeah I think we should go back to this discussion :) Checking for None
and allowing slices to be None is simply  very convenient, and doesn't
involve any drastic changes. I was never really against it, I just
never got around to implementing it.

> What I typically do is more like this:
>
> def f(double[:] input, double[:] out=None):
> ? ?if out is None:
> ? ? ? ?out = np.empty_like(input)
> ? ?...
>
> Having to use another variable name is a bit of a pain. (Come on -- do you
> use "a" in real code? What do you actually call "the other obj"? I sometimes
> end up with "out_" and so on, but it creates smelly code quite quickly.)
>
> It's easy to segfault with cdef classes anyway, so decent nonechecking
> should be implemented at some point, and then memoryviews would use the same
> mechanisms. Java has decent null-checking...
>
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From dtcaciuc at gmail.com  Sun Feb  5 23:39:25 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Sun, 5 Feb 2012 14:39:25 -0800
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
Message-ID: <CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>

Mark,

Couple of thoughts based on some experience with OpenCL...

1. This may be going outside the proposed purpose, but some algorithms
such as molecular simulations can benefit from a fairly large amount
of constant data loaded at the beginning of the program and persisted
in between invocations of a function. If I understand the proposal,
entire program would need to be within one `with` block, which would
certainly be limiting to the architecture. Eg.

? ? # run.py
? ? from cython_module import Evaluator

? ? # Arrays are loaded into device memory here
? ? x = Evaluator(params...)
? ? for i in range(N):
? ? ? ? # Calculations are performed with
? ? ? ? # mostly data in the device memory
? ? ? ? data_i = x.step()
? ? ? ? ...

2. AFAIK, given a device, OpenCL basically takes it over (which would
be eg. 8 cores on 2 CPU x 4 cores machine), so I'm not sure how
`num_cores` parameter would work here. There's the fission extension
that allows you to selectively run on a portion of the device, but the
idea is that you're still dedicating entire device to your process,
but merely giving more organization to your processing tasks, where
you have to specify the core numbers you want to use. I may very well
be wrong here, bashing is welcome :)

3. Does it make sense to make OpenCL more explicit? Heuristics and
automatic switching between, say, CPU and GPU is great for eg. Sage
users, but maybe not so much if you know exactly what you're doing
with your machine resources. E.g just having a library with thin
cython-adapted wrappers would be awesome. I imagine this can be
augmented by arrays having a knowledge of device-side/client-side
(which would go towards addressing the issue 1. above)

Cheers,


Dimitri.

On Sun, Feb 5, 2012 at 1:57 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Hey,
>
> I created a CEP for opencl support: http://wiki.cython.org/enhancements/opencl
> What do you think?
>
> Mark
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Mon Feb  6 00:12:47 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 5 Feb 2012 23:12:47 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
Message-ID: <CANg26EVevHhQpirtet0E2bev+Lr+B37o_fsOw9eA8xm=YWoBPw@mail.gmail.com>

On 5 February 2012 22:39, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> Mark,
>
> Couple of thoughts based on some experience with OpenCL...
>
> 1. This may be going outside the proposed purpose, but some algorithms
> such as molecular simulations can benefit from a fairly large amount
> of constant data loaded at the beginning of the program and persisted
> in between invocations of a function. If I understand the proposal,
> entire program would need to be within one `with` block, which would
> certainly be limiting to the architecture. Eg.
>
> ? ? # run.py
> ? ? from cython_module import Evaluator
>
> ? ? # Arrays are loaded into device memory here
> ? ? x = Evaluator(params...)
> ? ? for i in range(N):
> ? ? ? ? # Calculations are performed with
> ? ? ? ? # mostly data in the device memory
> ? ? ? ? data_i = x.step()
> ? ? ? ? ...

The point of the proposal is that the slices will actually stay on the
GPU as long as possible, until they absolutely need to be copied back
(e.g. when you go back to NumPy-land). You can do anything you want
in-between (outside any parallel section), e.g. call other functions
that run on the CPU, call python functions, whatever. When you
continue processing data that is still on the GPU, it will simply
continue from there.

But point taken, the compiler could think "oh but this is not too much
work, and I have more data in main memory than on the GPU, so let me
use the CPU and copy that constant data back". So perhaps the pinning
should not just work for main memory, stuff could also be pinned on
the GPU. Then if there is a "pinning conflict" Cython would raise an
exception.

> 2. AFAIK, given a device, OpenCL basically takes it over (which would
> be eg. 8 cores on 2 CPU x 4 cores machine), so I'm not sure how
> `num_cores` parameter would work here. There's the fission extension
> that allows you to selectively run on a portion of the device, but the
> idea is that you're still dedicating entire device to your process,
> but merely giving more organization to your processing tasks, where
> you have to specify the core numbers you want to use. I may very well
> be wrong here, bashing is welcome :)

Oh, yes. I think the num_threads clause could simply be ignored in
that context, it's only supposed to be an upper limit. Scheduling
hints like chunksize could also be ignored :)

> 3. Does it make sense to make OpenCL more explicit? Heuristics and
> automatic switching between, say, CPU and GPU is great for eg. Sage
> users, but maybe not so much if you know exactly what you're doing
> with your machine resources. E.g just having a library with thin
> cython-adapted wrappers would be awesome. I imagine this can be
> augmented by arrays having a knowledge of device-side/client-side
> (which would go towards addressing the issue 1. above)

Hm, there are several advantages to supporting this in the language.
One is that you can support parallel sections, and that your code can
transparently execute in parallel on whatever device the compiler and
runtime think will be best. Excluding the cython.parallel stuff I
don't think there is enough room for a library, you might as well use
pyopencl directly in that case right?
Not OpenCL perse, but part of that will also solve the numpy-temporary
problem, which we have numexpr for. But it would be more convenient to
express oneself natively in the programming language of choice (Cython
:).

> Cheers,
>
>
> Dimitri.
>
> On Sun, Feb 5, 2012 at 1:57 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> Hey,
>>
>> I created a CEP for opencl support: http://wiki.cython.org/enhancements/opencl
>> What do you think?
>>
>> Mark
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Mon Feb  6 08:22:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 06 Feb 2012 08:22:26 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EVevHhQpirtet0E2bev+Lr+B37o_fsOw9eA8xm=YWoBPw@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<CANg26EVevHhQpirtet0E2bev+Lr+B37o_fsOw9eA8xm=YWoBPw@mail.gmail.com>
Message-ID: <4F2F7FB2.5020604@behnel.de>

mark florisson, 06.02.2012 00:12:
> On 5 February 2012 22:39, Dimitri Tcaciuc wrote:
>> 3. Does it make sense to make OpenCL more explicit? Heuristics and
>> automatic switching between, say, CPU and GPU is great for eg. Sage
>> users, but maybe not so much if you know exactly what you're doing
>> with your machine resources. E.g just having a library with thin
>> cython-adapted wrappers would be awesome. I imagine this can be
>> augmented by arrays having a knowledge of device-side/client-side
>> (which would go towards addressing the issue 1. above)
> 
> Hm, there are several advantages to supporting this in the language.

... and there's always the obvious disadvantage of making the language too
complex and magic to learn and understand. Worth balancing.

Stefan

From markflorisson88 at gmail.com  Mon Feb  6 11:21:53 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 6 Feb 2012 10:21:53 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <4F2F7FB2.5020604@behnel.de>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<CANg26EVevHhQpirtet0E2bev+Lr+B37o_fsOw9eA8xm=YWoBPw@mail.gmail.com>
	<4F2F7FB2.5020604@behnel.de>
Message-ID: <CANg26EXVpwa-DeVX2FToVHe1dFVAhgBwiJ7kZGmgtTDjc8UGKQ@mail.gmail.com>

On 6 February 2012 07:22, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 06.02.2012 00:12:
>> On 5 February 2012 22:39, Dimitri Tcaciuc wrote:
>>> 3. Does it make sense to make OpenCL more explicit? Heuristics and
>>> automatic switching between, say, CPU and GPU is great for eg. Sage
>>> users, but maybe not so much if you know exactly what you're doing
>>> with your machine resources. E.g just having a library with thin
>>> cython-adapted wrappers would be awesome. I imagine this can be
>>> augmented by arrays having a knowledge of device-side/client-side
>>> (which would go towards addressing the issue 1. above)
>>
>> Hm, there are several advantages to supporting this in the language.
>
> ... and there's always the obvious disadvantage of making the language too
> complex and magic to learn and understand. Worth balancing.

Definitely. This would however introduce very minor changes to the
language (no new syntax at least, just a few memoryview methods), but
more major changes to the compiler. The support would mostly be
transparent.
Clyther (http://srossross.github.com/Clyther/) is a related project,
which does a similar thing by compiling python (bytecode) to opencl.
What I want for Cython is something even more transparent, the user
wouldn't perhaps even know opencl was involved, and the compiler has
more control over how data is handled.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From dtcaciuc at gmail.com  Mon Feb  6 18:23:19 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Mon, 6 Feb 2012 09:23:19 -0800
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EXVpwa-DeVX2FToVHe1dFVAhgBwiJ7kZGmgtTDjc8UGKQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<CANg26EVevHhQpirtet0E2bev+Lr+B37o_fsOw9eA8xm=YWoBPw@mail.gmail.com>
	<4F2F7FB2.5020604@behnel.de>
	<CANg26EXVpwa-DeVX2FToVHe1dFVAhgBwiJ7kZGmgtTDjc8UGKQ@mail.gmail.com>
Message-ID: <CAL41sfSCgjZKiqazY7Z9Ds3QXjHmVQC29pR35QAi-XzJL+3fyQ@mail.gmail.com>

On Mon, Feb 6, 2012 at 2:21 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 6 February 2012 07:22, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 06.02.2012 00:12:
>>> On 5 February 2012 22:39, Dimitri Tcaciuc wrote:
>>>> 3. Does it make sense to make OpenCL more explicit? Heuristics and
>>>> automatic switching between, say, CPU and GPU is great for eg. Sage
>>>> users, but maybe not so much if you know exactly what you're doing
>>>> with your machine resources. E.g just having a library with thin
>>>> cython-adapted wrappers would be awesome. I imagine this can be
>>>> augmented by arrays having a knowledge of device-side/client-side
>>>> (which would go towards addressing the issue 1. above)
>>>
>>> Hm, there are several advantages to supporting this in the language.
>>
>> ... and there's always the obvious disadvantage of making the language too
>> complex and magic to learn and understand. Worth balancing.
>
> Definitely. This would however introduce very minor changes to the
> language (no new syntax at least, just a few memoryview methods), but
> more major changes to the compiler. The support would mostly be
> transparent.
> Clyther (http://srossross.github.com/Clyther/) is a related project,
> which does a similar thing by compiling python (bytecode) to opencl.
> What I want for Cython is something even more transparent, the user
> wouldn't perhaps even know opencl was involved, and the compiler has
> more control over how data is handled.

What I'm absolutely certain of is that sort of complete transparency
will eventually start getting edge cases and from there on additional
development and design will have to be made, so I it's better to plan
not-as-transparent elements and user-side control right form the
start.

I think another reason I would go for a less automatic solution is
because I imagine the alternative would inevitably complicate Cython
internals. I think keeping that as simple as possible is huge
advantage in the long run, which is arguably as important as reducing
amount of code a language user has to write (hence me initially
suggesting a more library-like Cython integration, although pyopencl
did work quite well already :).

Dimitri.

>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From sturla at molden.no  Tue Feb  7 14:52:05 2012
From: sturla at molden.no (Sturla Molden)
Date: Tue, 07 Feb 2012 14:52:05 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
Message-ID: <4F312C85.7050805@molden.no>

On 05.02.2012 23:39, Dimitri Tcaciuc wrote:

> 3. Does it make sense to make OpenCL more explicit?

No, it takes the usefuness of OpenCL away, which is that kernels are 
text strings and compiled at run-time.

> Heuristics and
> automatic switching between, say, CPU and GPU is great for eg. Sage
> users, but maybe not so much if you know exactly what you're doing
> with your machine resources. E.g just having a library with thin
> cython-adapted wrappers would be awesome. I imagine this can be
> augmented by arrays having a knowledge of device-side/client-side
> (which would go towards addressing the issue 1. above)

Just use PyOpenCL and manipulate kernels as text. Python is excellent 
for that - Cython is not needed. If you think using Cython instead of 
Python (PyOpenCL and NumPy) will be important, you don't have a CPU 
bound problem that warrants the use of OpenCL.

Sturla




From dtcaciuc at gmail.com  Tue Feb  7 18:22:59 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Tue, 7 Feb 2012 09:22:59 -0800
Subject: [Cython] OpenCL support
In-Reply-To: <4F312C85.7050805@molden.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
Message-ID: <CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>

On Tue, Feb 7, 2012 at 5:52 AM, Sturla Molden <sturla at molden.no> wrote:
> On 05.02.2012 23:39, Dimitri Tcaciuc wrote:
>
>> 3. Does it make sense to make OpenCL more explicit?
>
>
> No, it takes the usefuness of OpenCL away, which is that kernels are text
> strings and compiled at run-time.

I'm not sure I understand you, maybe you could elaborate on that? By
"explicit" I merely meant that the user will explicitly specify that
they're working on OpenCL-enabled array or certain bit of Cython code
will get compiled into OpenCL program etc.

>
>> Heuristics and
>> automatic switching between, say, CPU and GPU is great for eg. Sage
>> users, but maybe not so much if you know exactly what you're doing
>> with your machine resources. E.g just having a library with thin
>> cython-adapted wrappers would be awesome. I imagine this can be
>> augmented by arrays having a knowledge of device-side/client-side
>> (which would go towards addressing the issue 1. above)
>
>
> Just use PyOpenCL and manipulate kernels as text. Python is excellent for
> that - Cython is not needed. If you think using Cython instead of Python
> (PyOpenCL and NumPy) will be important, you don't have a CPU bound problem
> that warrants the use of OpenCL.

Again, not sure what you mean here. As I mentioned in the thread,
PyOpenCL worked quite fine, however if Cython is getting OpenCL
support, I'd much rather use that than keeping a dependency on another
library.

> Sturla
>
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb  7 18:58:20 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 7 Feb 2012 17:58:20 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <4F312C85.7050805@molden.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
Message-ID: <CANg26EWq-jFySSTB+UPnAsBrDY1wLEqZchBVA283osdTwvZNEg@mail.gmail.com>

On 7 February 2012 13:52, Sturla Molden <sturla at molden.no> wrote:
> On 05.02.2012 23:39, Dimitri Tcaciuc wrote:
>
>> 3. Does it make sense to make OpenCL more explicit?
>
>
> No, it takes the usefuness of OpenCL away, which is that kernels are text
> strings and compiled at run-time.
>

I don't know why you think that is necessary. Obviously Cython's
translated opencl would also be compiled at runtime (or loaded from a
cache). If you mean you can't do string interpolation, I don't see why
you would need that.

>> Heuristics and
>> automatic switching between, say, CPU and GPU is great for eg. Sage
>> users, but maybe not so much if you know exactly what you're doing
>> with your machine resources. E.g just having a library with thin
>> cython-adapted wrappers would be awesome. I imagine this can be
>> augmented by arrays having a knowledge of device-side/client-side
>> (which would go towards addressing the issue 1. above)
>
>
> Just use PyOpenCL and manipulate kernels as text. Python is excellent for
> that - Cython is not needed. If you think using Cython instead of Python
> (PyOpenCL and NumPy) will be important, you don't have a CPU bound problem
> that warrants the use of OpenCL.
>
> Sturla
>

That is not very constructive input. If you use PyOpenCL you have to
basically rethink and rewrite your kernels just for OpenCL. That is
far from trivial and there is not much pyopencl does to keep you away
from the pain of OpenCL and general GPU computing.

There are existing approaches (compiler directives for C or Fortran)
to do similar things, and an OpenMP (sub-)committee is working on
adding/defining such features to the standard. Why? Because GPU
programming is still a major pain in the ass. And although automatic
translation will probably not yield the best performance for your
particular hardware as a handwritten version, it will have saved you
hours of coding and possibly rewriting.

>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From sturla at molden.no  Tue Feb  7 18:58:41 2012
From: sturla at molden.no (Sturla Molden)
Date: Tue, 07 Feb 2012 18:58:41 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>	<4F312C85.7050805@molden.no>
	<CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
Message-ID: <4F316651.1040504@molden.no>

On 07.02.2012 18:22, Dimitri Tcaciuc wrote:

> I'm not sure I understand you, maybe you could elaborate on that?

OpenCL code is a text string that is compiled when the program runs. So 
it can be generated from run-time data. Think of it like dynamic HTML.

> Again, not sure what you mean here. As I mentioned in the thread,
> PyOpenCL worked quite fine, however if Cython is getting OpenCL
> support, I'd much rather use that than keeping a dependency on another
> library.

You can use PyOpenCL or OpenCL C or C++ headers with Cython. The latter 
you just use as you would with any other C or C++ library. You don't 
need to change the compiler to use a library: It seems like you think 
OpenCL is compiled from code when you build the program. It is actually 
compiled from text strings when you run the program. It is meaningless 
to ask if Cython supports OpenCL because Cython supports any C library.

Sturla



From markflorisson88 at gmail.com  Tue Feb  7 19:01:41 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 7 Feb 2012 18:01:41 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
	<CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
Message-ID: <CANg26EXo-eE_s3WWh+Tr=Ahba4OOejrU=5EGAy5d3N6O27Y=YA@mail.gmail.com>

On 7 February 2012 17:22, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> On Tue, Feb 7, 2012 at 5:52 AM, Sturla Molden <sturla at molden.no> wrote:
>> On 05.02.2012 23:39, Dimitri Tcaciuc wrote:
>>
>>> 3. Does it make sense to make OpenCL more explicit?
>>
>>
>> No, it takes the usefuness of OpenCL away, which is that kernels are text
>> strings and compiled at run-time.
>
> I'm not sure I understand you, maybe you could elaborate on that? By
> "explicit" I merely meant that the user will explicitly specify that
> they're working on OpenCL-enabled array or certain bit of Cython code
> will get compiled into OpenCL program etc.

I gave that some thought as well, like 'cdef double[::view.gpu, :]
myarray', which would mean that the data is on the gpu. Generally
though, I think it kind of defeats the purpose. E.g. if you have small
arrays you probably don't want anything to be on the gpu, whereas if
you have larger ones and sufficient computation operating on them, it
might be worthwhile. The point is, as a user you don't care, you want
your runtime to make a sensible decision. If you don't want anything
to do with OpenCL, you can disable it, or if you want to only ever
stay on the CPU, you could "pin" it there.

>>
>>> Heuristics and
>>> automatic switching between, say, CPU and GPU is great for eg. Sage
>>> users, but maybe not so much if you know exactly what you're doing
>>> with your machine resources. E.g just having a library with thin
>>> cython-adapted wrappers would be awesome. I imagine this can be
>>> augmented by arrays having a knowledge of device-side/client-side
>>> (which would go towards addressing the issue 1. above)
>>
>>
>> Just use PyOpenCL and manipulate kernels as text. Python is excellent for
>> that - Cython is not needed. If you think using Cython instead of Python
>> (PyOpenCL and NumPy) will be important, you don't have a CPU bound problem
>> that warrants the use of OpenCL.
>
> Again, not sure what you mean here. As I mentioned in the thread,
> PyOpenCL worked quite fine, however if Cython is getting OpenCL
> support, I'd much rather use that than keeping a dependency on another
> library.
>
>> Sturla
>>
>>
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb  7 19:03:02 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 7 Feb 2012 18:03:02 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EXo-eE_s3WWh+Tr=Ahba4OOejrU=5EGAy5d3N6O27Y=YA@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
	<CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
	<CANg26EXo-eE_s3WWh+Tr=Ahba4OOejrU=5EGAy5d3N6O27Y=YA@mail.gmail.com>
Message-ID: <CANg26EXkTsmUm8Ze7=Bm1smMNMr2+_o+xYDer838CVbOEFfvGQ@mail.gmail.com>

On 7 February 2012 18:01, mark florisson <markflorisson88 at gmail.com> wrote:
> On 7 February 2012 17:22, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
>> On Tue, Feb 7, 2012 at 5:52 AM, Sturla Molden <sturla at molden.no> wrote:
>>> On 05.02.2012 23:39, Dimitri Tcaciuc wrote:
>>>
>>>> 3. Does it make sense to make OpenCL more explicit?
>>>
>>>
>>> No, it takes the usefuness of OpenCL away, which is that kernels are text
>>> strings and compiled at run-time.
>>
>> I'm not sure I understand you, maybe you could elaborate on that? By
>> "explicit" I merely meant that the user will explicitly specify that
>> they're working on OpenCL-enabled array or certain bit of Cython code
>> will get compiled into OpenCL program etc.
>
> I gave that some thought as well, like 'cdef double[::view.gpu, :]
> myarray', which would mean that the data is on the gpu. Generally
> though, I think it kind of defeats the purpose. E.g. if you have small
> arrays you probably don't want anything to be on the gpu, whereas if
> you have larger ones and sufficient computation operating on them, it
> might be worthwhile. The point is, as a user you don't care, you want
> your runtime to make a sensible decision. If you don't want anything
> to do with OpenCL, you can disable it, or if you want to only ever
> stay on the CPU, you could "pin" it there.

As for code regions, only operations on memoryview slices (most
notably vector operations) and prange sections would be compiled (and
only if possible at all). Maybe normal loops could be compiled as
well, but it's best to start with prange only.

>>>
>>>> Heuristics and
>>>> automatic switching between, say, CPU and GPU is great for eg. Sage
>>>> users, but maybe not so much if you know exactly what you're doing
>>>> with your machine resources. E.g just having a library with thin
>>>> cython-adapted wrappers would be awesome. I imagine this can be
>>>> augmented by arrays having a knowledge of device-side/client-side
>>>> (which would go towards addressing the issue 1. above)
>>>
>>>
>>> Just use PyOpenCL and manipulate kernels as text. Python is excellent for
>>> that - Cython is not needed. If you think using Cython instead of Python
>>> (PyOpenCL and NumPy) will be important, you don't have a CPU bound problem
>>> that warrants the use of OpenCL.
>>
>> Again, not sure what you mean here. As I mentioned in the thread,
>> PyOpenCL worked quite fine, however if Cython is getting OpenCL
>> support, I'd much rather use that than keeping a dependency on another
>> library.
>>
>>> Sturla
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb  7 19:05:04 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 7 Feb 2012 18:05:04 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <4F316651.1040504@molden.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
	<CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
	<4F316651.1040504@molden.no>
Message-ID: <CANg26EVoqmv9udX4K0U7u=carxBO0+QXn+avqQn-tQOCUag8kg@mail.gmail.com>

On 7 February 2012 17:58, Sturla Molden <sturla at molden.no> wrote:
> On 07.02.2012 18:22, Dimitri Tcaciuc wrote:
>
>> I'm not sure I understand you, maybe you could elaborate on that?
>
>
> OpenCL code is a text string that is compiled when the program runs. So it
> can be generated from run-time data. Think of it like dynamic HTML.
>
>
>> Again, not sure what you mean here. As I mentioned in the thread,
>> PyOpenCL worked quite fine, however if Cython is getting OpenCL
>> support, I'd much rather use that than keeping a dependency on another
>> library.
>
>
> You can use PyOpenCL or OpenCL C or C++ headers with Cython. The latter you
> just use as you would with any other C or C++ library. You don't need to
> change the compiler to use a library: It seems like you think OpenCL is
> compiled from code when you build the program. It is actually compiled from
> text strings when you run the program. It is meaningless to ask if Cython
> supports OpenCL because Cython supports any C library.
>

Sturla, in general we appreciate your input, you usually have useful
things to say. But I really don't believe you have read the CEP, so
please do, and then comment on what is proposed there if you want.
Here is the link: http://wiki.cython.org/enhancements/opencl

> Sturla
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb  7 23:21:38 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 7 Feb 2012 22:21:38 +0000
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <CANg26EWK91FRgcmV+JZ5x77KPJs2xcB+ffPLHMrF_x_nuF7Ykg@mail.gmail.com>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWK91FRgcmV+JZ5x77KPJs2xcB+ffPLHMrF_x_nuF7Ykg@mail.gmail.com>
Message-ID: <CANg26EXe9k1vGrH7sOVX+ez8YdUiVWGcR36yXxgHr4eieZNWOA@mail.gmail.com>

On 5 February 2012 22:03, mark florisson <markflorisson88 at gmail.com> wrote:
> On 2 February 2012 21:38, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 02/02/2012 10:16 PM, mark florisson wrote:
>>>
>>> On 2 February 2012 12:19, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> ?wrote:
>>>>
>>>> I just realized that
>>>>
>>>> cdef int[:] a = None
>>>>
>>>> raises an exception; even though I'd argue that 'a' is of the "reference"
>>>> kind of type where Cython usually allow None (i.e., "cdef MyClass b =
>>>> None"
>>>> is allowed even if type(None) is NoneType). Is this a bug or not, and is
>>>> it
>>>> possible to do something about it?
>>>>
>>>> Dag Sverre
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>> Yeah I disabled that quite early. It was supposed to be working but
>>> gave a lot of trouble in cases (segfaults, mainly). At the time I was
>>> trying to get rid of all the segfaults and get the basic functionality
>>> working, so I disabled it. Personally, I have never liked how things
>>
>>
>> Well, you can segfault quite easily with
>>
>> cdef MyClass a = None
>> print a.field
>>
>> so it doesn't make sense to slices different from cdef classes IMO.
>>
>>
>>> can be None unchecked. I personally prefer to write
>>>
>>> cdef foo(obj=None):
>>> ? ? cdef int[:] a
>>> ? ? if obj is None:
>>> ? ? ? ? obj = ...
>>> ? ? a = obj
>>>
>>> Often you forget to write 'not None' when declaring the parameter (and
>>> apparently that it only allowed for 'def' functions).
>>>
>>> As such, I never bothered to re-enable it. However, it does support
>>> control flow with uninitialized slices, and will raise an error if it
>>> is uninitialized. Do we want this behaviour (e.g. for consistency)?
>>
>>
>> When in doubt, go for consistency. So +1 for that reason. I do believe that
>> setting stuff to None is rather vital in Python.
>
> Yeah I think we should go back to this discussion :) Checking for None
> and allowing slices to be None is simply ?very convenient, and doesn't
> involve any drastic changes. I was never really against it, I just
> never got around to implementing it.

We should now be able to use None memoryview slices:
https://github.com/markflorisson88/cython/commit/a24495ac1348926af5e085334c4e6a960e723f87

They also coerce back to None when coercing to object.

>> What I typically do is more like this:
>>
>> def f(double[:] input, double[:] out=None):
>> ? ?if out is None:
>> ? ? ? ?out = np.empty_like(input)
>> ? ?...
>>
>> Having to use another variable name is a bit of a pain. (Come on -- do you
>> use "a" in real code? What do you actually call "the other obj"? I sometimes
>> end up with "out_" and so on, but it creates smelly code quite quickly.)
>>
>> It's easy to segfault with cdef classes anyway, so decent nonechecking
>> should be implemented at some point, and then memoryviews would use the same
>> mechanisms. Java has decent null-checking...
>>
>>
>> Dag Sverre
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Wed Feb  8 09:22:34 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Feb 2012 00:22:34 -0800
Subject: [Cython] memoryview slices can't be None?
In-Reply-To: <4F2D8971.1000102@astro.uio.no>
References: <4F2A7F41.6010303@astro.uio.no>
	<CANg26EUBPjgOof47v5BLbdSYVTQbZHGNfpZH4qZUQ=pB6Mte2A@mail.gmail.com>
	<4F2B0247.8040008@astro.uio.no>
	<CANg26EWqX3qc9Wdxfkb-cV5AHQ9KNwdH6J7MFZE1N58kv8o2BA@mail.gmail.com>
	<4F2C1EFE.4050904@astro.uio.no>
	<CANg26EWyPcOf0VktL-G+vw=2Paz+wBVAx7eAcQXQiku5=C=U_w@mail.gmail.com>
	<CANg26EVhvm6+Lkt0DPCdTnXGEr_=EXAtKsBWcR=JfR6VmiAFNA@mail.gmail.com>
	<4F2C242B.9010309@astro.uio.no>
	<CANg26EVyGLia1vG2hPy+F7nsnrR5WNGd+0egBftacNZTTaSjNg@mail.gmail.com>
	<4F2D8971.1000102@astro.uio.no>
Message-ID: <CADiQ+QD2zmHKyzTdn4+rW8fAm+pFqvJ_ceohTC2wxkzwiOVo9A@mail.gmail.com>

On Sat, Feb 4, 2012 at 11:39 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
>>
>> Block-local declarations are definitely something we want, although I
>> think it would require some more (non-trivial) changes to the
>> compiler.
>
>
> Note that my proposal was actually not about block-local declarations.
>
> Block-local:
>
> {
> ? int x = 4;
> }
> /* x not available here */
>
> My idea was much more like hints to control flow analysis. That is, I wanted
> to have this raise an error:
>
> x = 'adf'
> if foo():
> ? ?cdef int x = y
> print x # type of x not known
>
> This is OK:
>
> if foo():
> ? ?cdef int x = y
> else:
> ? ?cdef int x = 4
> print x # ok, type the same anyway -- so type "escapes" block
>
> And I would allow
>
> cdef str x = y
> if foo:
> ? ?cdef int x = int(x)
> ? ?return g(x) # x must be int
> print x # x must be str at this point
>
>
> The reason for this madness is simply that control statements do NOT create
> blocks in Python, and making it so in Cython is just confusing. It would
> bring too much of C into the language for my taste.

I think the above examples (especially the last one) are a bit
confusing as well. Introducing the notion of (implicit) block scoping
is not very Pythonic. We would need something to be able to support
local cdef classes, but I think a with statement is more appropriate
for that as there's a notion of doing non-trivial work when exiting
the block.

> I think that in my Cython-utopia, Symtab.py is only responsible for
> resolving the scope of *names*, and types of things are not bound to blocks,
> just to the state at control flow points.
>
> Of course, implementing this would be a nightmare.
>
>
>> Maybe the cleanup code from functions, as well as the temp handling
>> etc could be re-factored to a BlockNode, that all block nodes could
>> subclass. They'd have to instantiate new symbol table environments as
>> well. I'm not yet entirely sure what else would be involved in the
>> implementation of that.
>>
>>> But I like int[:] as a way of making it pure Python syntax compatible as
>>> well. Perhaps the two are orthogonal -- a) make variable declaration a
>>> statement, b) make cython.int[:](x) do, essentially, a cdef declaration,
>>> for
>>> Python compatability.
>>>
>>
>> Don't we have cython.declare() for that? e.g.
>>
>> ? ? arr = cython.declare(cython.int[:])
>>
>> That would also be treated as a statement like normal declarations (if
>> and when implemented).
>
>
> This was what I said, but it wasn't what I meant. Sorry. I'll try to explain
> better:
>
> 1) ?There's no way to have the above actually do the right thing in Python.
> With "arr = cython.int[:](arr)" one could actually return a NumPy or
> NumPy-like array that works in Python (since "arr" might not have the
> "shape" attribute before the conversion, all we know is that it exports the
> buffer interface...).
>
> 2) I don't like the fact that we overload the assignment operator to acquire
> a view. "cdef np.ndarray[int] x = y" is fine since if you do "x.someattr"
> then a NumPy subclass could provide someattr and it works fine. Acquiring a
> view is just something different.
>
> 3) Hence I guess I like "arr = int[:](arr)" better both for Cython and
> Python; at least if "arr" is always type-inferred to be int[:], even if arr
> was an "object" further up in the code (really, if you do "x = f(x)" at the
> top-level of the function, then x can just take the identity of another
> variable from that point on -- I don't know if the current control flow
> analysis and type inferences does this though?)
>
>
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Wed Feb  8 09:53:12 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Feb 2012 00:53:12 -0800
Subject: [Cython] OpenCL support
In-Reply-To: <4F316651.1040504@molden.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<CAL41sfQQNpSPBJG4L30HYc-pc++BB+t=NZ=wjX=nqj_PMHyVsA@mail.gmail.com>
	<4F312C85.7050805@molden.no>
	<CAL41sfRiKEdRY9zLThqvAWAyZYxsce6tx3Ao5ojf39NpD_k3uQ@mail.gmail.com>
	<4F316651.1040504@molden.no>
Message-ID: <CADiQ+QAYHCaBszpfL0JtNA2VPDCkYf4HKSwuEzPniPVPtHd5zQ@mail.gmail.com>

On Tue, Feb 7, 2012 at 9:58 AM, Sturla Molden <sturla at molden.no> wrote:
> On 07.02.2012 18:22, Dimitri Tcaciuc wrote:
>
>> I'm not sure I understand you, maybe you could elaborate on that?
>
>
> OpenCL code is a text string that is compiled when the program runs. So it
> can be generated from run-time data. Think of it like dynamic HTML.
>
>
>> Again, not sure what you mean here. As I mentioned in the thread,
>> PyOpenCL worked quite fine, however if Cython is getting OpenCL
>> support, I'd much rather use that than keeping a dependency on another
>> library.
>
>
> You can use PyOpenCL or OpenCL C or C++ headers with Cython. The latter you
> just use as you would with any other C or C++ library. You don't need to
> change the compiler to use a library: It seems like you think OpenCL is
> compiled from code when you build the program. It is actually compiled from
> text strings when you run the program. It is meaningless to ask if Cython
> supports OpenCL because Cython supports any C library.

I view this more as a proposal to have an OpenCL backend for prange
loops and other vectorized operations. The advantage of integrating
OpenCL into Cython is that one can write a single implementation of
your algorithm (using traditional for...(p)range loops) and have it
use the GPU in the background transparently (without having to
manually learn and call the library yourself). This is analogous to
the compiler/runtime system deciding to use sse instructions for a
portion of your code because it thinks it will be faster. I really
like the idea of decoupling the logic of the algorithm from the SIMD
implementation (which is one of the reasons that prange, and in part
OpenMP, works so well) but I think this is best done at the language
level in our case.

Whether OpenCL is mature enough/the abstractions are clean enough/the
heuristics can be good enough to pull this off is another question,
but it'd be great if it can be done (ideally with minimal impact to
the language and isolated changes to the internals).

- Robert

From d.s.seljebotn at astro.uio.no  Wed Feb  8 15:46:23 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 08 Feb 2012 15:46:23 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
Message-ID: <4F328ABF.4010107@astro.uio.no>

On 02/05/2012 10:57 PM, mark florisson wrote:
> Hey,
>
> I created a CEP for opencl support: http://wiki.cython.org/enhancements/opencl
> What do you think?

To start with my own conclusion on this, my feel is that it is too 
little gain, at least for a GPU solution. There's already Theano for 
trivial SIMD-stuff and PyOpenCL for the getting-hands-dirty stuff. (Of 
course, this CEP would be more convenient to use than Theano if one is 
already using Cython.)

But that's just my feeling, and I'm not the one potentially signing up 
to do the work, so whether it is "worth it" is really not my decision, 
the weighing is done with your weights, not mine. Given an 
implementation, I definitely support the inclusion in Cython for these 
kind of features (FWIW).

First, CPU:

OpenCL is probably a very good way of portably making use of SSE/AVX 
etc. But to really get a payoff then I would think that the real value 
would be in *not* using OpenCL vector types, just many threads, so that 
the OpenCL driver does the dirty work of mapping each thread to each 
slot in the CPU registers? I'd think the gain in using OpenCL is to emit 
scalar code and leave the dirty work to OpenCL. If one does the hard 
part and mapped variables to vectors and memory accesses to shuffles, 
one might as well go the whole length and emit SSE/AVX rather than 
OpenCL to avoid the startup overhead.

I don't really know how good the Intel and AMD CPU drivers are w.r.t. 
this -- I have seen the Intel driver emit "vectorizing" and "could not 
vectorize", but didn't explore the circumstances.


Then, on to GPU:

It is not a generic-purpose solution, you still need to bring in 
pyopencl for lots of cases, and so the question is how many cases it 
fits with and if it is enough to grow a userbase around it. And, 
importantly, how much performance is sacrificed for the resulting 
user-friendlyness. 50% performance hit is usually OK, 95% maybe not. And 
a 95% hit is not unimaginable if the memory movement is done in a bad 
way for some code?

I think the fundamental problem is one of programming paradigms. 
Fortran, C++, Cython are all sequential in nature; even with OpenMP it 
is like you have a modest bit of parallelism tacked on to speed up a 
sequential-looking program. With "massively parallel" solutions such as 
CUDA and OpenCL, and also MPI in fact, the fundamental assumption that 
you have thousands or hundreds of thousands of threads. And that just 
changes how you need to think about writing code, which would tend to 
show up at a syntax level. So, at least if you want good performance, 
you need to change your way of thinking enough that a new syntax 
(loosely cooperating threads rather than parallel-for-loop or SIMD 
instruction) is actually an advantage, as it keeps you reminded of how 
the hardware works.

So I think the most important thing to do (if you bother) is: Gather a 
set of real worl(-ish) CUDA or OpenCL programs, port them to Cython + 
this CEP (without a working Cython implementation for it), and see how 
that goes. That's really the only way to evaluate it.

Some experiences from the single instance GPU code I've written:

  - For starters I had to give up OpenCL and use CUDA to use all the 48 
KB available shared memory on Nvidia compute-capability-2.0 (perhaps I 
just didn't find the OpenCL option for that). And increasing from 16 to 
48 KB allowed a fundamentally faster and qualitatively different 
algorithm to be used. But OpenCL vs. CUDA is kind of beside the point 
here....

  - When mucking about with various "obvious" ports of sequential code 
to GPU code, I got performance in the range of 5 to 20 GFLOP/s (out of 
490 GFLOP/s or so theoretical; NVidia Tesla M2050). When really 
understanding the hardware, and making good use of the 48 KB of 
thread-shared memory, I achieved 209 GFLOP/s, without really doing any 
microoptimization. I don't think the CEP includes any features for 
intra-thread communication, so that's off the table.

(My code is here:

https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in

Though it's badly documented and rush-for-deadline-quality; I plan to 
polish it up and publish it when I get time in autumn).

I guess I mention this as the kind of computation your CEP definitely 
does NOT cover. That's probably OK, but one should figure out 
specifically how many usecases it does cover (in particular with no 
control over thread blocks and intra-block communication). Is the CEP a 
80%-solution, or a 10%-solution?

Dag Sverre

From dtcaciuc at gmail.com  Wed Feb  8 18:35:52 2012
From: dtcaciuc at gmail.com (Dimitri Tcaciuc)
Date: Wed, 8 Feb 2012 09:35:52 -0800
Subject: [Cython] OpenCL support
In-Reply-To: <4F328ABF.4010107@astro.uio.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<4F328ABF.4010107@astro.uio.no>
Message-ID: <CAL41sfT8GHCBFumweFzJG6z4wXq3O97=Whg1vydMvY71hqDNMQ@mail.gmail.com>

On Wed, Feb 8, 2012 at 6:46 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/05/2012 10:57 PM, mark florisson wrote:
>
> I don't really know how good the Intel and AMD CPU drivers are w.r.t. this
> -- I have seen the Intel driver emit "vectorizing" and "could not
> vectorize", but didn't explore the circumstances.

For our project, we've tried both Intel and AMD (previously ATI)
backends. The AMD experience somewhat mirrors what this developer
described (http://www.msoos.org/2012/01/amds-opencl-heaven-and-hell/),
although not as bad in terms of silent failures (or maybe I just
havent caught any!).

Intel backend was great and clearly better in terms of performance,
sometimes by about 20-30%. However, when ran on older AMD-based
machine as opposed to Intel one, the resulting kernel simply
segfaulted without any warning about an unsupported architecture (I
think its because it didn't have SSE3 support).

>
> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


I know Intel is working with LLVM/Clang folks to introduce their
vectorization additions, at least to some degree, and LLVM seems to be
consistently improving in this regard (eg
http://blog.llvm.org/2011/12/llvm-31-vector-changes.html). I suppose
if Cython emitted vectorization-friendly numerical loops, then
appropriate C/C++ compiler should take care of this automatically, if
used. Intel C++ can already do certain stuff like that (see
http://software.intel.com/en-us/articles/a-guide-to-auto-vectorization-with-intel-c-compilers/),
and GCC as well AFAIK.

Dimitri.

From markflorisson88 at gmail.com  Wed Feb  8 23:11:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 8 Feb 2012 22:11:43 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <4F328ABF.4010107@astro.uio.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<4F328ABF.4010107@astro.uio.no>
Message-ID: <CANg26EWd7xv_3jnd2+bhib0CMXFQqtjTxQH0h=MJdCN42ZJt7g@mail.gmail.com>

On 8 February 2012 14:46, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/05/2012 10:57 PM, mark florisson wrote:
>>
>> Hey,
>>
>> I created a CEP for opencl support:
>> http://wiki.cython.org/enhancements/opencl
>> What do you think?
>
>
> To start with my own conclusion on this, my feel is that it is too little
> gain, at least for a GPU solution. There's already Theano for trivial
> SIMD-stuff and PyOpenCL for the getting-hands-dirty stuff. (Of course, this
> CEP would be more convenient to use than Theano if one is already using
> Cython.)

Yes, vector operations and elemental or reduction functions operator
on vectors (which is what we can use Theano for, right?) don't quite
merit the use of OpenCL. However, the upside is that OpenCL allows
easier vectorization and multi-threading. We can appease to
auto-vectorizing compilers, but e.g. using OpenMP for multithreading
will still segfault the program if used outside the main thread with
gcc's implementation. I believe intel allows you to use it in any
thread. (Of course, keeping a thread pool around and managing it
manually isn't too hard, but...)

> But that's just my feeling, and I'm not the one potentially signing up to do
> the work, so whether it is "worth it" is really not my decision, the
> weighing is done with your weights, not mine. Given an implementation, I
> definitely support the inclusion in Cython for these kind of features
> (FWIW).
>
> First, CPU:
>
> OpenCL is probably a very good way of portably making use of SSE/AVX etc.
> But to really get a payoff then I would think that the real value would be
> in *not* using OpenCL vector types, just many threads, so that the OpenCL
> driver does the dirty work of mapping each thread to each slot in the CPU
> registers? I'd think the gain in using OpenCL is to emit scalar code and
> leave the dirty work to OpenCL. If one does the hard part and mapped
> variables to vectors and memory accesses to shuffles, one might as well go
> the whole length and emit SSE/AVX rather than OpenCL to avoid the startup
> overhead.
>
> I don't really know how good the Intel and AMD CPU drivers are w.r.t. this
> -- I have seen the Intel driver emit "vectorizing" and "could not
> vectorize", but didn't explore the circumstances.
>

I initially thought the same thing, single kernel invocations should
be trivially auto-vectorizable one would think. At least with Apple
OpenCL I am getting better performance with vector types though on the
CPU (up to 35%). I would personally consider emitting vector data
types bonus points.

But I don't quite agree that emitting SSE or AVX directly would be
almost as easy in that case. You'd still have to detect at runtime
which instruction set is supported and generate SSE, SSE2, (SSE4?) and
AVX. And that's not even all of them :) The OpenCL drivers just hide
that pain. With handwritten code you might be coding for a specific
architecture and might be fine with only SSE2, but as a compiler we
can't really make that same decision.

> Then, on to GPU:
>
> It is not a generic-purpose solution, you still need to bring in pyopencl
> for lots of cases, and so the question is how many cases it fits with and if
> it is enough to grow a userbase around it. And, importantly, how much
> performance is sacrificed for the resulting user-friendlyness. 50%
> performance hit is usually OK, 95% maybe not. And a 95% hit is not
> unimaginable if the memory movement is done in a bad way for some code?

Yes, I don't expect this to change a lot suddenly. In the long term I
think the implementation could be sufficiently good to support at
least most codes. And the user still has full control over data
movement, if wanted (the pinning thing, which isn't mentioned in the
CEP).

> I think the fundamental problem is one of programming paradigms. Fortran,
> C++, Cython are all sequential in nature; even with OpenMP it is like you
> have a modest bit of parallelism tacked on to speed up a sequential-looking
> program. With "massively parallel" solutions such as CUDA and OpenCL, and
> also MPI in fact, the fundamental assumption that you have thousands or
> hundreds of thousands of threads. And that just changes how you need to
> think about writing code, which would tend to show up at a syntax level. So,
> at least if you want good performance, you need to change your way of
> thinking enough that a new syntax (loosely cooperating threads rather than
> parallel-for-loop or SIMD instruction) is actually an advantage, as it keeps
> you reminded of how the hardware works.
>
> So I think the most important thing to do (if you bother) is: Gather a set
> of real worl(-ish) CUDA or OpenCL programs, port them to Cython + this CEP
> (without a working Cython implementation for it), and see how that goes.
> That's really the only way to evaluate it.

I've been wanting to do that for a long time now, also to evaluate the
capabilities of cython.parallel as it stands now. It's a really good
idea, I'll try to port some codes, and not just the trivial ones like
Jacobi's method :).

> Some experiences from the single instance GPU code I've written:
>
> ?- For starters I had to give up OpenCL and use CUDA to use all the 48 KB
> available shared memory on Nvidia compute-capability-2.0 (perhaps I just
> didn't find the OpenCL option for that). And increasing from 16 to 48 KB
> allowed a fundamentally faster and qualitatively different algorithm to be
> used. But OpenCL vs. CUDA is kind of beside the point here....
>
> ?- When mucking about with various "obvious" ports of sequential code to GPU
> code, I got performance in the range of 5 to 20 GFLOP/s (out of 490 GFLOP/s
> or so theoretical; NVidia Tesla M2050). When really understanding the
> hardware, and making good use of the 48 KB of thread-shared memory, I
> achieved 209 GFLOP/s, without really doing any microoptimization. I don't
> think the CEP includes any features for intra-thread communication, so
> that's off the table.

The CEP doesn't mention barriers (discussed earlier), but they should
be supported, and __local memory (that's "shared memory" in CUDA terms
right?) could be utilized using a more explicit scheme (or implicitly
if the compiler is smart). The only issue with barriers is that with
OpenCL you have multiple levels of synchronization, but barriers only
work within the work group / thread block, whereas with openmp it
works simply for all your threads. I think a global barrier would have
to mean kernel termination + start of a new one, which could be hard
to support depending on where it is placed in the code...

> (My code is here:
>
> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
>
> Though it's badly documented and rush-for-deadline-quality; I plan to polish
> it up and publish it when I get time in autumn).
>
> I guess I mention this as the kind of computation your CEP definitely does
> NOT cover. That's probably OK, but one should figure out specifically how
> many usecases it does cover (in particular with no control over thread
> blocks and intra-block communication). Is the CEP a 80%-solution, or a
> 10%-solution?

I haven't looked too carefully at the code, but a large portion is
dedicated to a reduction right? What I don't see is how your reduction
spans across multiple work-groups / thread blocks? Because
__syncthreads should only sync stuff within a single block. The CEP
didn't mention reductions, but they should be supported (I'm thinking
multi-stage or sequential within the workgroup (whatever works
better), followed by another kernel invocation if the result is
needed).
As mentioned earlier in a different thread (on parallelism I think),
reduction arrays (i.e. memoryviews or C arrays) as well as generally
private arrays should be supported. An issue with that is that you
can't really dedicate an array to each work item / thread (too much
memory would be consumed).

Again, declarations within blocks would solve many problems:

    cdef float[n] shared_by_work_group
    with parallel():
        cdef float[n] local_to_work_group
        for i in prange(...):
            cdef float[n] local_to_work_item

For arrays, the reductions could be somewhat more explicit, where
there is an explicit 'my_memoryview += my_local_scratch_data'. That
should probably only be allowed for memory local to the work group.

Anyway, I'll try porting some numerical codes to this scheme over the
coming weeks and see what is missing and how it can be solved. I still
believe it can all be made to work quite properly, without adjusting
the language to fit the hardware model. The prange (and OpenMP) model
look like sequential code, but they tell the compiler a lot, namely
that each iteration is independent and could therefore be scheduled as
a separate thread.

> Dag Sverre
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Wed Feb  8 23:13:45 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 8 Feb 2012 22:13:45 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <CAL41sfT8GHCBFumweFzJG6z4wXq3O97=Whg1vydMvY71hqDNMQ@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<4F328ABF.4010107@astro.uio.no>
	<CAL41sfT8GHCBFumweFzJG6z4wXq3O97=Whg1vydMvY71hqDNMQ@mail.gmail.com>
Message-ID: <CANg26EWu4ZEj3dchZzfUJVb434a2+-ST6Jz52jeUE70HSN2b0w@mail.gmail.com>

On 8 February 2012 17:35, Dimitri Tcaciuc <dtcaciuc at gmail.com> wrote:
> On Wed, Feb 8, 2012 at 6:46 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 02/05/2012 10:57 PM, mark florisson wrote:
>>
>> I don't really know how good the Intel and AMD CPU drivers are w.r.t. this
>> -- I have seen the Intel driver emit "vectorizing" and "could not
>> vectorize", but didn't explore the circumstances.
>
> For our project, we've tried both Intel and AMD (previously ATI)
> backends. The AMD experience somewhat mirrors what this developer
> described (http://www.msoos.org/2012/01/amds-opencl-heaven-and-hell/),
> although not as bad in terms of silent failures (or maybe I just
> havent caught any!).
>
> Intel backend was great and clearly better in terms of performance,
> sometimes by about 20-30%. However, when ran on older AMD-based
> machine as opposed to Intel one, the resulting kernel simply
> segfaulted without any warning about an unsupported architecture (I
> think its because it didn't have SSE3 support).
>
>>
>> Dag Sverre
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
> I know Intel is working with LLVM/Clang folks to introduce their
> vectorization additions, at least to some degree, and LLVM seems to be
> consistently improving in this regard (eg
> http://blog.llvm.org/2011/12/llvm-31-vector-changes.html). I suppose
> if Cython emitted vectorization-friendly numerical loops, then
> appropriate C/C++ compiler should take care of this automatically, if
> used. Intel C++ can already do certain stuff like that (see
> http://software.intel.com/en-us/articles/a-guide-to-auto-vectorization-with-intel-c-compilers/),
> and GCC as well AFAIK.

Indeed, native C (hopefully auto-vectorized whenever possible) is what
we also hope to use (depending on heuristics). But what it doesn't
give you is multithreading for the CPU (and e.g. the grand central
dispatch on OS X).

> Dimitri.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From d.s.seljebotn at astro.uio.no  Thu Feb  9 00:15:43 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 09 Feb 2012 00:15:43 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <CANg26EWd7xv_3jnd2+bhib0CMXFQqtjTxQH0h=MJdCN42ZJt7g@mail.gmail.com>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>	<4F328ABF.4010107@astro.uio.no>
	<CANg26EWd7xv_3jnd2+bhib0CMXFQqtjTxQH0h=MJdCN42ZJt7g@mail.gmail.com>
Message-ID: <4F33021F.7010903@astro.uio.no>

On 02/08/2012 11:11 PM, mark florisson wrote:
> On 8 February 2012 14:46, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 02/05/2012 10:57 PM, mark florisson wrote:
>>>
>>> Hey,
>>>
>>> I created a CEP for opencl support:
>>> http://wiki.cython.org/enhancements/opencl
>>> What do you think?
>>
>>
>> To start with my own conclusion on this, my feel is that it is too little
>> gain, at least for a GPU solution. There's already Theano for trivial
>> SIMD-stuff and PyOpenCL for the getting-hands-dirty stuff. (Of course, this
>> CEP would be more convenient to use than Theano if one is already using
>> Cython.)
>
> Yes, vector operations and elemental or reduction functions operator
> on vectors (which is what we can use Theano for, right?) don't quite
> merit the use of OpenCL. However, the upside is that OpenCL allows
> easier vectorization and multi-threading. We can appease to
> auto-vectorizing compilers, but e.g. using OpenMP for multithreading
> will still segfault the program if used outside the main thread with
> gcc's implementation. I believe intel allows you to use it in any
> thread. (Of course, keeping a thread pool around and managing it
> manually isn't too hard, but...)
>
>> But that's just my feeling, and I'm not the one potentially signing up to do
>> the work, so whether it is "worth it" is really not my decision, the
>> weighing is done with your weights, not mine. Given an implementation, I
>> definitely support the inclusion in Cython for these kind of features
>> (FWIW).
>>
>> First, CPU:
>>
>> OpenCL is probably a very good way of portably making use of SSE/AVX etc.
>> But to really get a payoff then I would think that the real value would be
>> in *not* using OpenCL vector types, just many threads, so that the OpenCL
>> driver does the dirty work of mapping each thread to each slot in the CPU
>> registers? I'd think the gain in using OpenCL is to emit scalar code and
>> leave the dirty work to OpenCL. If one does the hard part and mapped
>> variables to vectors and memory accesses to shuffles, one might as well go
>> the whole length and emit SSE/AVX rather than OpenCL to avoid the startup
>> overhead.
>>
>> I don't really know how good the Intel and AMD CPU drivers are w.r.t. this
>> -- I have seen the Intel driver emit "vectorizing" and "could not
>> vectorize", but didn't explore the circumstances.
>>
>
> I initially thought the same thing, single kernel invocations should
> be trivially auto-vectorizable one would think. At least with Apple
> OpenCL I am getting better performance with vector types though on the
> CPU (up to 35%). I would personally consider emitting vector data
> types bonus points.
>
> But I don't quite agree that emitting SSE or AVX directly would be
> almost as easy in that case. You'd still have to detect at runtime
> which instruction set is supported and generate SSE, SSE2, (SSE4?) and
> AVX. And that's not even all of them :) The OpenCL drivers just hide
> that pain. With handwritten code you might be coding for a specific
> architecture and might be fine with only SSE2, but as a compiler we
> can't really make that same decision.

You make good points.

>
>> Then, on to GPU:
>>
>> It is not a generic-purpose solution, you still need to bring in pyopencl
>> for lots of cases, and so the question is how many cases it fits with and if
>> it is enough to grow a userbase around it. And, importantly, how much
>> performance is sacrificed for the resulting user-friendlyness. 50%
>> performance hit is usually OK, 95% maybe not. And a 95% hit is not
>> unimaginable if the memory movement is done in a bad way for some code?
>
> Yes, I don't expect this to change a lot suddenly. In the long term I
> think the implementation could be sufficiently good to support at
> least most codes. And the user still has full control over data
> movement, if wanted (the pinning thing, which isn't mentioned in the
> CEP).
>
>> I think the fundamental problem is one of programming paradigms. Fortran,
>> C++, Cython are all sequential in nature; even with OpenMP it is like you
>> have a modest bit of parallelism tacked on to speed up a sequential-looking
>> program. With "massively parallel" solutions such as CUDA and OpenCL, and
>> also MPI in fact, the fundamental assumption that you have thousands or
>> hundreds of thousands of threads. And that just changes how you need to
>> think about writing code, which would tend to show up at a syntax level. So,
>> at least if you want good performance, you need to change your way of
>> thinking enough that a new syntax (loosely cooperating threads rather than
>> parallel-for-loop or SIMD instruction) is actually an advantage, as it keeps
>> you reminded of how the hardware works.
>>
>> So I think the most important thing to do (if you bother) is: Gather a set
>> of real worl(-ish) CUDA or OpenCL programs, port them to Cython + this CEP
>> (without a working Cython implementation for it), and see how that goes.
>> That's really the only way to evaluate it.
>
> I've been wanting to do that for a long time now, also to evaluate the
> capabilities of cython.parallel as it stands now. It's a really good
> idea, I'll try to port some codes, and not just the trivial ones like
> Jacobi's method :).
>
>> Some experiences from the single instance GPU code I've written:
>>
>>   - For starters I had to give up OpenCL and use CUDA to use all the 48 KB
>> available shared memory on Nvidia compute-capability-2.0 (perhaps I just
>> didn't find the OpenCL option for that). And increasing from 16 to 48 KB
>> allowed a fundamentally faster and qualitatively different algorithm to be
>> used. But OpenCL vs. CUDA is kind of beside the point here....
>>
>>   - When mucking about with various "obvious" ports of sequential code to GPU
>> code, I got performance in the range of 5 to 20 GFLOP/s (out of 490 GFLOP/s
>> or so theoretical; NVidia Tesla M2050). When really understanding the
>> hardware, and making good use of the 48 KB of thread-shared memory, I
>> achieved 209 GFLOP/s, without really doing any microoptimization. I don't
>> think the CEP includes any features for intra-thread communication, so
>> that's off the table.
>
> The CEP doesn't mention barriers (discussed earlier), but they should
> be supported, and __local memory (that's "shared memory" in CUDA terms
> right?) could be utilized using a more explicit scheme (or implicitly
> if the compiler is smart). The only issue with barriers is that with
> OpenCL you have multiple levels of synchronization, but barriers only
> work within the work group / thread block, whereas with openmp it
> works simply for all your threads. I think a global barrier would have
> to mean kernel termination + start of a new one, which could be hard
> to support depending on where it is placed in the code...
>
>> (My code is here:
>>
>> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
>>
>> Though it's badly documented and rush-for-deadline-quality; I plan to polish
>> it up and publish it when I get time in autumn).
>>
>> I guess I mention this as the kind of computation your CEP definitely does
>> NOT cover. That's probably OK, but one should figure out specifically how
>> many usecases it does cover (in particular with no control over thread
>> blocks and intra-block communication). Is the CEP a 80%-solution, or a
>> 10%-solution?
>
> I haven't looked too carefully at the code, but a large portion is
> dedicated to a reduction right? What I don't see is how your reduction
> spans across multiple work-groups / thread blocks? Because
> __syncthreads should only sync stuff within a single block. The CEP

There's no need to reduce across thread blocks because (conveniently 
enough) there's 8000 independent computations to be performed with 
different parameters. I simply used one thread block for each problem.

It's basically a matrix-vector product where the matrix must be 
generated on the fly columnwise (one entry can be generated from the 
preceding two in the same column), but the summation is row-wise.

And turns out that getting inter-thread sum-reduction to work well was 
harder than I expected; a 32-by-32 matrix (needed since warps are 32 
threads) is too big to fit in memory, but tree-reduction makes a lot of 
the threads in a warp do nothing. So I ended up with a hybrid approach; 
there's visual demo from page 49 onwards here:

http://folk.uio.no/dagss/talk-gpusht.pdf

Getting back to Cython, I'll admit that this form of inter-thread 
reduction is quite generic, and that my specific problem could be solved 
by basically coding a set of inter-thread reduction algorithms suitable 
for different hardware into Cython.

> didn't mention reductions, but they should be supported (I'm thinking
> multi-stage or sequential within the workgroup (whatever works
> better), followed by another kernel invocation if the result is
> needed).

Multiple kernel invocations for global barriers appear to be pretty 
standard, and it's why OpenCL support queueing tasks with dependencies etc.

> As mentioned earlier in a different thread (on parallelism I think),
> reduction arrays (i.e. memoryviews or C arrays) as well as generally
> private arrays should be supported. An issue with that is that you
> can't really dedicate an array to each work item / thread (too much
> memory would be consumed).
>
> Again, declarations within blocks would solve many problems:
>
>      cdef float[n] shared_by_work_group
>      with parallel():
>          cdef float[n] local_to_work_group
>          for i in prange(...):
>              cdef float[n] local_to_work_item
>
> For arrays, the reductions could be somewhat more explicit, where
> there is an explicit 'my_memoryview += my_local_scratch_data'. That
> should probably only be allowed for memory local to the work group.
>
> Anyway, I'll try porting some numerical codes to this scheme over the
> coming weeks and see what is missing and how it can be solved. I still
> believe it can all be made to work quite properly, without adjusting
> the language to fit the hardware model. The prange (and OpenMP) model
> look like sequential code, but they tell the compiler a lot, namely
> that each iteration is independent and could therefore be scheduled as
> a separate thread.

Again, good points.

Dag

From d.s.seljebotn at astro.uio.no  Thu Feb  9 00:28:29 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 09 Feb 2012 00:28:29 +0100
Subject: [Cython] OpenCL support
In-Reply-To: <4F33021F.7010903@astro.uio.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>	<4F328ABF.4010107@astro.uio.no>	<CANg26EWd7xv_3jnd2+bhib0CMXFQqtjTxQH0h=MJdCN42ZJt7g@mail.gmail.com>
	<4F33021F.7010903@astro.uio.no>
Message-ID: <4F33051D.5050309@astro.uio.no>

On 02/09/2012 12:15 AM, Dag Sverre Seljebotn wrote:
> On 02/08/2012 11:11 PM, mark florisson wrote:
>> On 8 February 2012 14:46, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 02/05/2012 10:57 PM, mark florisson wrote:
>>>>
>>>> Hey,
>>>>
>>>> I created a CEP for opencl support:
>>>> http://wiki.cython.org/enhancements/opencl
>>>> What do you think?
>>>
>>>
>>> To start with my own conclusion on this, my feel is that it is too
>>> little
>>> gain, at least for a GPU solution. There's already Theano for trivial
>>> SIMD-stuff and PyOpenCL for the getting-hands-dirty stuff. (Of
>>> course, this
>>> CEP would be more convenient to use than Theano if one is already using
>>> Cython.)
>>
>> Yes, vector operations and elemental or reduction functions operator
>> on vectors (which is what we can use Theano for, right?) don't quite
>> merit the use of OpenCL. However, the upside is that OpenCL allows
>> easier vectorization and multi-threading. We can appease to
>> auto-vectorizing compilers, but e.g. using OpenMP for multithreading
>> will still segfault the program if used outside the main thread with
>> gcc's implementation. I believe intel allows you to use it in any
>> thread. (Of course, keeping a thread pool around and managing it
>> manually isn't too hard, but...)
>>
>>> But that's just my feeling, and I'm not the one potentially signing
>>> up to do
>>> the work, so whether it is "worth it" is really not my decision, the
>>> weighing is done with your weights, not mine. Given an implementation, I
>>> definitely support the inclusion in Cython for these kind of features
>>> (FWIW).
>>>
>>> First, CPU:
>>>
>>> OpenCL is probably a very good way of portably making use of SSE/AVX
>>> etc.
>>> But to really get a payoff then I would think that the real value
>>> would be
>>> in *not* using OpenCL vector types, just many threads, so that the
>>> OpenCL
>>> driver does the dirty work of mapping each thread to each slot in the
>>> CPU
>>> registers? I'd think the gain in using OpenCL is to emit scalar code and
>>> leave the dirty work to OpenCL. If one does the hard part and mapped
>>> variables to vectors and memory accesses to shuffles, one might as
>>> well go
>>> the whole length and emit SSE/AVX rather than OpenCL to avoid the
>>> startup
>>> overhead.
>>>
>>> I don't really know how good the Intel and AMD CPU drivers are w.r.t.
>>> this
>>> -- I have seen the Intel driver emit "vectorizing" and "could not
>>> vectorize", but didn't explore the circumstances.
>>>
>>
>> I initially thought the same thing, single kernel invocations should
>> be trivially auto-vectorizable one would think. At least with Apple
>> OpenCL I am getting better performance with vector types though on the
>> CPU (up to 35%). I would personally consider emitting vector data
>> types bonus points.
>>
>> But I don't quite agree that emitting SSE or AVX directly would be
>> almost as easy in that case. You'd still have to detect at runtime
>> which instruction set is supported and generate SSE, SSE2, (SSE4?) and
>> AVX. And that's not even all of them :) The OpenCL drivers just hide
>> that pain. With handwritten code you might be coding for a specific
>> architecture and might be fine with only SSE2, but as a compiler we
>> can't really make that same decision.
>
> You make good points.
>
>>
>>> Then, on to GPU:
>>>
>>> It is not a generic-purpose solution, you still need to bring in
>>> pyopencl
>>> for lots of cases, and so the question is how many cases it fits with
>>> and if
>>> it is enough to grow a userbase around it. And, importantly, how much
>>> performance is sacrificed for the resulting user-friendlyness. 50%
>>> performance hit is usually OK, 95% maybe not. And a 95% hit is not
>>> unimaginable if the memory movement is done in a bad way for some code?
>>
>> Yes, I don't expect this to change a lot suddenly. In the long term I
>> think the implementation could be sufficiently good to support at
>> least most codes. And the user still has full control over data
>> movement, if wanted (the pinning thing, which isn't mentioned in the
>> CEP).
>>
>>> I think the fundamental problem is one of programming paradigms.
>>> Fortran,
>>> C++, Cython are all sequential in nature; even with OpenMP it is like
>>> you
>>> have a modest bit of parallelism tacked on to speed up a
>>> sequential-looking
>>> program. With "massively parallel" solutions such as CUDA and OpenCL,
>>> and
>>> also MPI in fact, the fundamental assumption that you have thousands or
>>> hundreds of thousands of threads. And that just changes how you need to
>>> think about writing code, which would tend to show up at a syntax
>>> level. So,
>>> at least if you want good performance, you need to change your way of
>>> thinking enough that a new syntax (loosely cooperating threads rather
>>> than
>>> parallel-for-loop or SIMD instruction) is actually an advantage, as
>>> it keeps
>>> you reminded of how the hardware works.
>>>
>>> So I think the most important thing to do (if you bother) is: Gather
>>> a set
>>> of real worl(-ish) CUDA or OpenCL programs, port them to Cython +
>>> this CEP
>>> (without a working Cython implementation for it), and see how that goes.
>>> That's really the only way to evaluate it.
>>
>> I've been wanting to do that for a long time now, also to evaluate the
>> capabilities of cython.parallel as it stands now. It's a really good
>> idea, I'll try to port some codes, and not just the trivial ones like
>> Jacobi's method :).
>>
>>> Some experiences from the single instance GPU code I've written:
>>>
>>> - For starters I had to give up OpenCL and use CUDA to use all the 48 KB
>>> available shared memory on Nvidia compute-capability-2.0 (perhaps I just
>>> didn't find the OpenCL option for that). And increasing from 16 to 48 KB
>>> allowed a fundamentally faster and qualitatively different algorithm
>>> to be
>>> used. But OpenCL vs. CUDA is kind of beside the point here....
>>>
>>> - When mucking about with various "obvious" ports of sequential code
>>> to GPU
>>> code, I got performance in the range of 5 to 20 GFLOP/s (out of 490
>>> GFLOP/s
>>> or so theoretical; NVidia Tesla M2050). When really understanding the
>>> hardware, and making good use of the 48 KB of thread-shared memory, I
>>> achieved 209 GFLOP/s, without really doing any microoptimization. I
>>> don't
>>> think the CEP includes any features for intra-thread communication, so
>>> that's off the table.
>>
>> The CEP doesn't mention barriers (discussed earlier), but they should
>> be supported, and __local memory (that's "shared memory" in CUDA terms
>> right?) could be utilized using a more explicit scheme (or implicitly
>> if the compiler is smart). The only issue with barriers is that with
>> OpenCL you have multiple levels of synchronization, but barriers only
>> work within the work group / thread block, whereas with openmp it
>> works simply for all your threads. I think a global barrier would have
>> to mean kernel termination + start of a new one, which could be hard
>> to support depending on where it is placed in the code...
>>
>>> (My code is here:
>>>
>>> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
>>>
>>>
>>> Though it's badly documented and rush-for-deadline-quality; I plan to
>>> polish
>>> it up and publish it when I get time in autumn).
>>>
>>> I guess I mention this as the kind of computation your CEP definitely
>>> does
>>> NOT cover. That's probably OK, but one should figure out specifically
>>> how
>>> many usecases it does cover (in particular with no control over thread
>>> blocks and intra-block communication). Is the CEP a 80%-solution, or a
>>> 10%-solution?
>>
>> I haven't looked too carefully at the code, but a large portion is
>> dedicated to a reduction right? What I don't see is how your reduction
>> spans across multiple work-groups / thread blocks? Because
>> __syncthreads should only sync stuff within a single block. The CEP

Most of the time there's actually no explicit synchronization, but the 
code relies on all threads of a warp being on the same instruction in 
the scheduler. __synchtreads is then only used at the end of the 
reduction when all within-warp additions have been done. Calling 
__syncthreads at each step of the algorithm would have totally killed 
performance.

Dag Sverre

>
> There's no need to reduce across thread blocks because (conveniently
> enough) there's 8000 independent computations to be performed with
> different parameters. I simply used one thread block for each problem.
>
> It's basically a matrix-vector product where the matrix must be
> generated on the fly columnwise (one entry can be generated from the
> preceding two in the same column), but the summation is row-wise.
>
> And turns out that getting inter-thread sum-reduction to work well was
> harder than I expected; a 32-by-32 matrix (needed since warps are 32
> threads) is too big to fit in memory, but tree-reduction makes a lot of
> the threads in a warp do nothing. So I ended up with a hybrid approach;
> there's visual demo from page 49 onwards here:
>
> http://folk.uio.no/dagss/talk-gpusht.pdf
>
> Getting back to Cython, I'll admit that this form of inter-thread
> reduction is quite generic, and that my specific problem could be solved
> by basically coding a set of inter-thread reduction algorithms suitable
> for different hardware into Cython.
>
>> didn't mention reductions, but they should be supported (I'm thinking
>> multi-stage or sequential within the workgroup (whatever works
>> better), followed by another kernel invocation if the result is
>> needed).
>
> Multiple kernel invocations for global barriers appear to be pretty
> standard, and it's why OpenCL support queueing tasks with dependencies etc.
>
>> As mentioned earlier in a different thread (on parallelism I think),
>> reduction arrays (i.e. memoryviews or C arrays) as well as generally
>> private arrays should be supported. An issue with that is that you
>> can't really dedicate an array to each work item / thread (too much
>> memory would be consumed).
>>
>> Again, declarations within blocks would solve many problems:
>>
>> cdef float[n] shared_by_work_group
>> with parallel():
>> cdef float[n] local_to_work_group
>> for i in prange(...):
>> cdef float[n] local_to_work_item
>>
>> For arrays, the reductions could be somewhat more explicit, where
>> there is an explicit 'my_memoryview += my_local_scratch_data'. That
>> should probably only be allowed for memory local to the work group.
>>
>> Anyway, I'll try porting some numerical codes to this scheme over the
>> coming weeks and see what is missing and how it can be solved. I still
>> believe it can all be made to work quite properly, without adjusting
>> the language to fit the hardware model. The prange (and OpenMP) model
>> look like sequential code, but they tell the compiler a lot, namely
>> that each iteration is independent and could therefore be scheduled as
>> a separate thread.
>
> Again, good points.
>
> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From markflorisson88 at gmail.com  Thu Feb  9 13:52:07 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 9 Feb 2012 12:52:07 +0000
Subject: [Cython] OpenCL support
In-Reply-To: <4F33051D.5050309@astro.uio.no>
References: <CANg26EW3ZKBpAOP_V2e9kvvgrG=VOUTpE1Q2=CqHmC7iLJQ9KQ@mail.gmail.com>
	<4F328ABF.4010107@astro.uio.no>
	<CANg26EWd7xv_3jnd2+bhib0CMXFQqtjTxQH0h=MJdCN42ZJt7g@mail.gmail.com>
	<4F33021F.7010903@astro.uio.no> <4F33051D.5050309@astro.uio.no>
Message-ID: <CANg26EVkXzPhdMEXkK3=ytGaovqpU=was0tTt55-yAwY+X=-Rg@mail.gmail.com>

On 8 February 2012 23:28, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/09/2012 12:15 AM, Dag Sverre Seljebotn wrote:
>>
>> On 02/08/2012 11:11 PM, mark florisson wrote:
>>>
>>> On 8 February 2012 14:46, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>
>>>> On 02/05/2012 10:57 PM, mark florisson wrote:
>>>>>
>>>>>
>>>>> Hey,
>>>>>
>>>>> I created a CEP for opencl support:
>>>>> http://wiki.cython.org/enhancements/opencl
>>>>> What do you think?
>>>>
>>>>
>>>>
>>>> To start with my own conclusion on this, my feel is that it is too
>>>> little
>>>> gain, at least for a GPU solution. There's already Theano for trivial
>>>> SIMD-stuff and PyOpenCL for the getting-hands-dirty stuff. (Of
>>>> course, this
>>>> CEP would be more convenient to use than Theano if one is already using
>>>> Cython.)
>>>
>>>
>>> Yes, vector operations and elemental or reduction functions operator
>>> on vectors (which is what we can use Theano for, right?) don't quite
>>> merit the use of OpenCL. However, the upside is that OpenCL allows
>>> easier vectorization and multi-threading. We can appease to
>>> auto-vectorizing compilers, but e.g. using OpenMP for multithreading
>>> will still segfault the program if used outside the main thread with
>>> gcc's implementation. I believe intel allows you to use it in any
>>> thread. (Of course, keeping a thread pool around and managing it
>>> manually isn't too hard, but...)
>>>
>>>> But that's just my feeling, and I'm not the one potentially signing
>>>> up to do
>>>> the work, so whether it is "worth it" is really not my decision, the
>>>> weighing is done with your weights, not mine. Given an implementation, I
>>>> definitely support the inclusion in Cython for these kind of features
>>>> (FWIW).
>>>>
>>>> First, CPU:
>>>>
>>>> OpenCL is probably a very good way of portably making use of SSE/AVX
>>>> etc.
>>>> But to really get a payoff then I would think that the real value
>>>> would be
>>>> in *not* using OpenCL vector types, just many threads, so that the
>>>> OpenCL
>>>> driver does the dirty work of mapping each thread to each slot in the
>>>> CPU
>>>> registers? I'd think the gain in using OpenCL is to emit scalar code and
>>>> leave the dirty work to OpenCL. If one does the hard part and mapped
>>>> variables to vectors and memory accesses to shuffles, one might as
>>>> well go
>>>> the whole length and emit SSE/AVX rather than OpenCL to avoid the
>>>> startup
>>>> overhead.
>>>>
>>>> I don't really know how good the Intel and AMD CPU drivers are w.r.t.
>>>> this
>>>> -- I have seen the Intel driver emit "vectorizing" and "could not
>>>> vectorize", but didn't explore the circumstances.
>>>>
>>>
>>> I initially thought the same thing, single kernel invocations should
>>> be trivially auto-vectorizable one would think. At least with Apple
>>> OpenCL I am getting better performance with vector types though on the
>>> CPU (up to 35%). I would personally consider emitting vector data
>>> types bonus points.
>>>
>>> But I don't quite agree that emitting SSE or AVX directly would be
>>> almost as easy in that case. You'd still have to detect at runtime
>>> which instruction set is supported and generate SSE, SSE2, (SSE4?) and
>>> AVX. And that's not even all of them :) The OpenCL drivers just hide
>>> that pain. With handwritten code you might be coding for a specific
>>> architecture and might be fine with only SSE2, but as a compiler we
>>> can't really make that same decision.
>>
>>
>> You make good points.
>>
>>>
>>>> Then, on to GPU:
>>>>
>>>> It is not a generic-purpose solution, you still need to bring in
>>>> pyopencl
>>>> for lots of cases, and so the question is how many cases it fits with
>>>> and if
>>>> it is enough to grow a userbase around it. And, importantly, how much
>>>> performance is sacrificed for the resulting user-friendlyness. 50%
>>>> performance hit is usually OK, 95% maybe not. And a 95% hit is not
>>>> unimaginable if the memory movement is done in a bad way for some code?
>>>
>>>
>>> Yes, I don't expect this to change a lot suddenly. In the long term I
>>> think the implementation could be sufficiently good to support at
>>> least most codes. And the user still has full control over data
>>> movement, if wanted (the pinning thing, which isn't mentioned in the
>>> CEP).
>>>
>>>> I think the fundamental problem is one of programming paradigms.
>>>> Fortran,
>>>> C++, Cython are all sequential in nature; even with OpenMP it is like
>>>> you
>>>> have a modest bit of parallelism tacked on to speed up a
>>>> sequential-looking
>>>> program. With "massively parallel" solutions such as CUDA and OpenCL,
>>>> and
>>>> also MPI in fact, the fundamental assumption that you have thousands or
>>>> hundreds of thousands of threads. And that just changes how you need to
>>>> think about writing code, which would tend to show up at a syntax
>>>> level. So,
>>>> at least if you want good performance, you need to change your way of
>>>> thinking enough that a new syntax (loosely cooperating threads rather
>>>> than
>>>> parallel-for-loop or SIMD instruction) is actually an advantage, as
>>>> it keeps
>>>> you reminded of how the hardware works.
>>>>
>>>> So I think the most important thing to do (if you bother) is: Gather
>>>> a set
>>>> of real worl(-ish) CUDA or OpenCL programs, port them to Cython +
>>>> this CEP
>>>> (without a working Cython implementation for it), and see how that goes.
>>>> That's really the only way to evaluate it.
>>>
>>>
>>> I've been wanting to do that for a long time now, also to evaluate the
>>> capabilities of cython.parallel as it stands now. It's a really good
>>> idea, I'll try to port some codes, and not just the trivial ones like
>>> Jacobi's method :).
>>>
>>>> Some experiences from the single instance GPU code I've written:
>>>>
>>>> - For starters I had to give up OpenCL and use CUDA to use all the 48 KB
>>>> available shared memory on Nvidia compute-capability-2.0 (perhaps I just
>>>> didn't find the OpenCL option for that). And increasing from 16 to 48 KB
>>>> allowed a fundamentally faster and qualitatively different algorithm
>>>> to be
>>>> used. But OpenCL vs. CUDA is kind of beside the point here....
>>>>
>>>> - When mucking about with various "obvious" ports of sequential code
>>>> to GPU
>>>> code, I got performance in the range of 5 to 20 GFLOP/s (out of 490
>>>> GFLOP/s
>>>> or so theoretical; NVidia Tesla M2050). When really understanding the
>>>> hardware, and making good use of the 48 KB of thread-shared memory, I
>>>> achieved 209 GFLOP/s, without really doing any microoptimization. I
>>>> don't
>>>> think the CEP includes any features for intra-thread communication, so
>>>> that's off the table.
>>>
>>>
>>> The CEP doesn't mention barriers (discussed earlier), but they should
>>> be supported, and __local memory (that's "shared memory" in CUDA terms
>>> right?) could be utilized using a more explicit scheme (or implicitly
>>> if the compiler is smart). The only issue with barriers is that with
>>> OpenCL you have multiple levels of synchronization, but barriers only
>>> work within the work group / thread block, whereas with openmp it
>>> works simply for all your threads. I think a global barrier would have
>>> to mean kernel termination + start of a new one, which could be hard
>>> to support depending on where it is placed in the code...
>>>
>>>> (My code is here:
>>>>
>>>>
>>>> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
>>>>
>>>>
>>>> Though it's badly documented and rush-for-deadline-quality; I plan to
>>>> polish
>>>> it up and publish it when I get time in autumn).
>>>>
>>>> I guess I mention this as the kind of computation your CEP definitely
>>>> does
>>>> NOT cover. That's probably OK, but one should figure out specifically
>>>> how
>>>> many usecases it does cover (in particular with no control over thread
>>>> blocks and intra-block communication). Is the CEP a 80%-solution, or a
>>>> 10%-solution?
>>>
>>>
>>> I haven't looked too carefully at the code, but a large portion is
>>> dedicated to a reduction right? What I don't see is how your reduction
>>> spans across multiple work-groups / thread blocks? Because
>>> __syncthreads should only sync stuff within a single block. The CEP
>
>
> Most of the time there's actually no explicit synchronization, but the code
> relies on all threads of a warp being on the same instruction in the
> scheduler. __synchtreads is then only used at the end of the reduction when
> all within-warp additions have been done. Calling __syncthreads at each step
> of the algorithm would have totally killed performance.
>
> Dag Sverre
>

Ah, clever. I don't think there's any way to figure out the warp size
with OpenCL, but maybe if the user specifies it in some way similar
optimizations can be made.

>>
>> There's no need to reduce across thread blocks because (conveniently
>> enough) there's 8000 independent computations to be performed with
>> different parameters. I simply used one thread block for each problem.
>>
>> It's basically a matrix-vector product where the matrix must be
>> generated on the fly columnwise (one entry can be generated from the
>> preceding two in the same column), but the summation is row-wise.
>>
>> And turns out that getting inter-thread sum-reduction to work well was
>> harder than I expected; a 32-by-32 matrix (needed since warps are 32
>> threads) is too big to fit in memory, but tree-reduction makes a lot of
>> the threads in a warp do nothing. So I ended up with a hybrid approach;
>> there's visual demo from page 49 onwards here:
>>
>> http://folk.uio.no/dagss/talk-gpusht.pdf
>>
>> Getting back to Cython, I'll admit that this form of inter-thread
>> reduction is quite generic, and that my specific problem could be solved
>> by basically coding a set of inter-thread reduction algorithms suitable
>> for different hardware into Cython.
>>
>>> didn't mention reductions, but they should be supported (I'm thinking
>>> multi-stage or sequential within the workgroup (whatever works
>>> better), followed by another kernel invocation if the result is
>>> needed).
>>
>>
>> Multiple kernel invocations for global barriers appear to be pretty
>> standard, and it's why OpenCL support queueing tasks with dependencies
>> etc.
>>
>>> As mentioned earlier in a different thread (on parallelism I think),
>>> reduction arrays (i.e. memoryviews or C arrays) as well as generally
>>> private arrays should be supported. An issue with that is that you
>>> can't really dedicate an array to each work item / thread (too much
>>> memory would be consumed).
>>>
>>> Again, declarations within blocks would solve many problems:
>>>
>>> cdef float[n] shared_by_work_group
>>> with parallel():
>>> cdef float[n] local_to_work_group
>>> for i in prange(...):
>>> cdef float[n] local_to_work_item
>>>
>>> For arrays, the reductions could be somewhat more explicit, where
>>> there is an explicit 'my_memoryview += my_local_scratch_data'. That
>>> should probably only be allowed for memory local to the work group.
>>>
>>> Anyway, I'll try porting some numerical codes to this scheme over the
>>> coming weeks and see what is missing and how it can be solved. I still
>>> believe it can all be made to work quite properly, without adjusting
>>> the language to fit the hardware model. The prange (and OpenMP) model
>>> look like sequential code, but they tell the compiler a lot, namely
>>> that each iteration is independent and could therefore be scheduled as
>>> a separate thread.
>>
>>
>> Again, good points.
>>
>> Dag
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Sat Feb 11 20:52:28 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Feb 2012 11:52:28 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
Message-ID: <CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>

All of Sage passes except for one test:

sage -t  devel/sage/sage/misc/sageinspect.py
**********************************************************************
File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
line 970:
    sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
Expected:
    ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
Got:
    ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
**********************************************************************
File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
line 973:
    sage: sage_getargspec(BooleanMonomialMonoid.gen)
Expected:
    ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
Got:
    ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
**********************************************************************
1 items had failures:
   2 of  31 in __main__.example_21
***Test Failed*** 2 failures.

Any ideas why this would have changed?


On Sun, Feb 5, 2012 at 2:31 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Sat, Feb 4, 2012 at 10:49 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/1/31 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Sat, Jan 28, 2012 at 8:05 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/1/26 Jason Grout <jason-sage at creativetrax.com>:
>>>>> On 1/25/12 11:39 AM, Robert Bradshaw wrote:
>>>>>>
>>>>>> install
>>>>>>
>>>>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>>>> by downloading it and running "sage -i cython-devel.spkg"
>>>>>
>>>>>
>>>>>
>>>>> In fact, you could just do
>>>>>
>>>>> sage -i
>>>>> https://sage.math.washington.edu:8091/hudson/view/ext-libs/job/sage-build/lastSuccessfulBuild/artifact/cython-devel.spkg
>>>>>
>>>>> and Sage will (at least, should) download it for you, so that's even one
>>>>> less step!
>>>>>
>>>>> Jason
>>>>>
>>>>
>>>> Thanks for detailed instruction! I've successfully built it.
>>>>
>>>> "sage -t -gdb ./...." doesn't work, is that a bug?
>>>>
>>>> vitja at mchome:~/Downloads/sage-4.8$ ./sage ?-t -gdb
>>>> devel/sage/sage/combinat/sf/macdonald.py
>>>> sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"
>>>> ********************************************************************************
>>>> Type r at the (gdb) prompt to run the doctests.
>>>> Type bt if there is a crash to see a traceback.
>>>> ********************************************************************************
>>>> gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>>>> starting cmd gdb --args python /home/vitja/.sage//tmp/macdonald_6182.py
>>>> ImportError: No module named site
>>>> ? ? ? ? [0.2 s]
>>>>
>>>> ----------------------------------------------------------------------
>>>> The following tests failed:
>>>>
>>>>
>>>> ? ? ? ?sage -t -gdb "devel/sage/sage/combinat/sf/macdonald.py"release
>>>> Total time for all tests: 0.2 seconds
>>>
>>> Yes, that's a bug.
>>>
>>>> I've found another way to run tests (using sage -sh and then direct
>>>> python ~/.sage/tmp/...py)
>>>>
>>>> So I found one of the problems. Here is minimal cython example:
>>>>
>>>> def foo(values):
>>>> ? ?return (0,)*len(values)
>>>> foo([1,2,3])
>>>>
>>>> len(values) somehow is passed as an integer to PyObject_Multiply()
>>>
>>> Yeah, that's a bug too :).
>>
>> I've fixed tuple mult_factor bug here:
>>
>> https://github.com/cython/cython/commit/2d4b85dbcef885fbdaf6a3b2daef7a017184a56f
>>
>> No more segfaults in sage-tests but still 7 errors.
>>
>
> Thanks! I've looked into the other errors and I think it boils down to
> our use of --disable-function-redefinition being incompatible with how
> decorators work. Of course that was just a hack, so I've fixed sage to
> build/startup without using that flag, but there's some strangeness
> with import order now that I haven't had time to resolve yet.

From vitja.makarov at gmail.com  Sun Feb 12 07:45:46 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 12 Feb 2012 10:45:46 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
Message-ID: <CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>

2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
> All of Sage passes except for one test:
>
> sage -t ?devel/sage/sage/misc/sageinspect.py
> **********************************************************************
> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
> line 970:
> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
> Expected:
> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
> Got:
> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
> **********************************************************************
> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
> line 973:
> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
> Expected:
> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
> Got:
> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
> **********************************************************************
> 1 items had failures:
> ? 2 of ?31 in __main__.example_21
> ***Test Failed*** 2 failures.
>
> Any ideas why this would have changed?
>

CyFunction now provides its own code object. So inspect.getargs() is
called instead of
inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
seems like func.func_defaults should be implemented.


-- 
vitja.

From vitja.makarov at gmail.com  Sun Feb 12 15:06:52 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 12 Feb 2012 18:06:52 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPRMWWMVbpWh8PH6M2csT8w1NAMiv44UHk7rro7V0jJLTw@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
	<4F21A92F.6050401@behnel.de>
	<CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>
	<CAKGHGPRMWWMVbpWh8PH6M2csT8w1NAMiv44UHk7rro7V0jJLTw@mail.gmail.com>
Message-ID: <CAKGHGPTtbnG8nEB-mUTb43neOxte3wTAGv2kqWpZssGVMi2Cmg@mail.gmail.com>

2012/2/4 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/1/26 mark florisson <markflorisson88 at gmail.com>:
>> On 26 January 2012 19:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 26.01.2012 20:15:
>>>> On 26 January 2012 18:53, Stefan Behnel wrote:
>>>>> mark florisson, 26.01.2012 16:20:
>>>>>> I think this problem can trivially be solved by creating a ProxyNode
>>>>>> that should never be replaced by any transform, but it's argument may
>>>>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>>>>> create your CloneNodes.
>>>>>
>>>>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
>>>>
>>>> It wouldn't be a replacement, merely an addition (an extra indirection).
>>>
>>> What I was trying to say was that a ProxyNode would always be required by a
>>> CloneNode, but I don't see where a ProxyNode would be needed outside of a
>>> CloneNode. So it seems rather redundant and I don't know if we need a
>>> separate node for it.
>>
>> Yes it would be needed only for that, but I think the only real
>> alternative is to not use CloneNode at all, i.e. make the
>> transformation Dag mentioned, where you create new rhs (NameNode?)
>> references to the temporary result.
>>
>
> Now it seems to be the only case when we got problem like this. It
> means that clones may be safely created at very late stage.
> So transforming CascadeAssignment into SingleAssignments doesn't solve
> generic problem.
>
> I tried to implement conditional inlining the same problem may happen
> there (ConditionalCallNode owns arguments and replaces
> SimpleCallNode's args with clones). Splitting analyse_expressions()
> would help. On the other hand moving this optimization after
> OptimizeBuiltinCalls() would help too.
>

I tried to introduce finalize_expressions() here:

https://github.com/vitek/cython/tree/_finalize_expressions

I moved arg_tuple creation logic from SimpleCallNode's analyse_types()
to finalize_expressions() so few tests are broken now.

Now inlining is done right before AnalyseExpressions before arg_tuple
is created (before pyobject coercion nodes are created). It must be
run after expression analysis. So I'm completely sure that
analyse_types() must be split.

-- 
vitja.

From markflorisson88 at gmail.com  Sun Feb 12 15:35:01 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 12 Feb 2012 14:35:01 +0000
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CAKGHGPTtbnG8nEB-mUTb43neOxte3wTAGv2kqWpZssGVMi2Cmg@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
	<4F21A92F.6050401@behnel.de>
	<CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>
	<CAKGHGPRMWWMVbpWh8PH6M2csT8w1NAMiv44UHk7rro7V0jJLTw@mail.gmail.com>
	<CAKGHGPTtbnG8nEB-mUTb43neOxte3wTAGv2kqWpZssGVMi2Cmg@mail.gmail.com>
Message-ID: <CANg26EUCU84LPXMKDrnS21yZhqCErce=BQ1UjDZPhnnxY5zRxA@mail.gmail.com>

On 12 February 2012 14:06, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/4 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/1/26 mark florisson <markflorisson88 at gmail.com>:
>>> On 26 January 2012 19:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 26.01.2012 20:15:
>>>>> On 26 January 2012 18:53, Stefan Behnel wrote:
>>>>>> mark florisson, 26.01.2012 16:20:
>>>>>>> I think this problem can trivially be solved by creating a ProxyNode
>>>>>>> that should never be replaced by any transform, but it's argument may
>>>>>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>>>>>> create your CloneNodes.
>>>>>>
>>>>>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
>>>>>
>>>>> It wouldn't be a replacement, merely an addition (an extra indirection).
>>>>
>>>> What I was trying to say was that a ProxyNode would always be required by a
>>>> CloneNode, but I don't see where a ProxyNode would be needed outside of a
>>>> CloneNode. So it seems rather redundant and I don't know if we need a
>>>> separate node for it.
>>>
>>> Yes it would be needed only for that, but I think the only real
>>> alternative is to not use CloneNode at all, i.e. make the
>>> transformation Dag mentioned, where you create new rhs (NameNode?)
>>> references to the temporary result.
>>>
>>
>> Now it seems to be the only case when we got problem like this. It
>> means that clones may be safely created at very late stage.
>> So transforming CascadeAssignment into SingleAssignments doesn't solve
>> generic problem.
>>
>> I tried to implement conditional inlining the same problem may happen
>> there (ConditionalCallNode owns arguments and replaces
>> SimpleCallNode's args with clones). Splitting analyse_expressions()
>> would help. On the other hand moving this optimization after
>> OptimizeBuiltinCalls() would help too.
>>
>
> I tried to introduce finalize_expressions() here:
>
> https://github.com/vitek/cython/tree/_finalize_expressions
>
> I moved arg_tuple creation logic from SimpleCallNode's analyse_types()
> to finalize_expressions() so few tests are broken now.
>
> Now inlining is done right before AnalyseExpressions before arg_tuple
> is created (before pyobject coercion nodes are created). It must be
> run after expression analysis. So I'm completely sure that
> analyse_types() must be split.

Ah, I didn't realize you were working on that, I fixed the cascaded
assignment bug and pushed it to master a while ago. Anyway, if this
fixes the problem for cascaded assignment, feel free to do a hard
reset to remove those commits (probably want to keep the added tests
though).

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Sun Feb 12 16:15:23 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 12 Feb 2012 19:15:23 +0400
Subject: [Cython] Bug in Cython producing incorrect C code
In-Reply-To: <CANg26EUCU84LPXMKDrnS21yZhqCErce=BQ1UjDZPhnnxY5zRxA@mail.gmail.com>
References: <1327405070.15017.140661027320813@webmail.messagingengine.com>
	<CANg26EW8Nm30==phsW+_Mpza=k9+m=yQud5V0P1Wg3nFUvv3TA@mail.gmail.com>
	<4F1FB21A.9080407@behnel.de>
	<CAKGHGPQAjb0P+8c-PV70H1652OR2gwUAUiubsmpGnkQoJpqn-Q@mail.gmail.com>
	<CANg26EU=pQtKwuj6YoyEerVvWjhW=xmXNUecftf6yu4X1ko+xw@mail.gmail.com>
	<4F21A112.30803@behnel.de>
	<CANg26EVNzYU9LUDhoi=-vRyahptsw-BoBuj4yeOJR=VSL1WSSQ@mail.gmail.com>
	<4F21A92F.6050401@behnel.de>
	<CANg26EUay_ZK4m0rg-g-jCebscSy2G_qT5QQiBfWhTK+h_543g@mail.gmail.com>
	<CAKGHGPRMWWMVbpWh8PH6M2csT8w1NAMiv44UHk7rro7V0jJLTw@mail.gmail.com>
	<CAKGHGPTtbnG8nEB-mUTb43neOxte3wTAGv2kqWpZssGVMi2Cmg@mail.gmail.com>
	<CANg26EUCU84LPXMKDrnS21yZhqCErce=BQ1UjDZPhnnxY5zRxA@mail.gmail.com>
Message-ID: <CAKGHGPRigOk3K2T=TMsSBu7pEeBRH0X3wgw8ysA=RpfusmcjCA@mail.gmail.com>

2012/2/12 mark florisson <markflorisson88 at gmail.com>:
> On 12 February 2012 14:06, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/4 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/1/26 mark florisson <markflorisson88 at gmail.com>:
>>>> On 26 January 2012 19:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>> mark florisson, 26.01.2012 20:15:
>>>>>> On 26 January 2012 18:53, Stefan Behnel wrote:
>>>>>>> mark florisson, 26.01.2012 16:20:
>>>>>>>> I think this problem can trivially be solved by creating a ProxyNode
>>>>>>>> that should never be replaced by any transform, but it's argument may
>>>>>>>> be replaced. So you wrap self.rhs in a ProxyNode and use that to
>>>>>>>> create your CloneNodes.
>>>>>>>
>>>>>>> I can't see what a ProxyNode would do that a CloneNode shouldn't do anyway.
>>>>>>
>>>>>> It wouldn't be a replacement, merely an addition (an extra indirection).
>>>>>
>>>>> What I was trying to say was that a ProxyNode would always be required by a
>>>>> CloneNode, but I don't see where a ProxyNode would be needed outside of a
>>>>> CloneNode. So it seems rather redundant and I don't know if we need a
>>>>> separate node for it.
>>>>
>>>> Yes it would be needed only for that, but I think the only real
>>>> alternative is to not use CloneNode at all, i.e. make the
>>>> transformation Dag mentioned, where you create new rhs (NameNode?)
>>>> references to the temporary result.
>>>>
>>>
>>> Now it seems to be the only case when we got problem like this. It
>>> means that clones may be safely created at very late stage.
>>> So transforming CascadeAssignment into SingleAssignments doesn't solve
>>> generic problem.
>>>
>>> I tried to implement conditional inlining the same problem may happen
>>> there (ConditionalCallNode owns arguments and replaces
>>> SimpleCallNode's args with clones). Splitting analyse_expressions()
>>> would help. On the other hand moving this optimization after
>>> OptimizeBuiltinCalls() would help too.
>>>
>>
>> I tried to introduce finalize_expressions() here:
>>
>> https://github.com/vitek/cython/tree/_finalize_expressions
>>
>> I moved arg_tuple creation logic from SimpleCallNode's analyse_types()
>> to finalize_expressions() so few tests are broken now.
>>
>> Now inlining is done right before AnalyseExpressions before arg_tuple
>> is created (before pyobject coercion nodes are created). It must be
>> run after expression analysis. So I'm completely sure that
>> analyse_types() must be split.
>
> Ah, I didn't realize you were working on that, I fixed the cascaded
> assignment bug and pushed it to master a while ago. Anyway, if this
> fixes the problem for cascaded assignment, feel free to do a hard
> reset to remove those commits (probably want to keep the added tests
> though).
>

Nice, I'm ok with your fix! Btw finalize_expressions() isn't done yet.
It's a major change so it's better to wait until next release.

-- 
vitja.

From vitja.makarov at gmail.com  Sun Feb 12 21:53:35 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 13 Feb 2012 00:53:35 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
Message-ID: <CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>

2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>> All of Sage passes except for one test:
>>
>> sage -t ?devel/sage/sage/misc/sageinspect.py
>> **********************************************************************
>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>> line 970:
>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>> Expected:
>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>> Got:
>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>> **********************************************************************
>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>> line 973:
>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>> Expected:
>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>> Got:
>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>> **********************************************************************
>> 1 items had failures:
>> ? 2 of ?31 in __main__.example_21
>> ***Test Failed*** 2 failures.
>>
>> Any ideas why this would have changed?
>>
>
> CyFunction now provides its own code object. So inspect.getargs() is
> called instead of
> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
> seems like func.func_defaults should be implemented.
>
>

I've created a pull request:

https://github.com/cython/cython/pull/88

-- 
vitja.

From robertwb at math.washington.edu  Tue Feb 14 08:07:11 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 13 Feb 2012 23:07:11 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
Message-ID: <CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>

On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>> All of Sage passes except for one test:
>>>
>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>> **********************************************************************
>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>> line 970:
>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>> Expected:
>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>> Got:
>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>> **********************************************************************
>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>> line 973:
>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>> Expected:
>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>> Got:
>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>> **********************************************************************
>>> 1 items had failures:
>>> ? 2 of ?31 in __main__.example_21
>>> ***Test Failed*** 2 failures.
>>>
>>> Any ideas why this would have changed?
>>>
>>
>> CyFunction now provides its own code object. So inspect.getargs() is
>> called instead of
>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>> seems like func.func_defaults should be implemented.
>>
>>
>
> I've created a pull request:
>
> https://github.com/cython/cython/pull/88

Thanks! The only other thing I can think of was a question of using
caching to mitigate the longer compile times, but I can't remember if
this was resolved.

As I'm going to be MIA any day now, someone else should take up the
banner to push this long awaited release.

- Robert

From stefan_ml at behnel.de  Tue Feb 14 09:20:01 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Feb 2012 09:20:01 +0100
Subject: [Cython] Cython compatibility thread on PyPy mailing list
Message-ID: <4F3A1931.6060505@behnel.de>

Hi,

just wanted to point you to a thread that currently runs on the PyPy-dev
mailing list about how Cython and PyPy could improve their interoperability.

http://thread.gmane.org/gmane.comp.python.pypy/9437/focus=9452

Stefan

From markflorisson88 at gmail.com  Tue Feb 14 16:49:01 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 14 Feb 2012 15:49:01 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
Message-ID: <CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>

On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>> All of Sage passes except for one test:
>>>>
>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>> **********************************************************************
>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>> line 970:
>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>> Expected:
>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>> Got:
>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>> **********************************************************************
>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>> line 973:
>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>> Expected:
>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>> Got:
>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>> **********************************************************************
>>>> 1 items had failures:
>>>> ? 2 of ?31 in __main__.example_21
>>>> ***Test Failed*** 2 failures.
>>>>
>>>> Any ideas why this would have changed?
>>>>
>>>
>>> CyFunction now provides its own code object. So inspect.getargs() is
>>> called instead of
>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>> seems like func.func_defaults should be implemented.
>>>
>>>
>>
>> I've created a pull request:
>>
>> https://github.com/cython/cython/pull/88
>
> Thanks! The only other thing I can think of was a question of using
> caching to mitigate the longer compile times, but I can't remember if
> this was resolved.

The compiler has like 2 or 3 seconds of constant overhead if you use
memoryviews.

> As I'm going to be MIA any day now, someone else should take up the
> banner to push this long awaited release.

"Missing in action"? Are you planning to desert? :) I can't find any
relevant abbreviation, but I think I know what it means,
congratulations in advance.
Stefan, you have been involved the longest, would you feel up to the
task? You probably have the best understanding and experience with any
issues (no pressure :). Otherwise I could have a try...

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Tue Feb 14 18:19:16 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Feb 2012 09:19:16 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
Message-ID: <CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>

On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>> All of Sage passes except for one test:
>>>>>
>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>> **********************************************************************
>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>> line 970:
>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>> Expected:
>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>> Got:
>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>> **********************************************************************
>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>> line 973:
>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>> Expected:
>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>> Got:
>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>> **********************************************************************
>>>>> 1 items had failures:
>>>>> ? 2 of ?31 in __main__.example_21
>>>>> ***Test Failed*** 2 failures.
>>>>>
>>>>> Any ideas why this would have changed?
>>>>>
>>>>
>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>> called instead of
>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>> seems like func.func_defaults should be implemented.
>>>>
>>>>
>>>
>>> I've created a pull request:
>>>
>>> https://github.com/cython/cython/pull/88
>>
>> Thanks! The only other thing I can think of was a question of using
>> caching to mitigate the longer compile times, but I can't remember if
>> this was resolved.
>
> The compiler has like 2 or 3 seconds of constant overhead if you use
> memoryviews.

That'd be nice to cut down, but certainly not a blocker.

>> As I'm going to be MIA any day now, someone else should take up the
>> banner to push this long awaited release.
>
> "Missing in action"? Are you planning to desert? :) I can't find any
> relevant abbreviation, but I think I know what it means,
> congratulations in advance.

Twin boys coming any day now!

> Stefan, you have been involved the longest, would you feel up to the
> task? You probably have the best understanding and experience with any
> issues (no pressure :). Otherwise I could have a try...

It's pretty easy. Once the defaults change is in it's probably worth
cutting a beta or release candidate to email to dev/users, and if
there's no blocking feedback you go ahead and push it out (basically
writing up the release notes on the wiki, cleaning up trac, tagging
the repository, making sure everything we care about on hudson is
still passing, uploading to pypi and the website (the sdist tarball),
emailing our lists and python-announce, re-building and updating the
pointer to the documentation, ...) If it goes on for a while it's
worth making/using a release branch on github.

- Robert

From markflorisson88 at gmail.com  Tue Feb 14 22:09:06 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 14 Feb 2012 21:09:06 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
Message-ID: <CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>

On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>> All of Sage passes except for one test:
>>>>>>
>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>> **********************************************************************
>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>> line 970:
>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>> Expected:
>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>> Got:
>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>> **********************************************************************
>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>> line 973:
>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>> Expected:
>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>> Got:
>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>> **********************************************************************
>>>>>> 1 items had failures:
>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>> ***Test Failed*** 2 failures.
>>>>>>
>>>>>> Any ideas why this would have changed?
>>>>>>
>>>>>
>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>> called instead of
>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>> seems like func.func_defaults should be implemented.
>>>>>
>>>>>
>>>>
>>>> I've created a pull request:
>>>>
>>>> https://github.com/cython/cython/pull/88
>>>
>>> Thanks! The only other thing I can think of was a question of using
>>> caching to mitigate the longer compile times, but I can't remember if
>>> this was resolved.
>>
>> The compiler has like 2 or 3 seconds of constant overhead if you use
>> memoryviews.
>
> That'd be nice to cut down, but certainly not a blocker.
>
>>> As I'm going to be MIA any day now, someone else should take up the
>>> banner to push this long awaited release.
>>
>> "Missing in action"? Are you planning to desert? :) I can't find any
>> relevant abbreviation, but I think I know what it means,
>> congratulations in advance.
>
> Twin boys coming any day now!

And the Cython team just keeps on growing!

>> Stefan, you have been involved the longest, would you feel up to the
>> task? You probably have the best understanding and experience with any
>> issues (no pressure :). Otherwise I could have a try...
>
> It's pretty easy. Once the defaults change is in it's probably worth
> cutting a beta or release candidate to email to dev/users, and if
> there's no blocking feedback you go ahead and push it out (basically
> writing up the release notes on the wiki, cleaning up trac, tagging
> the repository, making sure everything we care about on hudson is
> still passing, uploading to pypi and the website (the sdist tarball),
> emailing our lists and python-announce, re-building and updating the
> pointer to the documentation, ...) If it goes on for a while it's
> worth making/using a release branch on github.

Thanks for the summary, I'm sure I would have missed one or two :) Ok,
I'll volunteer then. Maybe I can create a beta somewhere next week and
then we can see the community tear it apart.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Tue Feb 14 22:33:12 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Feb 2012 13:33:12 -0800
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
Message-ID: <CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>

On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>> All of Sage passes except for one test:
>>>>>>>
>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>> **********************************************************************
>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>> line 970:
>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>> Expected:
>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>> Got:
>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>> **********************************************************************
>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>> line 973:
>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>> Expected:
>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>> Got:
>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>> **********************************************************************
>>>>>>> 1 items had failures:
>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>
>>>>>>> Any ideas why this would have changed?
>>>>>>>
>>>>>>
>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>> called instead of
>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>> seems like func.func_defaults should be implemented.
>>>>>>
>>>>>>
>>>>>
>>>>> I've created a pull request:
>>>>>
>>>>> https://github.com/cython/cython/pull/88
>>>>
>>>> Thanks! The only other thing I can think of was a question of using
>>>> caching to mitigate the longer compile times, but I can't remember if
>>>> this was resolved.
>>>
>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>> memoryviews.
>>
>> That'd be nice to cut down, but certainly not a blocker.
>>
>>>> As I'm going to be MIA any day now, someone else should take up the
>>>> banner to push this long awaited release.
>>>
>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>> relevant abbreviation, but I think I know what it means,
>>> congratulations in advance.
>>
>> Twin boys coming any day now!
>
> And the Cython team just keeps on growing!

:)

>>> Stefan, you have been involved the longest, would you feel up to the
>>> task? You probably have the best understanding and experience with any
>>> issues (no pressure :). Otherwise I could have a try...
>>
>> It's pretty easy. Once the defaults change is in it's probably worth
>> cutting a beta or release candidate to email to dev/users, and if
>> there's no blocking feedback you go ahead and push it out (basically
>> writing up the release notes on the wiki, cleaning up trac, tagging
>> the repository, making sure everything we care about on hudson is
>> still passing, uploading to pypi and the website (the sdist tarball),
>> emailing our lists and python-announce, re-building and updating the
>> pointer to the documentation, ...) If it goes on for a while it's
>> worth making/using a release branch on github.
>
> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
> I'll volunteer then. Maybe I can create a beta somewhere next week and
> then we can see the community tear it apart.

Thanks!

- Robert

From stefan_ml at behnel.de  Wed Feb 15 10:37:20 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Feb 2012 10:37:20 +0100
Subject: [Cython] overwriting pull requests and the related discussion
Message-ID: <4F3B7CD0.5010804@behnel.de>

Hi,

I'd like to suggest that instead of overwriting pull requests and all of
their comments on github by pushing replaced commits over them, it would be
better to keep any existing discussions accessible by rejecting the current
pull request and creating a new one.

Does that make sense for everyone?

Stefan

From markflorisson88 at gmail.com  Wed Feb 15 10:54:20 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 15 Feb 2012 09:54:20 +0000
Subject: [Cython] overwriting pull requests and the related discussion
In-Reply-To: <4F3B7CD0.5010804@behnel.de>
References: <4F3B7CD0.5010804@behnel.de>
Message-ID: <CANg26EUUHVLti0Ft=t6-j7TKusrawYNCp6DqGtaFy3=UA5po=g@mail.gmail.com>

On 15 February 2012 09:37, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I'd like to suggest that instead of overwriting pull requests and all of
> their comments on github by pushing replaced commits over them, it would be
> better to keep any existing discussions accessible by rejecting the current
> pull request and creating a new one.
>
> Does that make sense for everyone?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Hm, rebasing + force pushing is very common for pull requests, does
that delete the inline (or all) comments?

From vitja.makarov at gmail.com  Wed Feb 15 11:02:03 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 15 Feb 2012 14:02:03 +0400
Subject: [Cython] overwriting pull requests and the related discussion
In-Reply-To: <CANg26EUUHVLti0Ft=t6-j7TKusrawYNCp6DqGtaFy3=UA5po=g@mail.gmail.com>
References: <4F3B7CD0.5010804@behnel.de>
	<CANg26EUUHVLti0Ft=t6-j7TKusrawYNCp6DqGtaFy3=UA5po=g@mail.gmail.com>
Message-ID: <CAKGHGPQ-5edkLR2UjPOv4Bk2CjEn=huRHb4eH=0sAD1dvZQuug@mail.gmail.com>

2012/2/15 mark florisson <markflorisson88 at gmail.com>:
> On 15 February 2012 09:37, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> I'd like to suggest that instead of overwriting pull requests and all of
>> their comments on github by pushing replaced commits over them, it would be
>> better to keep any existing discussions accessible by rejecting the current
>> pull request and creating a new one.
>>
>> Does that make sense for everyone?
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Hm, rebasing + force pushing is very common for pull requests, does
> that delete the inline (or all) comments?

Yeah, vanished comments are really annoying on the other hand it's
really hard to avoid rebases and forced pusing.
Forced pushing (via git rebase -i HEAD~N) is rather useful to cleanup
commits: fix typos, obvious bugs and so on.



-- 
vitja.

From stefan_ml at behnel.de  Wed Feb 15 12:35:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Feb 2012 12:35:35 +0100
Subject: [Cython] Bringing Cython and PyPy closer together
Message-ID: <4F3B9887.8020108@behnel.de>

Hi,

following up on the thread on the PyPy mailing list where this topic was
started, I've started a CEP in our Wiki in order to focus the different
ideas and opinions.

http://wiki.cython.org/enhancements/pypy

The current state of the discussion seems to be that PyPy provides ways to
talk to C code, but nothing as complete as CPython's C-API in the sense
that it allows efficient two-way communication between C code and Python
objects. Thus, we need to either improve this or look for alternatives.

Please add to the CEP as you see fit.

Stefan

From markflorisson88 at gmail.com  Wed Feb 15 16:45:33 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 15 Feb 2012 15:45:33 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
Message-ID: <CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>

On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>> All of Sage passes except for one test:
>>>>>>>>
>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>> **********************************************************************
>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>> line 970:
>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>> Expected:
>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>> Got:
>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>> **********************************************************************
>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>> line 973:
>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>> Expected:
>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>> Got:
>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>> **********************************************************************
>>>>>>>> 1 items had failures:
>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>
>>>>>>>> Any ideas why this would have changed?
>>>>>>>>
>>>>>>>
>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>> called instead of
>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I've created a pull request:
>>>>>>
>>>>>> https://github.com/cython/cython/pull/88
>>>>>
>>>>> Thanks! The only other thing I can think of was a question of using
>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>> this was resolved.
>>>>
>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>> memoryviews.
>>>
>>> That'd be nice to cut down, but certainly not a blocker.
>>>
>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>> banner to push this long awaited release.
>>>>
>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>> relevant abbreviation, but I think I know what it means,
>>>> congratulations in advance.
>>>
>>> Twin boys coming any day now!
>>
>> And the Cython team just keeps on growing!
>
> :)
>
>>>> Stefan, you have been involved the longest, would you feel up to the
>>>> task? You probably have the best understanding and experience with any
>>>> issues (no pressure :). Otherwise I could have a try...
>>>
>>> It's pretty easy. Once the defaults change is in it's probably worth
>>> cutting a beta or release candidate to email to dev/users, and if
>>> there's no blocking feedback you go ahead and push it out (basically
>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>> the repository, making sure everything we care about on hudson is
>>> still passing, uploading to pypi and the website (the sdist tarball),
>>> emailing our lists and python-announce, re-building and updating the
>>> pointer to the documentation, ...) If it goes on for a while it's
>>> worth making/using a release branch on github.
>>
>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>> then we can see the community tear it apart.
>
> Thanks!
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Sorry, my previous email with attachment bounced. Here goes.

I'm getting a substantial amount of failing tests on MSVC,
https://gist.github.com/1836766. I think most complex number tests are
failing because they cast
a struct of a certain type to itself like ((struct_A) my_struct_A),
which MSVC doesn't allow.

Some tests seem to fail because they can't be imported: "compiling (c)
and running numpy_parallel: ImportError: No module named
numpy_parallel".

And then there is a huge number of permission errors: WindowsError:
[Error 5] Access is denied:
'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
something is broken in the test runner (or in my setup somehow)?

From markflorisson88 at gmail.com  Wed Feb 15 16:47:57 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 15 Feb 2012 15:47:57 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
Message-ID: <CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>

On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>
>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>> **********************************************************************
>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>> line 970:
>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>> Expected:
>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>> Got:
>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>> **********************************************************************
>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>> line 973:
>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>> Expected:
>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>> Got:
>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>> **********************************************************************
>>>>>>>>> 1 items had failures:
>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>
>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>
>>>>>>>>
>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>> called instead of
>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> I've created a pull request:
>>>>>>>
>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>
>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>> this was resolved.
>>>>>
>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>> memoryviews.
>>>>
>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>
>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>> banner to push this long awaited release.
>>>>>
>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>> relevant abbreviation, but I think I know what it means,
>>>>> congratulations in advance.
>>>>
>>>> Twin boys coming any day now!
>>>
>>> And the Cython team just keeps on growing!
>>
>> :)
>>
>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>> task? You probably have the best understanding and experience with any
>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>
>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>> cutting a beta or release candidate to email to dev/users, and if
>>>> there's no blocking feedback you go ahead and push it out (basically
>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>> the repository, making sure everything we care about on hudson is
>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>> emailing our lists and python-announce, re-building and updating the
>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>> worth making/using a release branch on github.
>>>
>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>> then we can see the community tear it apart.
>>
>> Thanks!
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Sorry, my previous email with attachment bounced. Here goes.
>
> I'm getting a substantial amount of failing tests on MSVC,
> https://gist.github.com/1836766. I think most complex number tests are
> failing because they cast
> a struct of a certain type to itself like ((struct_A) my_struct_A),
> which MSVC doesn't allow.
>
> Some tests seem to fail because they can't be imported: "compiling (c)
> and running numpy_parallel: ImportError: No module named
> numpy_parallel".
>
> And then there is a huge number of permission errors: WindowsError:
> [Error 5] Access is denied:
> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
> something is broken in the test runner (or in my setup somehow)?

The pasted output is a little munged because it was redirected to a
log (and stdout is probably block buffering, something we could also
fix to line buffering).

From markflorisson88 at gmail.com  Thu Feb 16 19:49:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 16 Feb 2012 18:49:28 +0000
Subject: [Cython] Cython Library
Message-ID: <CANg26EVug23jJnf=Szwy1TvaM25kG0SspJNNGGs3vJNvzAdcrw@mail.gmail.com>

Hey,

I created a CEP for the optional ability to have Cython create a
version-dependent library to share types and common utility functions
internal to the compiler, you can find it here:
http://wiki.cython.org/enhancements/libcython . It also addresses
entry caching, both for pxd files and Cython utility codes.

Mark

From fperez.net at gmail.com  Mon Feb 13 22:55:45 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 13 Feb 2012 13:55:45 -0800
Subject: [Cython] Discussion with Guido van Rossum and (hopefully) core
 python-dev on scientific Python and Python3
Message-ID: <CAHAreOrWczOVZ7fR-WazzTSuzXJqiyk-CCotOdhMR53AY0tzJA@mail.gmail.com>

Hi folks,

[ I'm broadcasting this widely for maximum reach, but I'd appreciate
it if replies can be kept to the *numpy* list, which is sort of the
'base' list for scientific/numerical work.  It will make it much
easier to organize a coherent set of notes later on.  Apology if
you're subscribed to all and get it 10 times. ]

As part of the PyData workshop (http://pydataworkshop.eventbrite.com)
to be held March 2 and 3 at the Mountain View Google offices, we have
scheduled a session for an open discussion with Guido van Rossum and
hopefully as many core python-dev members who can make it.  We wanted
to seize the combined opportunity of the PyData workshop bringing a
number of 'scipy people' to Google with the timeline for Python 3.3,
the first release after the Python language moratorium, being within
sight: http://www.python.org/dev/peps/pep-0398.

While a number of scientific Python packages are already available for
Python 3 (either in released form or in their master git branches),
it's fair to say that there hasn't been a major transition of the
scientific community to Python3.  Since there is no more development
being done on the Python2 series, eventually we will all want to find
ways to make this transition, and we think that this is an excellent
time to engage the core python development team and consider ideas
that would make Python3 generally a more appealing language for
scientific work.  Guido has made it clear that he doesn't speak for
the day-to-day development of Python anymore, so we all should be
aware that any ideas that come out of this panel will still need to be
discussed with python-dev itself via standard mechanisms before
anything is implemented.  Nonetheless, the opportunity for a solid
face-to-face dialog for brainstorming was too good to pass up.

The purpose of this email is then to solicit, from all of our
community, ideas for this discussion.  In a week or so we'll need to
summarize the main points brought up here and make a more concrete
agenda out of it; I will also post a summary of the meeting afterwards
here.

Anything is a valid topic, some points just to get the conversation started:

- Extra operators/PEP 225.  Here's a summary from the last time we
went over this, years ago at Scipy 2008:
http://mail.scipy.org/pipermail/numpy-discussion/2008-October/038234.html,
and the current status of the document we wrote about it is here:
file:///home/fperez/www/site/_build/html/py4science/numpy-pep225/numpy-pep225.html.

- Improved syntax/support for rationals or decimal literals?  While
Python now has both decimals
(http://docs.python.org/library/decimal.html) and rationals
(http://docs.python.org/library/fractions.html), they're quite clunky
to use because they require full constructor calls.  Guido has
mentioned in previous discussions toying with ideas about support for
different kinds of numeric literals...

- Using the numpy docstring standard python-wide, and thus having
python improve the pathetic state of the stdlib's docstrings?  This is
an area where our community is light years ahead of the standard
library, but we'd all benefit from Python itself improving on this
front.  I'm toying with the idea of giving a lighting talk at PyConn
about this, comparing the great, robust culture and tools of good
docstrings across the Scipy ecosystem with the sad, sad state of
docstrings in the stdlib.  It might spur some movement on that front
from the stdlib authors, esp. if the core python-dev team realizes the
value and benefit it can bring (at relatively low cost, given how most
of the information does exist, it's just in the wrong places).  But
more importantly for us, if there was truly a universal standard for
high-quality docstrings across Python projects, building good
documentation/help machinery would be a lot easier, as we'd know what
to expect and search for (such as rendering them nicely in the ipython
notebook, providing high-quality cross-project help search, etc).

- Literal syntax for arrays?  Sage has been floating a discussion
about a literal matrix syntax
(https://groups.google.com/forum/#!topic/sage-devel/mzwepqZBHnA).  For
something like this to go into python in any meaningful way there
would have to be core multidimensional arrays in the language, but
perhaps it's time to think about a piece of the numpy array itself
into Python?  This is one of the more 'out there' ideas, but after
all, that's the point of a discussion like this, especially
considering we'll have both Travis and Guido in one room.

- Other syntactic sugar? Sage has "a..b" <=> range(a, b+1), which I
actually think is  both nice and useful... There's also the question
of allowing "a:b:c" notation outside of [], which has come up a few
times in conversation over the last few years. Others?

- The packaging quagmire?  This continues to be a problem, though
python3 does have new improvements to distutils.  I'm not really up to
speed on the situation, to be frank.  If we want to bring this up,
someone will have to provide a solid reference or volunteer to do it
in person.

- etc...

I'm putting the above just to *start* the discussion, but the real
point is for the rest of the community to contribute ideas, so don't
be shy.

Final note: while I am here commiting to organizing and presenting
this at the discussion with Guido (as well as contacting python-dev),
I would greatly appreciate help with the task of summarizing this
prior to the meeting as I'm pretty badly swamped in the run-in to
pydata/pycon.  So if anyone is willing to help draft the summary as
the date draws closer (we can put it up on a github wiki, gist,
whatever), I will be very grateful.  I'm sure it will be better than
what I'll otherwise do the last night at 2am :)

Cheers,

f

ps - to the obvious question about webcasting the discussion live for
remote participation: yes, we looked into it already; no,
unfortunately it appears it won't be possible.  We'll try to at least
have the audio recorded (and possibly video) for posting later on.

pps- if you are close to Mountain View and are interested in attending
this panel in person, drop me a line at fernando.perez at berkeley.edu.
We have a few spots available *for this discussion only* on top of the
pydata regular attendance (which is long closed, I'm afraid).  But
we'll need to provide Google with a list of those attendees in
advance.  Please indicate if you are a core python committer in your
email, as we'll give priority for this overflow pool to core python
developers (but will otherwise accommodate as many people as Google
lets us).

From stefan_ml at behnel.de  Sat Feb 18 09:54:11 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 18 Feb 2012 09:54:11 +0100
Subject: [Cython] Bringing Cython and PyPy closer together
In-Reply-To: <jhg54k$dte$1@dough.gmane.org>
References: <jhg54k$dte$1@dough.gmane.org>
Message-ID: <4F3F6733.3070609@behnel.de>

[copied here from PyPy mailing list]

Stefan Behnel, 15.02.2012 12:32:
> The current state of the discussion seems to be that PyPy provides ways to
> talk to C code, but nothing as complete as CPython's C-API in the sense
> that it allows efficient two-way communication between C code and Python
> objects. Thus, we need to either improve this or look for alternatives.
> 
> In order to get us more focussed on what can be done and what the
> implications are, so that we may eventually be able to decide what should
> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement
> Proposal).
> 
> http://wiki.cython.org/enhancements/pypy

The discussion so far makes me rather certain that the most promising
short-term solution is to make Cython generate C code that PyPy's cpyext
can handle. This should get us a rather broad set of running code somewhat
quickly, while requiring the least design-from-scratch type of work in a
direction that does not yet allow us to see if it will really make existing
code work or not.

On top of the basic cpyext interface, it should then be easy to implement
obvious optimisations like native C level calls to Cython wrapped functions
from PyPy (and potentially also the other direction) and otherwise avoid
boxing/unboxing where unnecessary, e.g. for builtins. After all, it all
boils down to native code at some point and I'm sure there are various ways
to exploit that.

Also, going this route will help both projects to get to know each other
better. I think that's a required basis if we really aim for designing a
more high-level interface at some point.

The first steps I see are:

- get Cython's test suite to run on PyPy
- analyse the failing tests and decide how to fix them
- adapt the Cython generated C code accordingly, special casing for PyPy
  where required

Here is a "getting started" guide that tells you how testing works in Cython:

http://wiki.cython.org/HackerGuide

Once we have the test suite runnable, we can set up a PyPy instance on our
CI server to get feed-back on any advances.

https://sage.math.washington.edu:8091/hudson/

So, any volunteers or otherwise interested parties to help in getting this
to work? Anyone in for financial support?

Stefan

From stefan_ml at behnel.de  Sat Feb 18 17:11:15 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 18 Feb 2012 17:11:15 +0100
Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer
	together
In-Reply-To: <4F3F6733.3070609@behnel.de>
References: <jhg54k$dte$1@dough.gmane.org> <4F3F6733.3070609@behnel.de>
Message-ID: <4F3FCDA3.7090102@behnel.de>

Stefan Behnel, 18.02.2012 09:54:
> Stefan Behnel, 15.02.2012 12:32:
>> The current state of the discussion seems to be that PyPy provides ways to
>> talk to C code, but nothing as complete as CPython's C-API in the sense
>> that it allows efficient two-way communication between C code and Python
>> objects. Thus, we need to either improve this or look for alternatives.
>>
>> In order to get us more focussed on what can be done and what the
>> implications are, so that we may eventually be able to decide what should
>> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement
>> Proposal).
>>
>> http://wiki.cython.org/enhancements/pypy
> 
> The discussion so far makes me rather certain that the most promising
> short-term solution is to make Cython generate C code that PyPy's cpyext
> can handle. This should get us a rather broad set of running code somewhat
> quickly, while requiring the least design-from-scratch type of work in a
> direction that does not yet allow us to see if it will really make existing
> code work or not.

Update:

Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run
(although with some clear bugs in the exception handling code). There is a
Jenkins job now to (try to) run the test suite of my own branch in the
latest PyPy nightly build:

https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/

It currently crashes rather badly at some point, but at least it looks like
it's actually getting somewhere.

Stefan

From vitja.makarov at gmail.com  Sun Feb 19 11:16:53 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 19 Feb 2012 14:16:53 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
Message-ID: <CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>

2012/2/15 mark florisson <markflorisson88 at gmail.com>:
> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>
>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>> **********************************************************************
>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>> line 970:
>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>> Expected:
>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>> Got:
>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>> **********************************************************************
>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>> line 973:
>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>> Expected:
>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>> Got:
>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>> **********************************************************************
>>>>>>>>>> 1 items had failures:
>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>
>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>> called instead of
>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> I've created a pull request:
>>>>>>>>
>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>
>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>> this was resolved.
>>>>>>
>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>> memoryviews.
>>>>>
>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>
>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>> banner to push this long awaited release.
>>>>>>
>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>> congratulations in advance.
>>>>>
>>>>> Twin boys coming any day now!
>>>>
>>>> And the Cython team just keeps on growing!
>>>
>>> :)
>>>
>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>> task? You probably have the best understanding and experience with any
>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>
>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>> the repository, making sure everything we care about on hudson is
>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>> emailing our lists and python-announce, re-building and updating the
>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>> worth making/using a release branch on github.
>>>>
>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>> then we can see the community tear it apart.
>>>
>>> Thanks!
>>>
>>> - Robert
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Sorry, my previous email with attachment bounced. Here goes.
>>
>> I'm getting a substantial amount of failing tests on MSVC,
>> https://gist.github.com/1836766. I think most complex number tests are
>> failing because they cast
>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>> which MSVC doesn't allow.
>>
>> Some tests seem to fail because they can't be imported: "compiling (c)
>> and running numpy_parallel: ImportError: No module named
>> numpy_parallel".
>>
>> And then there is a huge number of permission errors: WindowsError:
>> [Error 5] Access is denied:
>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>> something is broken in the test runner (or in my setup somehow)?
>
> The pasted output is a little munged because it was redirected to a
> log (and stdout is probably block buffering, something we could also
> fix to line buffering).

I've merged cydefaults branch and now sage-tests is blue.

-- 
vitja.

From vitja.makarov at gmail.com  Sun Feb 19 12:14:37 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 19 Feb 2012 15:14:37 +0400
Subject: [Cython] p3k pyregr tests problem
Message-ID: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>

Hi!

I've noticed problems with py3k pyregr tests now it shows ~8K tests
instead of 13K

Is that related to changes in cython or python?

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/


-- 
vitja.

From vitja.makarov at gmail.com  Sun Feb 19 12:48:11 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 19 Feb 2012 15:48:11 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
Message-ID: <CAKGHGPQJZZxuwxu1mFYF1drBs+t=dFVuyh5sVyiP87qR_wXERA@mail.gmail.com>

2012/2/19 Vitja Makarov <vitja.makarov at gmail.com>:
> Hi!
>
> I've noticed problems with py3k pyregr tests now it shows ~8K tests
> instead of 13K
>
> Is that related to changes in cython or python?
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>
>
> --
> vitja.

For instance, test_argparse had 1612 tests but now it shows only 1105.

I was unable to reproduce this behavior with same py3k-hg (Python
3.3.0a0 (default:a9f090728729, Feb 18 2012, 23:32:22 and current
master) package it shows 1612 testcases on localhost.

-- 
vitja.

From stefan_ml at behnel.de  Sun Feb 19 12:53:06 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Feb 2012 12:53:06 +0100
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
Message-ID: <4F40E2A2.8030907@behnel.de>

Vitja Makarov, 19.02.2012 12:14:
> I've noticed problems with py3k pyregr tests now it shows ~8K tests
> instead of 13K
> 
> Is that related to changes in cython or python?
> 
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/

At least I don't see any changes on our side that could trigger this:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792

And there doesn't seem to be anything obvious in the change list of CPython
either:

https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642

Stefan

From markflorisson88 at gmail.com  Sun Feb 19 15:14:20 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 19 Feb 2012 14:14:20 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
Message-ID: <CANg26EV2yR-C8ZYRh2_B4UmimHAzkUuKVBe3WdLzQ8tOHH-UjQ@mail.gmail.com>

On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>
>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>> line 970:
>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>> Expected:
>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>> Got:
>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>> line 973:
>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>> Expected:
>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>> Got:
>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>> called instead of
>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've created a pull request:
>>>>>>>>>
>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>
>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>> this was resolved.
>>>>>>>
>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>> memoryviews.
>>>>>>
>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>
>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>> banner to push this long awaited release.
>>>>>>>
>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>> congratulations in advance.
>>>>>>
>>>>>> Twin boys coming any day now!
>>>>>
>>>>> And the Cython team just keeps on growing!
>>>>
>>>> :)
>>>>
>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>
>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>> the repository, making sure everything we care about on hudson is
>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>> worth making/using a release branch on github.
>>>>>
>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>> then we can see the community tear it apart.
>>>>
>>>> Thanks!
>>>>
>>>> - Robert
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Sorry, my previous email with attachment bounced. Here goes.
>>>
>>> I'm getting a substantial amount of failing tests on MSVC,
>>> https://gist.github.com/1836766. I think most complex number tests are
>>> failing because they cast
>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>> which MSVC doesn't allow.
>>>
>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>> and running numpy_parallel: ImportError: No module named
>>> numpy_parallel".
>>>
>>> And then there is a huge number of permission errors: WindowsError:
>>> [Error 5] Access is denied:
>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>> something is broken in the test runner (or in my setup somehow)?
>>
>> The pasted output is a little munged because it was redirected to a
>> log (and stdout is probably block buffering, something we could also
>> fix to line buffering).
>
> I've merged cydefaults branch and now sage-tests is blue.

Great, thanks. I'll add some tests with default arguments for fused types.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sun Feb 19 22:29:04 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 19 Feb 2012 21:29:04 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
Message-ID: <CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>

On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>
>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>> line 970:
>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>> Expected:
>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>> Got:
>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>> line 973:
>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>> Expected:
>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>> Got:
>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>> **********************************************************************
>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>> called instead of
>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've created a pull request:
>>>>>>>>>
>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>
>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>> this was resolved.
>>>>>>>
>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>> memoryviews.
>>>>>>
>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>
>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>> banner to push this long awaited release.
>>>>>>>
>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>> congratulations in advance.
>>>>>>
>>>>>> Twin boys coming any day now!
>>>>>
>>>>> And the Cython team just keeps on growing!
>>>>
>>>> :)
>>>>
>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>
>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>> the repository, making sure everything we care about on hudson is
>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>> worth making/using a release branch on github.
>>>>>
>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>> then we can see the community tear it apart.
>>>>
>>>> Thanks!
>>>>
>>>> - Robert
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Sorry, my previous email with attachment bounced. Here goes.
>>>
>>> I'm getting a substantial amount of failing tests on MSVC,
>>> https://gist.github.com/1836766. I think most complex number tests are
>>> failing because they cast
>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>> which MSVC doesn't allow.
>>>
>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>> and running numpy_parallel: ImportError: No module named
>>> numpy_parallel".
>>>
>>> And then there is a huge number of permission errors: WindowsError:
>>> [Error 5] Access is denied:
>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>> something is broken in the test runner (or in my setup somehow)?
>>
>> The pasted output is a little munged because it was redirected to a
>> log (and stdout is probably block buffering, something we could also
>> fix to line buffering).
>
> I've merged cydefaults branch and now sage-tests is blue.

So, if the defaults are literals you build a tuple and set them on the
function, but if they are not literals you save everything in a struct
and use a callback that builds a tuple from the elements of that
struct, correct? Why can't you always just build a tuple, i.e., why do
you need the callback to build the tuple?

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Mon Feb 20 07:53:31 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 20 Feb 2012 10:53:31 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
Message-ID: <CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>

2012/2/20 mark florisson <markflorisson88 at gmail.com>:
> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>
>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>> line 970:
>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>> Expected:
>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>> Got:
>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>> line 973:
>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>> Expected:
>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>> Got:
>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>
>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>> called instead of
>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've created a pull request:
>>>>>>>>>>
>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>
>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>> this was resolved.
>>>>>>>>
>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>> memoryviews.
>>>>>>>
>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>
>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>> banner to push this long awaited release.
>>>>>>>>
>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>> congratulations in advance.
>>>>>>>
>>>>>>> Twin boys coming any day now!
>>>>>>
>>>>>> And the Cython team just keeps on growing!
>>>>>
>>>>> :)
>>>>>
>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>
>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>> worth making/using a release branch on github.
>>>>>>
>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>> then we can see the community tear it apart.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> - Robert
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>
>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>> failing because they cast
>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>> which MSVC doesn't allow.
>>>>
>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>> and running numpy_parallel: ImportError: No module named
>>>> numpy_parallel".
>>>>
>>>> And then there is a huge number of permission errors: WindowsError:
>>>> [Error 5] Access is denied:
>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>> something is broken in the test runner (or in my setup somehow)?
>>>
>>> The pasted output is a little munged because it was redirected to a
>>> log (and stdout is probably block buffering, something we could also
>>> fix to line buffering).
>>
>> I've merged cydefaults branch and now sage-tests is blue.
>
> So, if the defaults are literals you build a tuple and set them on the
> function, but if they are not literals you save everything in a struct
> and use a callback that builds a tuple from the elements of that
> struct, correct? Why can't you always just build a tuple, i.e., why do
> you need the callback to build the tuple?
>

So if defaults are literals const tuple is created once at constant
initialization.  Since CyFunction.defaults are already there (remember
http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
defaults tuple initialization at function create time. Instead I've
introduced constructor (defaults_getter) it's run only once and caches
result in CyFunction.defaults_tuple.

ps: We should wait with release until pyregr tests issue is solved.

-- 
vitja.

From stefan_ml at behnel.de  Mon Feb 20 09:45:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Feb 2012 09:45:35 +0100
Subject: [Cython] star-imports in Cython/Includes/cpython considered harmful
Message-ID: <4F42082F.9070809@behnel.de>

Hi,

I just noticed that the star-imports in the cpython package have serious
side effects. The cpython/__init__.pxd file has this in it:

"""
from cpython.version cimport *
from cpython.ref cimport *
from cpython.exc cimport *
from cpython.module cimport *
from cpython.mem cimport *
...
"""

This means that a direct cimport of any of those recursively cimported
names from the cpython package namespace will always trigger a complete
cimport of all cpython.* modules, thus

1) wasting compile time

2) polluting the cpython package namespace

3) polluting the internal state of Cython's list of cimported modules

4) leading to things being defined in the generated C code that don't need
to be there, specifically the object structs of PyBoolObject and
PyComplexObject, which we provide definitions for in cpython.bool and
cpython.complex.

The point where I noticed this is when I got test errors in PyPy because it
doesn't expose the bool/complex structs. That wasn't because the test
actually used them, it was purely because it had this cimport in it:

  from cpython cimport PyObject

If it had used this instead:

  from cpython.object cimport PyObject

Cython would still have parsed all of those .pxd files, but at least it
wouldn't have used those declarations to do any harm.

Now, I'm 100% certain that there is user code out there that does this as
well. Fixing this bug by removing all the star-imports will break that
code, but keeping the cimports in the package will trap the average lazy
user into cimporting stuff from there directly.

I'll add a big warning to the package for now, but I wonder if we shouldn't
accept breaking this code at some point (and sooner is always better than
later).

Stefan

From vitja.makarov at gmail.com  Mon Feb 20 15:50:37 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 20 Feb 2012 18:50:37 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <4F40E2A2.8030907@behnel.de>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
	<4F40E2A2.8030907@behnel.de>
Message-ID: <CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>

2012/2/19 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 19.02.2012 12:14:
>> I've noticed problems with py3k pyregr tests now it shows ~8K tests
>> instead of 13K
>>
>> Is that related to changes in cython or python?
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>
> At least I don't see any changes on our side that could trigger this:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792
>
> And there doesn't seem to be anything obvious in the change list of CPython
> either:
>
> https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642
>

May that be related to recent jenkins update?

-- 
vitja.

From vitja.makarov at gmail.com  Mon Feb 20 15:56:53 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 20 Feb 2012 18:56:53 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
	<4F40E2A2.8030907@behnel.de>
	<CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>
Message-ID: <CAKGHGPQRpe2i+cYtkG8EbExEG5ycXS2NwQj+HTP+hwPy88n4Kg@mail.gmail.com>

2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/19 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 19.02.2012 12:14:
>>> I've noticed problems with py3k pyregr tests now it shows ~8K tests
>>> instead of 13K
>>>
>>> Is that related to changes in cython or python?
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>>
>> At least I don't see any changes on our side that could trigger this:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792
>>
>> And there doesn't seem to be anything obvious in the change list of CPython
>> either:
>>
>> https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642
>>
>
> May that be related to recent jenkins update?
>

Very strange.

Take a look at test_fnmatch testcase:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/124/testReport/test_fnmatch/

Jenkins shows only 2 tests.


Analysing jenkins raw log

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/124/BACKEND=c,PYVERSION=py3k/consoleText

compiling (c) and running test_fnmatch ...   test_bytes
(test_fnmatch.FnmatchTestCase) ... (0.002s) OK
  test_fnmatch (test_fnmatch.FnmatchTestCase) ... (0.005s) OK
  test_fnmatchcase (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
  test_mix_bytes_str (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
  test_translate (test_fnmatch.TranslateTestCase) ... (0.001s) OK
  test_filter (test_fnmatch.FilterTestCase) ... (0.000s) OK
  runTest (__main__.CythonPyregrTestCase)


I see 4 more tests. And it seems that Jenkins missed all
test_fnmatch.FnmatchTestCase testcases.


-- 
vitja.

From vitja.makarov at gmail.com  Mon Feb 20 16:32:29 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 20 Feb 2012 19:32:29 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPQRpe2i+cYtkG8EbExEG5ycXS2NwQj+HTP+hwPy88n4Kg@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
	<4F40E2A2.8030907@behnel.de>
	<CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>
	<CAKGHGPQRpe2i+cYtkG8EbExEG5ycXS2NwQj+HTP+hwPy88n4Kg@mail.gmail.com>
Message-ID: <CAKGHGPTtxb_ANQPtS-KDrsB0DK_us0b2FgkQQ3TR5+83GzqG6Q@mail.gmail.com>

2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/2/19 Stefan Behnel <stefan_ml at behnel.de>:
>>> Vitja Makarov, 19.02.2012 12:14:
>>>> I've noticed problems with py3k pyregr tests now it shows ~8K tests
>>>> instead of 13K
>>>>
>>>> Is that related to changes in cython or python?
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>>>
>>> At least I don't see any changes on our side that could trigger this:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792
>>>
>>> And there doesn't seem to be anything obvious in the change list of CPython
>>> either:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642
>>>
>>
>> May that be related to recent jenkins update?
>>
>
> Very strange.
>
> Take a look at test_fnmatch testcase:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/124/testReport/test_fnmatch/
>
> Jenkins shows only 2 tests.
>
>
> Analysing jenkins raw log
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/124/BACKEND=c,PYVERSION=py3k/consoleText
>
> compiling (c) and running test_fnmatch ... ? test_bytes
> (test_fnmatch.FnmatchTestCase) ... (0.002s) OK
> ?test_fnmatch (test_fnmatch.FnmatchTestCase) ... (0.005s) OK
> ?test_fnmatchcase (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
> ?test_mix_bytes_str (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
> ?test_translate (test_fnmatch.TranslateTestCase) ... (0.001s) OK
> ?test_filter (test_fnmatch.FilterTestCase) ... (0.000s) OK
> ?runTest (__main__.CythonPyregrTestCase)
>
>
> I see 4 more tests. And it seems that Jenkins missed all
> test_fnmatch.FnmatchTestCase testcases.
>
>

Hmm, that's strange it seems to be a problem with XML test writer:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/ws/test-results/

There are only 2 files output, TEST-test_fnmatch.FnmatchTestCase.xml is missing.



-- 
vitja.

From d.s.seljebotn at astro.uio.no  Tue Feb 21 05:09:27 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 20 Feb 2012 20:09:27 -0800
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
Message-ID: <4F4318F7.70705@astro.uio.no>

This has got to be the most incredible and interesting turn of events 
I've seen in a while! :-)

Dag

-------- Original Message --------
Subject: 	Re: [Numpy-discussion] Proposed Roadmap Overview
Date: 	Mon, 20 Feb 2012 22:04:00 -0600
From: 	Travis Oliphant <travis at continuum.io>
Reply-To: 	Discussion of Numerical Python <numpy-discussion at scipy.org>
To: 	Discussion of Numerical Python <numpy-discussion at scipy.org>



Interesting you bring this up. I actually have a working prototype of
using Python to emit LLVM. I will be showing it at the HPC tutorial that
I am giving at PyCon. I will be making this available after PyCon to a
wider audience as open source.

It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do
the translation from Python byte-code to LLVM. This LLVM can then be
"JIT"ed. I have several applications that I would like to use this for.
It would be possible to write "more of NumPy" using this approach.
Initially, it makes it *very* easy to create a machine-code ufunc from
Python code. There are other use-cases of having loops written in Python
and plugged in to a calculation, filtering, or indexing framework that
this system will be useful for.

There is still a need for a core data-type object, a core array object,
and a core calculation object. Maybe some-day these cores can be shrunk
to a smaller subset and more of something along the lines of LLVM
generation from Python can be used. But, there is a lot of work to do
before that is possible. But, a lot of the currently pre-compiled loops
can be done on the fly instead using this approach. There are several
things I'm working on in that direction.

This is not PyPy. It certainly uses the same ideas that they are using,
but instead it fits into the CPython run-time and doesn't require
changing the whole ecosystem. If you are interested in this work let me
know. I think I'm going to call the project numpy-llvm, or fast-py, or
something like that. It is available on github and will be open source
(but it's still under active development).

Here is an example of the code to create a ufunc using the system (this
is like vectorize, but it creates machine code and by-passes the
interpreter and so is 100x faster).

from math import sin, pi

def sinc(x):
if x==0:
return 1.0
else:
return sin(x*pi)/(pi*x)

from translate import Translate
t = Translate(sinc)
t.translate()
print t.mod

res = t.make_ufunc('sinc')


-Travis



On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:

> Den 20.02.2012 17:42, skrev Sturla Molden:
>> There are still other options than C or C++ that are worth considering.
>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>> JIT-compiler and produce the performance critical code we need on the
>> fly.
>>
>>
>
> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
> than GCC and often produces better machine code. They can therefore be
> used inside an array library. It would give a faster NumPy, and we could
> keep most of it in Python.
>
> Sturla
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Attached Message Part
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120220/9adaa66b/attachment.ksh>

From robertwb at math.washington.edu  Tue Feb 21 05:42:09 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Feb 2012 20:42:09 -0800
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
In-Reply-To: <4F4318F7.70705@astro.uio.no>
References: <4F4318F7.70705@astro.uio.no>
Message-ID: <CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>

Python bytecode -> LLVM is a great idea for creating ufuncs, the
overhead of Cython + GCC is atrocious for stuff like this. (I think
Cython could make a good frontent as well, especially if we generated
just the .c code for the function rather than a full extension module
and used a good compiler to generate machine code in-memory
on-the-fly.)

Going too deeply though and you'll be starting to duplicate the work
of Unladen Swallow...

On Mon, Feb 20, 2012 at 8:09 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> This has got to be the most incredible and interesting turn of events I've
> seen in a while! :-)
>
> Dag
>
> -------- Original Message --------
> Subject: ? ? ? ?Re: [Numpy-discussion] Proposed Roadmap Overview
> Date: ? Mon, 20 Feb 2012 22:04:00 -0600
> From: ? Travis Oliphant <travis at continuum.io>
> Reply-To: ? ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
> To: ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
>
>
>
> Interesting you bring this up. I actually have a working prototype of
> using Python to emit LLVM. I will be showing it at the HPC tutorial that
> I am giving at PyCon. I will be making this available after PyCon to a
> wider audience as open source.
>
> It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do
> the translation from Python byte-code to LLVM. This LLVM can then be
> "JIT"ed. I have several applications that I would like to use this for.
> It would be possible to write "more of NumPy" using this approach.
> Initially, it makes it *very* easy to create a machine-code ufunc from
> Python code. There are other use-cases of having loops written in Python
> and plugged in to a calculation, filtering, or indexing framework that
> this system will be useful for.
>
> There is still a need for a core data-type object, a core array object,
> and a core calculation object. Maybe some-day these cores can be shrunk
> to a smaller subset and more of something along the lines of LLVM
> generation from Python can be used. But, there is a lot of work to do
> before that is possible. But, a lot of the currently pre-compiled loops
> can be done on the fly instead using this approach. There are several
> things I'm working on in that direction.
>
> This is not PyPy. It certainly uses the same ideas that they are using,
> but instead it fits into the CPython run-time and doesn't require
> changing the whole ecosystem. If you are interested in this work let me
> know. I think I'm going to call the project numpy-llvm, or fast-py, or
> something like that. It is available on github and will be open source
> (but it's still under active development).
>
> Here is an example of the code to create a ufunc using the system (this
> is like vectorize, but it creates machine code and by-passes the
> interpreter and so is 100x faster).
>
> from math import sin, pi
>
> def sinc(x):
> if x==0:
> return 1.0
> else:
> return sin(x*pi)/(pi*x)
>
> from translate import Translate
> t = Translate(sinc)
> t.translate()
> print t.mod
>
> res = t.make_ufunc('sinc')
>
>
> -Travis
>
>
>
> On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:
>
>> Den 20.02.2012 17:42, skrev Sturla Molden:
>>>
>>> There are still other options than C or C++ that are worth considering.
>>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>>> JIT-compiler and produce the performance critical code we need on the
>>> fly.
>>>
>>>
>>
>> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
>> than GCC and often produces better machine code. They can therefore be
>> used inside an array library. It would give a faster NumPy, and we could
>> keep most of it in Python.
>>
>> Sturla
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Feb 21 18:19:29 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 21 Feb 2012 17:19:29 +0000
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
In-Reply-To: <CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>
References: <4F4318F7.70705@astro.uio.no>
	<CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>
Message-ID: <CANg26EWXukok72Hv7Uhnb8riowAYwYm_gvbzJMF+L9fWpmJwhA@mail.gmail.com>

On 21 February 2012 04:42, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> Python bytecode -> LLVM is a great idea for creating ufuncs, the
> overhead of Cython + GCC is atrocious for stuff like this. (I think
> Cython could make a good frontent as well, especially if we generated
> just the .c code for the function rather than a full extension module
> and used a good compiler to generate machine code in-memory
> on-the-fly.)

I want elementwise functions and reductions for Cython as well, but
they would be specialized according to the types you use them for, so
it wouldn't be atrocious bloat (at least, no more then you'd already
be having).
I've been giving profile guided optimizations some thoughts as well, I
think they are most neatly handled through a python tracer that is
installed e.g. through sitecustomize which collects data about types,
time spent in which code regions etc during testing and development.
You can then tell Cython to compile the module, so e.g. if it
registered "a numpy array" it could generate some specializations for
dtypes either statically or dynamically through OpenCL. The
alternative approach using code instrumentation would be more of a
chore but would work just as well for Cython code with object-typed
arguments. That's all rather far off, and this looks like an
interesting development.

> Going too deeply though and you'll be starting to duplicate the work
> of Unladen Swallow...
>
> On Mon, Feb 20, 2012 at 8:09 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> This has got to be the most incredible and interesting turn of events I've
>> seen in a while! :-)
>>
>> Dag
>>
>> -------- Original Message --------
>> Subject: ? ? ? ?Re: [Numpy-discussion] Proposed Roadmap Overview
>> Date: ? Mon, 20 Feb 2012 22:04:00 -0600
>> From: ? Travis Oliphant <travis at continuum.io>
>> Reply-To: ? ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
>> To: ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
>>
>>
>>
>> Interesting you bring this up. I actually have a working prototype of
>> using Python to emit LLVM. I will be showing it at the HPC tutorial that
>> I am giving at PyCon. I will be making this available after PyCon to a
>> wider audience as open source.
>>
>> It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do
>> the translation from Python byte-code to LLVM. This LLVM can then be
>> "JIT"ed. I have several applications that I would like to use this for.
>> It would be possible to write "more of NumPy" using this approach.
>> Initially, it makes it *very* easy to create a machine-code ufunc from
>> Python code. There are other use-cases of having loops written in Python
>> and plugged in to a calculation, filtering, or indexing framework that
>> this system will be useful for.
>>
>> There is still a need for a core data-type object, a core array object,
>> and a core calculation object. Maybe some-day these cores can be shrunk
>> to a smaller subset and more of something along the lines of LLVM
>> generation from Python can be used. But, there is a lot of work to do
>> before that is possible. But, a lot of the currently pre-compiled loops
>> can be done on the fly instead using this approach. There are several
>> things I'm working on in that direction.
>>
>> This is not PyPy. It certainly uses the same ideas that they are using,
>> but instead it fits into the CPython run-time and doesn't require
>> changing the whole ecosystem. If you are interested in this work let me
>> know. I think I'm going to call the project numpy-llvm, or fast-py, or
>> something like that. It is available on github and will be open source
>> (but it's still under active development).
>>
>> Here is an example of the code to create a ufunc using the system (this
>> is like vectorize, but it creates machine code and by-passes the
>> interpreter and so is 100x faster).
>>
>> from math import sin, pi
>>
>> def sinc(x):
>> if x==0:
>> return 1.0
>> else:
>> return sin(x*pi)/(pi*x)
>>
>> from translate import Translate
>> t = Translate(sinc)
>> t.translate()
>> print t.mod
>>
>> res = t.make_ufunc('sinc')
>>
>>
>> -Travis
>>
>>
>>
>> On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:
>>
>>> Den 20.02.2012 17:42, skrev Sturla Molden:
>>>>
>>>> There are still other options than C or C++ that are worth considering.
>>>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>>>> JIT-compiler and produce the performance critical code we need on the
>>>> fly.
>>>>
>>>>
>>>
>>> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
>>> than GCC and often produces better machine code. They can therefore be
>>> used inside an array library. It would give a faster NumPy, and we could
>>> keep most of it in Python.
>>>
>>> Sturla
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Tue Feb 21 21:02:41 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Feb 2012 12:02:41 -0800
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
In-Reply-To: <CANg26EWXukok72Hv7Uhnb8riowAYwYm_gvbzJMF+L9fWpmJwhA@mail.gmail.com>
References: <4F4318F7.70705@astro.uio.no>
	<CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>
	<CANg26EWXukok72Hv7Uhnb8riowAYwYm_gvbzJMF+L9fWpmJwhA@mail.gmail.com>
Message-ID: <CADiQ+QABvF+1R_KtvxnVShfjY+GyQDUs7kJHai1VkrWmZ8sY1w@mail.gmail.com>

On Tue, Feb 21, 2012 at 9:19 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 21 February 2012 04:42, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> Python bytecode -> LLVM is a great idea for creating ufuncs, the
>> overhead of Cython + GCC is atrocious for stuff like this. (I think
>> Cython could make a good frontent as well, especially if we generated
>> just the .c code for the function rather than a full extension module
>> and used a good compiler to generate machine code in-memory
>> on-the-fly.)
>
> I want elementwise functions and reductions for Cython as well, but
> they would be specialized according to the types you use them for, so
> it wouldn't be atrocious bloat (at least, no more then you'd already
> be having).
> I've been giving profile guided optimizations some thoughts as well, I
> think they are most neatly handled through a python tracer that is
> installed e.g. through sitecustomize which collects data about types,
> time spent in which code regions etc during testing and development.
> You can then tell Cython to compile the module, so e.g. if it
> registered "a numpy array" it could generate some specializations for
> dtypes either statically or dynamically through OpenCL. The
> alternative approach using code instrumentation would be more of a
> chore but would work just as well for Cython code with object-typed
> arguments. That's all rather far off, and this looks like an
> interesting development.

One of the reasons JIT compilers can do so well is that they can use
runtime profile information to write specialized code. It'd be great
to be able to incorporate this kind of profiling into a Cython compile
too. One of the big missing features here is a "fallback mode" where
we can insert guards and fall back to generic code if they're not met,
which I think would open us up to a huge number of optimizations
without violating Python semantics.

>> Going too deeply though and you'll be starting to duplicate the work
>> of Unladen Swallow...
>>
>> On Mon, Feb 20, 2012 at 8:09 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> This has got to be the most incredible and interesting turn of events I've
>>> seen in a while! :-)
>>>
>>> Dag
>>>
>>> -------- Original Message --------
>>> Subject: ? ? ? ?Re: [Numpy-discussion] Proposed Roadmap Overview
>>> Date: ? Mon, 20 Feb 2012 22:04:00 -0600
>>> From: ? Travis Oliphant <travis at continuum.io>
>>> Reply-To: ? ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
>>> To: ? ? Discussion of Numerical Python <numpy-discussion at scipy.org>
>>>
>>>
>>>
>>> Interesting you bring this up. I actually have a working prototype of
>>> using Python to emit LLVM. I will be showing it at the HPC tutorial that
>>> I am giving at PyCon. I will be making this available after PyCon to a
>>> wider audience as open source.
>>>
>>> It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do
>>> the translation from Python byte-code to LLVM. This LLVM can then be
>>> "JIT"ed. I have several applications that I would like to use this for.
>>> It would be possible to write "more of NumPy" using this approach.
>>> Initially, it makes it *very* easy to create a machine-code ufunc from
>>> Python code. There are other use-cases of having loops written in Python
>>> and plugged in to a calculation, filtering, or indexing framework that
>>> this system will be useful for.
>>>
>>> There is still a need for a core data-type object, a core array object,
>>> and a core calculation object. Maybe some-day these cores can be shrunk
>>> to a smaller subset and more of something along the lines of LLVM
>>> generation from Python can be used. But, there is a lot of work to do
>>> before that is possible. But, a lot of the currently pre-compiled loops
>>> can be done on the fly instead using this approach. There are several
>>> things I'm working on in that direction.
>>>
>>> This is not PyPy. It certainly uses the same ideas that they are using,
>>> but instead it fits into the CPython run-time and doesn't require
>>> changing the whole ecosystem. If you are interested in this work let me
>>> know. I think I'm going to call the project numpy-llvm, or fast-py, or
>>> something like that. It is available on github and will be open source
>>> (but it's still under active development).
>>>
>>> Here is an example of the code to create a ufunc using the system (this
>>> is like vectorize, but it creates machine code and by-passes the
>>> interpreter and so is 100x faster).
>>>
>>> from math import sin, pi
>>>
>>> def sinc(x):
>>> if x==0:
>>> return 1.0
>>> else:
>>> return sin(x*pi)/(pi*x)
>>>
>>> from translate import Translate
>>> t = Translate(sinc)
>>> t.translate()
>>> print t.mod
>>>
>>> res = t.make_ufunc('sinc')
>>>
>>>
>>> -Travis
>>>
>>>
>>>
>>> On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:
>>>
>>>> Den 20.02.2012 17:42, skrev Sturla Molden:
>>>>>
>>>>> There are still other options than C or C++ that are worth considering.
>>>>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>>>>> JIT-compiler and produce the performance critical code we need on the
>>>>> fly.
>>>>>
>>>>>
>>>>
>>>> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
>>>> than GCC and often produces better machine code. They can therefore be
>>>> used inside an array library. It would give a faster NumPy, and we could
>>>> keep most of it in Python.
>>>>
>>>> Sturla
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From robertwb at math.washington.edu  Tue Feb 21 21:14:19 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Feb 2012 12:14:19 -0800
Subject: [Cython] star-imports in Cython/Includes/cpython considered
	harmful
In-Reply-To: <4F42082F.9070809@behnel.de>
References: <4F42082F.9070809@behnel.de>
Message-ID: <CADiQ+QBFDR4te+OQgsHgNq2HzKLYMMOxpo-yvkWtoV87EfMy1Q@mail.gmail.com>

On Mon, Feb 20, 2012 at 12:45 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I just noticed that the star-imports in the cpython package have serious
> side effects. The cpython/__init__.pxd file has this in it:
>
> """
> from cpython.version cimport *
> from cpython.ref cimport *
> from cpython.exc cimport *
> from cpython.module cimport *
> from cpython.mem cimport *
> ...
> """
>
> This means that a direct cimport of any of those recursively cimported
> names from the cpython package namespace will always trigger a complete
> cimport of all cpython.* modules, thus
>
> 1) wasting compile time

Which is still pretty cheap (and should be made extremely cheap for
unchanging pxd files).

> 2) polluting the cpython package namespace

It's neither inspected (by the end user) at runtime, nor in danger of
conflicts, so I don't see this as a big deal. "Flat is better than
nested."

> 3) polluting the internal state of Cython's list of cimported modules

This is really (1), other than that it doesn't really hurt to have
extra keys in a dict.

> 4) leading to things being defined in the generated C code that don't need
> to be there, specifically the object structs of PyBoolObject and
> PyComplexObject, which we provide definitions for in cpython.bool and
> cpython.complex.

This is an actual issue that should be fixed, and avoiding emitting
unused code is a worthwhile goal in many contexts.

> The point where I noticed this is when I got test errors in PyPy because it
> doesn't expose the bool/complex structs. That wasn't because the test
> actually used them, it was purely because it had this cimport in it:
>
> ?from cpython cimport PyObject
>
> If it had used this instead:
>
> ?from cpython.object cimport PyObject
>
> Cython would still have parsed all of those .pxd files, but at least it
> wouldn't have used those declarations to do any harm.
>
> Now, I'm 100% certain that there is user code out there that does this as
> well. Fixing this bug by removing all the star-imports will break that
> code, but keeping the cimports in the package will trap the average lazy
> user into cimporting stuff from there directly.
>
> I'll add a big warning to the package for now, but I wonder if we shouldn't
> accept breaking this code at some point (and sooner is always better than
> later).

I actually don't think it's a bad thing to allow people to be "lazy"
and import directly from cpython rather than have to hunt down the
specific subpackages that things live in. In fact, it's cleaner in
some sense to not have to know about the subdivisions of the cpython
library. (It's like including "Python.h" rather than all the
individual header files.)

- Robert

From njs at pobox.com  Tue Feb 21 21:18:44 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 21 Feb 2012 20:18:44 +0000
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
In-Reply-To: <CADiQ+QABvF+1R_KtvxnVShfjY+GyQDUs7kJHai1VkrWmZ8sY1w@mail.gmail.com>
References: <4F4318F7.70705@astro.uio.no>
	<CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>
	<CANg26EWXukok72Hv7Uhnb8riowAYwYm_gvbzJMF+L9fWpmJwhA@mail.gmail.com>
	<CADiQ+QABvF+1R_KtvxnVShfjY+GyQDUs7kJHai1VkrWmZ8sY1w@mail.gmail.com>
Message-ID: <CAPJVwB=62_nk78HFJ5bShX9fX1_fwy=SvoDoYtZhdSuFY6896Q@mail.gmail.com>

On Tue, Feb 21, 2012 at 8:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Tue, Feb 21, 2012 at 9:19 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 21 February 2012 04:42, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> Python bytecode -> LLVM is a great idea for creating ufuncs, the
>>> overhead of Cython + GCC is atrocious for stuff like this. (I think
>>> Cython could make a good frontent as well, especially if we generated
>>> just the .c code for the function rather than a full extension module
>>> and used a good compiler to generate machine code in-memory
>>> on-the-fly.)
>>
>> I want elementwise functions and reductions for Cython as well, but
>> they would be specialized according to the types you use them for, so
>> it wouldn't be atrocious bloat (at least, no more then you'd already
>> be having).
>> I've been giving profile guided optimizations some thoughts as well, I
>> think they are most neatly handled through a python tracer that is
>> installed e.g. through sitecustomize which collects data about types,
>> time spent in which code regions etc during testing and development.
>> You can then tell Cython to compile the module, so e.g. if it
>> registered "a numpy array" it could generate some specializations for
>> dtypes either statically or dynamically through OpenCL. The
>> alternative approach using code instrumentation would be more of a
>> chore but would work just as well for Cython code with object-typed
>> arguments. That's all rather far off, and this looks like an
>> interesting development.
>
> One of the reasons JIT compilers can do so well is that they can use
> runtime profile information to write specialized code. It'd be great
> to be able to incorporate this kind of profiling into a Cython compile
> too. One of the big missing features here is a "fallback mode" where
> we can insert guards and fall back to generic code if they're not met,
> which I think would open us up to a huge number of optimizations
> without violating Python semantics.

A really neat first-step would be to write a straight LLVM backend for
Cython -- instead of generating code, one could generate equivalent
in-memory IR. (The really hacky way to do this would be to throw the
output of the current C code generator through Clang.) That seems
achievable (at least as pie-in-the-sky projects go), and even if
no-one ever got around to adding fancier profile-guided optimizations,
it would still be useful. For instance, it would be possible to not
just import .pyx modules directly, but reload() them after editing
(!!). Or have a cython.inline() function that works like weave.inline,
but more awesome.

-- Nathaniel

From robertwb at math.washington.edu  Tue Feb 21 21:29:54 2012
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Feb 2012 12:29:54 -0800
Subject: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview
In-Reply-To: <CAPJVwB=62_nk78HFJ5bShX9fX1_fwy=SvoDoYtZhdSuFY6896Q@mail.gmail.com>
References: <4F4318F7.70705@astro.uio.no>
	<CADiQ+QCTAKBAYXpnxeg+oDX045GgJZ4GSw-qCkTxJ6NMmQ611g@mail.gmail.com>
	<CANg26EWXukok72Hv7Uhnb8riowAYwYm_gvbzJMF+L9fWpmJwhA@mail.gmail.com>
	<CADiQ+QABvF+1R_KtvxnVShfjY+GyQDUs7kJHai1VkrWmZ8sY1w@mail.gmail.com>
	<CAPJVwB=62_nk78HFJ5bShX9fX1_fwy=SvoDoYtZhdSuFY6896Q@mail.gmail.com>
Message-ID: <CADiQ+QBSQpUoNYWjxLZAjQ7SFK9YOPw3vrCSMZS2vSqE_m7yQQ@mail.gmail.com>

On Tue, Feb 21, 2012 at 12:18 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 21, 2012 at 8:02 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Tue, Feb 21, 2012 at 9:19 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 21 February 2012 04:42, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> Python bytecode -> LLVM is a great idea for creating ufuncs, the
>>>> overhead of Cython + GCC is atrocious for stuff like this. (I think
>>>> Cython could make a good frontent as well, especially if we generated
>>>> just the .c code for the function rather than a full extension module
>>>> and used a good compiler to generate machine code in-memory
>>>> on-the-fly.)
>>>
>>> I want elementwise functions and reductions for Cython as well, but
>>> they would be specialized according to the types you use them for, so
>>> it wouldn't be atrocious bloat (at least, no more then you'd already
>>> be having).
>>> I've been giving profile guided optimizations some thoughts as well, I
>>> think they are most neatly handled through a python tracer that is
>>> installed e.g. through sitecustomize which collects data about types,
>>> time spent in which code regions etc during testing and development.
>>> You can then tell Cython to compile the module, so e.g. if it
>>> registered "a numpy array" it could generate some specializations for
>>> dtypes either statically or dynamically through OpenCL. The
>>> alternative approach using code instrumentation would be more of a
>>> chore but would work just as well for Cython code with object-typed
>>> arguments. That's all rather far off, and this looks like an
>>> interesting development.
>>
>> One of the reasons JIT compilers can do so well is that they can use
>> runtime profile information to write specialized code. It'd be great
>> to be able to incorporate this kind of profiling into a Cython compile
>> too. One of the big missing features here is a "fallback mode" where
>> we can insert guards and fall back to generic code if they're not met,
>> which I think would open us up to a huge number of optimizations
>> without violating Python semantics.
>
> A really neat first-step would be to write a straight LLVM backend for
> Cython -- instead of generating code, one could generate equivalent
> in-memory IR. (The really hacky way to do this would be to throw the
> output of the current C code generator through Clang.) That seems
> achievable (at least as pie-in-the-sky projects go), and even if
> no-one ever got around to adding fancier profile-guided optimizations,
> it would still be useful.

For better or for worse, Python is still pretty tied to C (or C like
languages, with higher level concepts like function calls). But
spitting out C snippets for Clang to consume would still be cool.

> For instance, it would be possible to not
> just import .pyx modules directly, but reload() them after editing
> (!!). Or have a cython.inline() function that works like weave.inline,
> but more awesome.

Note that there is a cython.inline:
http://wiki.cython.org/enhancements/inline . The compile-time overhead
(it creates a full extension module to compile with gcc (but does do
caching)) could be greatly improved with something like this.

- Robert

From stefan_ml at behnel.de  Wed Feb 22 08:13:32 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Feb 2012 08:13:32 +0100
Subject: [Cython] star-imports in Cython/Includes/cpython considered
	harmful
In-Reply-To: <CADiQ+QBFDR4te+OQgsHgNq2HzKLYMMOxpo-yvkWtoV87EfMy1Q@mail.gmail.com>
References: <4F42082F.9070809@behnel.de>
	<CADiQ+QBFDR4te+OQgsHgNq2HzKLYMMOxpo-yvkWtoV87EfMy1Q@mail.gmail.com>
Message-ID: <4F44959C.30707@behnel.de>

Robert Bradshaw, 21.02.2012 21:14:
> On Mon, Feb 20, 2012 at 12:45 AM, Stefan Behnel wrote:
>> I just noticed that the star-imports in the cpython package have serious
>> side effects. The cpython/__init__.pxd file has this in it:
>>
>> """
>> from cpython.version cimport *
>> from cpython.ref cimport *
>> from cpython.exc cimport *
>> from cpython.module cimport *
>> from cpython.mem cimport *
>> ...
>> """
>>
>> This means that a direct cimport of any of those recursively cimported
>> names from the cpython package namespace will always trigger a complete
>> cimport of all cpython.* modules, thus
>>
>> 1) wasting compile time
> 
> Which is still pretty cheap (and should be made extremely cheap for
> unchanging pxd files).
> 
>> 2) polluting the cpython package namespace
> 
> It's neither inspected (by the end user) at runtime, nor in danger of
> conflicts, so I don't see this as a big deal. "Flat is better than
> nested."
> 
>> 3) polluting the internal state of Cython's list of cimported modules
> 
> This is really (1), other than that it doesn't really hurt to have
> extra keys in a dict.

Except when it has side-effects as in 4). If we can figure out what parts
of the cimported declarations are actually used, we could still drop
unnecessary declarations. But that doesn't currently happen.


>> 4) leading to things being defined in the generated C code that don't need
>> to be there, specifically the object structs of PyBoolObject and
>> PyComplexObject, which we provide definitions for in cpython.bool and
>> cpython.complex.
> 
> This is an actual issue that should be fixed, and avoiding emitting
> unused code is a worthwhile goal in many contexts.

This issue was the reason I brought this up in the first place.


>> The point where I noticed this is when I got test errors in PyPy because it
>> doesn't expose the bool/complex structs. That wasn't because the test
>> actually used them, it was purely because it had this cimport in it:
>>
>>  from cpython cimport PyObject
>>
>> If it had used this instead:
>>
>>  from cpython.object cimport PyObject
>>
>> Cython would still have parsed all of those .pxd files, but at least it
>> wouldn't have used those declarations to do any harm.
>>
>> Now, I'm 100% certain that there is user code out there that does this as
>> well. Fixing this bug by removing all the star-imports will break that
>> code, but keeping the cimports in the package will trap the average lazy
>> user into cimporting stuff from there directly.
>>
>> I'll add a big warning to the package for now, but I wonder if we shouldn't
>> accept breaking this code at some point (and sooner is always better than
>> later).
> 
> I actually don't think it's a bad thing to allow people to be "lazy"
> and import directly from cpython rather than have to hunt down the
> specific subpackages that things live in. In fact, it's cleaner in
> some sense to not have to know about the subdivisions of the cpython
> library. (It's like including "Python.h" rather than all the
> individual header files.)

As long as it doesn't have side-effects to do this, it's not a problem. And
I agree that it's better to actually fix the side-effects. Just not
importing from the cpython package doesn't make the problem go away that
your code won't compile on PyPy as soon as you cimport anything from
cpython.bool or cpython.complex.

Stefan

From vitja.makarov at gmail.com  Wed Feb 22 22:55:13 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 01:55:13 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPTtxb_ANQPtS-KDrsB0DK_us0b2FgkQQ3TR5+83GzqG6Q@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
	<4F40E2A2.8030907@behnel.de>
	<CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>
	<CAKGHGPQRpe2i+cYtkG8EbExEG5ycXS2NwQj+HTP+hwPy88n4Kg@mail.gmail.com>
	<CAKGHGPTtxb_ANQPtS-KDrsB0DK_us0b2FgkQQ3TR5+83GzqG6Q@mail.gmail.com>
Message-ID: <CAKGHGPRyJS6tW_yu32UBxxMTHfa-jy=Y6qfi_d28+W+DLz4HnQ@mail.gmail.com>

2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/2/19 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Vitja Makarov, 19.02.2012 12:14:
>>>>> I've noticed problems with py3k pyregr tests now it shows ~8K tests
>>>>> instead of 13K
>>>>>
>>>>> Is that related to changes in cython or python?
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>>>>
>>>> At least I don't see any changes on our side that could trigger this:
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792
>>>>
>>>> And there doesn't seem to be anything obvious in the change list of CPython
>>>> either:
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642
>>>>
>>>
>>> May that be related to recent jenkins update?
>>>
>>
>> Very strange.
>>
>> Take a look at test_fnmatch testcase:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/124/testReport/test_fnmatch/
>>
>> Jenkins shows only 2 tests.
>>
>>
>> Analysing jenkins raw log
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/124/BACKEND=c,PYVERSION=py3k/consoleText
>>
>> compiling (c) and running test_fnmatch ... ? test_bytes
>> (test_fnmatch.FnmatchTestCase) ... (0.002s) OK
>> ?test_fnmatch (test_fnmatch.FnmatchTestCase) ... (0.005s) OK
>> ?test_fnmatchcase (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
>> ?test_mix_bytes_str (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
>> ?test_translate (test_fnmatch.TranslateTestCase) ... (0.001s) OK
>> ?test_filter (test_fnmatch.FilterTestCase) ... (0.000s) OK
>> ?runTest (__main__.CythonPyregrTestCase)
>>
>>
>> I see 4 more tests. And it seems that Jenkins missed all
>> test_fnmatch.FnmatchTestCase testcases.
>>
>>
>
> Hmm, that's strange it seems to be a problem with XML test writer:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/ws/test-results/
>
> There are only 2 files output, TEST-test_fnmatch.FnmatchTestCase.xml is missing.
>
>

Yeah! I've found it, that was hard.

1. Py3k buffers are not flushed before _exit()
2. Patch/restore output doesn't work as expected
3. ]]> cannot be represented as single CDATA.

-- 
vitja.

From vitja.makarov at gmail.com  Wed Feb 22 22:56:20 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 01:56:20 +0400
Subject: [Cython] p3k pyregr tests problem
In-Reply-To: <CAKGHGPRyJS6tW_yu32UBxxMTHfa-jy=Y6qfi_d28+W+DLz4HnQ@mail.gmail.com>
References: <CAKGHGPT=Z0E4wSD1X=9qc+msfFkeTkbaNHZkTjN72NhNVqXW8g@mail.gmail.com>
	<4F40E2A2.8030907@behnel.de>
	<CAKGHGPS+djQCzQEYuzyAJSLNqtd4BveeyELf4Ui45zDx0Mxn6w@mail.gmail.com>
	<CAKGHGPQRpe2i+cYtkG8EbExEG5ycXS2NwQj+HTP+hwPy88n4Kg@mail.gmail.com>
	<CAKGHGPTtxb_ANQPtS-KDrsB0DK_us0b2FgkQQ3TR5+83GzqG6Q@mail.gmail.com>
	<CAKGHGPRyJS6tW_yu32UBxxMTHfa-jy=Y6qfi_d28+W+DLz4HnQ@mail.gmail.com>
Message-ID: <CAKGHGPRc467V_Gadra-RJw004d85-=AEPg9LyAtsS8czPXZNYA@mail.gmail.com>

2012/2/23 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2012/2/19 Stefan Behnel <stefan_ml at behnel.de>:
>>>>> Vitja Makarov, 19.02.2012 12:14:
>>>>>> I've noticed problems with py3k pyregr tests now it shows ~8K tests
>>>>>> instead of 13K
>>>>>>
>>>>>> Is that related to changes in cython or python?
>>>>>>
>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/
>>>>>
>>>>> At least I don't see any changes on our side that could trigger this:
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/changes?from=790&to=792
>>>>>
>>>>> And there doesn't seem to be anything obvious in the change list of CPython
>>>>> either:
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/py3k-hg/changes?from=640&to=642
>>>>>
>>>>
>>>> May that be related to recent jenkins update?
>>>>
>>>
>>> Very strange.
>>>
>>> Take a look at test_fnmatch testcase:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/124/testReport/test_fnmatch/
>>>
>>> Jenkins shows only 2 tests.
>>>
>>>
>>> Analysing jenkins raw log
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/124/BACKEND=c,PYVERSION=py3k/consoleText
>>>
>>> compiling (c) and running test_fnmatch ... ? test_bytes
>>> (test_fnmatch.FnmatchTestCase) ... (0.002s) OK
>>> ?test_fnmatch (test_fnmatch.FnmatchTestCase) ... (0.005s) OK
>>> ?test_fnmatchcase (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
>>> ?test_mix_bytes_str (test_fnmatch.FnmatchTestCase) ... (0.001s) OK
>>> ?test_translate (test_fnmatch.TranslateTestCase) ... (0.001s) OK
>>> ?test_filter (test_fnmatch.FilterTestCase) ... (0.000s) OK
>>> ?runTest (__main__.CythonPyregrTestCase)
>>>
>>>
>>> I see 4 more tests. And it seems that Jenkins missed all
>>> test_fnmatch.FnmatchTestCase testcases.
>>>
>>>
>>
>> Hmm, that's strange it seems to be a problem with XML test writer:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/BACKEND=c,PYVERSION=py3k/ws/test-results/
>>
>> There are only 2 files output, TEST-test_fnmatch.FnmatchTestCase.xml is missing.
>>
>>
>
> Yeah! I've found it, that was hard.
>
> 1. Py3k buffers are not flushed before _exit()
> 2. Patch/restore output doesn't work as expected
> 3. ]]> cannot be represented as single CDATA.
>

Now I'm gonna test my fixes and then push it to upstream.

https://github.com/vitek/cython/commits/_pyregr_failure


-- 
vitja.

From vitja.makarov at gmail.com  Thu Feb 23 09:30:25 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 12:30:25 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
Message-ID: <CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>

2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>
>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>> Got:
>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>> Got:
>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>> called instead of
>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>
>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>> this was resolved.
>>>>>>>>>
>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>> memoryviews.
>>>>>>>>
>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>
>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>
>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>> congratulations in advance.
>>>>>>>>
>>>>>>>> Twin boys coming any day now!
>>>>>>>
>>>>>>> And the Cython team just keeps on growing!
>>>>>>
>>>>>> :)
>>>>>>
>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>
>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>> worth making/using a release branch on github.
>>>>>>>
>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>> then we can see the community tear it apart.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> - Robert
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>
>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>> failing because they cast
>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>> which MSVC doesn't allow.
>>>>>
>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>> and running numpy_parallel: ImportError: No module named
>>>>> numpy_parallel".
>>>>>
>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>> [Error 5] Access is denied:
>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>
>>>> The pasted output is a little munged because it was redirected to a
>>>> log (and stdout is probably block buffering, something we could also
>>>> fix to line buffering).
>>>
>>> I've merged cydefaults branch and now sage-tests is blue.
>>
>> So, if the defaults are literals you build a tuple and set them on the
>> function, but if they are not literals you save everything in a struct
>> and use a callback that builds a tuple from the elements of that
>> struct, correct? Why can't you always just build a tuple, i.e., why do
>> you need the callback to build the tuple?
>>
>
> So if defaults are literals const tuple is created once at constant
> initialization. ?Since CyFunction.defaults are already there (remember
> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
> defaults tuple initialization at function create time. Instead I've
> introduced constructor (defaults_getter) it's run only once and caches
> result in CyFunction.defaults_tuple.
>
> ps: We should wait with release until pyregr tests issue is solved.
>

We can also fix this ticket before release
http://trac.cython.org/cython_trac/ticket/761

-- 
vitja.

From markflorisson88 at gmail.com  Thu Feb 23 09:34:33 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 23 Feb 2012 08:34:33 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
Message-ID: <CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>

On 23 February 2012 08:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>>> called instead of
>>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>>
>>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>>> this was resolved.
>>>>>>>>>>
>>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>>> memoryviews.
>>>>>>>>>
>>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>>
>>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>>
>>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>>> congratulations in advance.
>>>>>>>>>
>>>>>>>>> Twin boys coming any day now!
>>>>>>>>
>>>>>>>> And the Cython team just keeps on growing!
>>>>>>>
>>>>>>> :)
>>>>>>>
>>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>>
>>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>>> worth making/using a release branch on github.
>>>>>>>>
>>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>>> then we can see the community tear it apart.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> - Robert
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>
>>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>>
>>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>>> failing because they cast
>>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>>> which MSVC doesn't allow.
>>>>>>
>>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>>> and running numpy_parallel: ImportError: No module named
>>>>>> numpy_parallel".
>>>>>>
>>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>>> [Error 5] Access is denied:
>>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>>
>>>>> The pasted output is a little munged because it was redirected to a
>>>>> log (and stdout is probably block buffering, something we could also
>>>>> fix to line buffering).
>>>>
>>>> I've merged cydefaults branch and now sage-tests is blue.
>>>
>>> So, if the defaults are literals you build a tuple and set them on the
>>> function, but if they are not literals you save everything in a struct
>>> and use a callback that builds a tuple from the elements of that
>>> struct, correct? Why can't you always just build a tuple, i.e., why do
>>> you need the callback to build the tuple?
>>>
>>
>> So if defaults are literals const tuple is created once at constant
>> initialization. ?Since CyFunction.defaults are already there (remember
>> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
>> defaults tuple initialization at function create time. Instead I've
>> introduced constructor (defaults_getter) it's run only once and caches
>> result in CyFunction.defaults_tuple.
>>
>> ps: We should wait with release until pyregr tests issue is solved.
>>
>
> We can also fix this ticket before release
> http://trac.cython.org/cython_trac/ticket/761

Good idea. I think the ticket should read 'sys.path' instead of
PYTHONPATH, though.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Thu Feb 23 09:36:18 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 12:36:18 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
Message-ID: <CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>

2012/2/23 mark florisson <markflorisson88 at gmail.com>:
> On 23 February 2012 08:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>>>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>>>> called instead of
>>>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>>>> this was resolved.
>>>>>>>>>>>
>>>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>>>> memoryviews.
>>>>>>>>>>
>>>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>>>
>>>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>>>
>>>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>>>> congratulations in advance.
>>>>>>>>>>
>>>>>>>>>> Twin boys coming any day now!
>>>>>>>>>
>>>>>>>>> And the Cython team just keeps on growing!
>>>>>>>>
>>>>>>>> :)
>>>>>>>>
>>>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>>>
>>>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>>>> worth making/using a release branch on github.
>>>>>>>>>
>>>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>>>> then we can see the community tear it apart.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> - Robert
>>>>>>>> _______________________________________________
>>>>>>>> cython-devel mailing list
>>>>>>>> cython-devel at python.org
>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>
>>>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>>>
>>>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>>>> failing because they cast
>>>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>>>> which MSVC doesn't allow.
>>>>>>>
>>>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>>>> and running numpy_parallel: ImportError: No module named
>>>>>>> numpy_parallel".
>>>>>>>
>>>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>>>> [Error 5] Access is denied:
>>>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>>>
>>>>>> The pasted output is a little munged because it was redirected to a
>>>>>> log (and stdout is probably block buffering, something we could also
>>>>>> fix to line buffering).
>>>>>
>>>>> I've merged cydefaults branch and now sage-tests is blue.
>>>>
>>>> So, if the defaults are literals you build a tuple and set them on the
>>>> function, but if they are not literals you save everything in a struct
>>>> and use a callback that builds a tuple from the elements of that
>>>> struct, correct? Why can't you always just build a tuple, i.e., why do
>>>> you need the callback to build the tuple?
>>>>
>>>
>>> So if defaults are literals const tuple is created once at constant
>>> initialization. ?Since CyFunction.defaults are already there (remember
>>> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
>>> defaults tuple initialization at function create time. Instead I've
>>> introduced constructor (defaults_getter) it's run only once and caches
>>> result in CyFunction.defaults_tuple.
>>>
>>> ps: We should wait with release until pyregr tests issue is solved.
>>>
>>
>> We can also fix this ticket before release
>> http://trac.cython.org/cython_trac/ticket/761
>
> Good idea. I think the ticket should read 'sys.path' instead of
> PYTHONPATH, though.
>

Yeah, I think the fix is trivial we should prepend (or append?)
sys.path to cython includes


-- 
vitja.

From vitja.makarov at gmail.com  Thu Feb 23 09:37:38 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 12:37:38 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
Message-ID: <CAKGHGPSHgjrf3xhwc1c9bGrB6QDBjkccyeQcHLTRT4mvwynCgw@mail.gmail.com>

2012/2/23 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/23 mark florisson <markflorisson88 at gmail.com>:
>> On 23 February 2012 08:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>>>>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>>>>> called instead of
>>>>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>>>>> this was resolved.
>>>>>>>>>>>>
>>>>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>>>>> memoryviews.
>>>>>>>>>>>
>>>>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>>>>
>>>>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>>>>
>>>>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>>>>> congratulations in advance.
>>>>>>>>>>>
>>>>>>>>>>> Twin boys coming any day now!
>>>>>>>>>>
>>>>>>>>>> And the Cython team just keeps on growing!
>>>>>>>>>
>>>>>>>>> :)
>>>>>>>>>
>>>>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>>>>
>>>>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>>>>> worth making/using a release branch on github.
>>>>>>>>>>
>>>>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>>>>> then we can see the community tear it apart.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> - Robert
>>>>>>>>> _______________________________________________
>>>>>>>>> cython-devel mailing list
>>>>>>>>> cython-devel at python.org
>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>
>>>>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>>>>
>>>>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>>>>> failing because they cast
>>>>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>>>>> which MSVC doesn't allow.
>>>>>>>>
>>>>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>>>>> and running numpy_parallel: ImportError: No module named
>>>>>>>> numpy_parallel".
>>>>>>>>
>>>>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>>>>> [Error 5] Access is denied:
>>>>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>>>>
>>>>>>> The pasted output is a little munged because it was redirected to a
>>>>>>> log (and stdout is probably block buffering, something we could also
>>>>>>> fix to line buffering).
>>>>>>
>>>>>> I've merged cydefaults branch and now sage-tests is blue.
>>>>>
>>>>> So, if the defaults are literals you build a tuple and set them on the
>>>>> function, but if they are not literals you save everything in a struct
>>>>> and use a callback that builds a tuple from the elements of that
>>>>> struct, correct? Why can't you always just build a tuple, i.e., why do
>>>>> you need the callback to build the tuple?
>>>>>
>>>>
>>>> So if defaults are literals const tuple is created once at constant
>>>> initialization. ?Since CyFunction.defaults are already there (remember
>>>> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
>>>> defaults tuple initialization at function create time. Instead I've
>>>> introduced constructor (defaults_getter) it's run only once and caches
>>>> result in CyFunction.defaults_tuple.
>>>>
>>>> ps: We should wait with release until pyregr tests issue is solved.
>>>>
>>>
>>> We can also fix this ticket before release
>>> http://trac.cython.org/cython_trac/ticket/761
>>
>> Good idea. I think the ticket should read 'sys.path' instead of
>> PYTHONPATH, though.
>>
>
> Yeah, I think the fix is trivial we should prepend (or append?)
> sys.path to cython includes
>
>

Btw we have 3 more regressions for py3k, I think some of them are
related to hash randomization.

https://sage.math.washington.edu:8091/hudson/view/dev-vitek/job/cython-vitek-tests/BACKEND=c,PYVERSION=py3k/lastCompletedBuild/testReport/




-- 
vitja.

From markflorisson88 at gmail.com  Thu Feb 23 09:38:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 23 Feb 2012 08:38:22 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
Message-ID: <CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>

On 23 February 2012 08:36, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/23 mark florisson <markflorisson88 at gmail.com>:
>> On 23 February 2012 08:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>>>>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>>>>> called instead of
>>>>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>>>>> this was resolved.
>>>>>>>>>>>>
>>>>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>>>>> memoryviews.
>>>>>>>>>>>
>>>>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>>>>
>>>>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>>>>
>>>>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>>>>> congratulations in advance.
>>>>>>>>>>>
>>>>>>>>>>> Twin boys coming any day now!
>>>>>>>>>>
>>>>>>>>>> And the Cython team just keeps on growing!
>>>>>>>>>
>>>>>>>>> :)
>>>>>>>>>
>>>>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>>>>
>>>>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>>>>> worth making/using a release branch on github.
>>>>>>>>>>
>>>>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>>>>> then we can see the community tear it apart.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> - Robert
>>>>>>>>> _______________________________________________
>>>>>>>>> cython-devel mailing list
>>>>>>>>> cython-devel at python.org
>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>
>>>>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>>>>
>>>>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>>>>> failing because they cast
>>>>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>>>>> which MSVC doesn't allow.
>>>>>>>>
>>>>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>>>>> and running numpy_parallel: ImportError: No module named
>>>>>>>> numpy_parallel".
>>>>>>>>
>>>>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>>>>> [Error 5] Access is denied:
>>>>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>>>>
>>>>>>> The pasted output is a little munged because it was redirected to a
>>>>>>> log (and stdout is probably block buffering, something we could also
>>>>>>> fix to line buffering).
>>>>>>
>>>>>> I've merged cydefaults branch and now sage-tests is blue.
>>>>>
>>>>> So, if the defaults are literals you build a tuple and set them on the
>>>>> function, but if they are not literals you save everything in a struct
>>>>> and use a callback that builds a tuple from the elements of that
>>>>> struct, correct? Why can't you always just build a tuple, i.e., why do
>>>>> you need the callback to build the tuple?
>>>>>
>>>>
>>>> So if defaults are literals const tuple is created once at constant
>>>> initialization. ?Since CyFunction.defaults are already there (remember
>>>> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
>>>> defaults tuple initialization at function create time. Instead I've
>>>> introduced constructor (defaults_getter) it's run only once and caches
>>>> result in CyFunction.defaults_tuple.
>>>>
>>>> ps: We should wait with release until pyregr tests issue is solved.
>>>>
>>>
>>> We can also fix this ticket before release
>>> http://trac.cython.org/cython_trac/ticket/761
>>
>> Good idea. I think the ticket should read 'sys.path' instead of
>> PYTHONPATH, though.
>>
>
> Yeah, I think the fix is trivial we should prepend (or append?)
> sys.path to cython includes
>

I think append, you'd want local things to override installed things
along sys.path.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Thu Feb 23 09:40:07 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 23 Feb 2012 12:40:07 +0400
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CADiQ+QADOEdLm6L5BA4iBwV6y+raCtHRjcuDMkEGBYOW2KVnEA@mail.gmail.com>
	<CANg26EXskkUz7+dMmix=ubS0PW1V6ekZ3kCr2+4Y2wiWN3fQAQ@mail.gmail.com>
	<4F1FEEEE.2060605@behnel.de>
	<CANg26EUZQwas7Sbi=-1jkcowrXOhbPhJpFJ7CCaWvGxYBj+vjQ@mail.gmail.com>
	<CADiQ+QC=YfZpSt=z4jCs21-=fezuXNkw8OrbALDPV-YF3rFijQ@mail.gmail.com>
	<4F2083B0.9020209@creativetrax.com>
	<CAKGHGPSMO9Nf3iv41aeNO1kAR4x8TbkV8T9TpP7NBSxUTFoWhw@mail.gmail.com>
	<CADiQ+QDOJsO9Jy7fr4egSNJJvdG5Tj=iuCAHvWiZTPNgAJwaZQ@mail.gmail.com>
	<CAKGHGPS284T2_-GDO72z9padh=ZdO54RfuE8k3TBMzL3vFVsqw@mail.gmail.com>
	<CADiQ+QD0nQ9+GvJyJQ8aHn6-sq5pGs3JGhmmpg3fBvC78fhHwg@mail.gmail.com>
	<CADiQ+QD=tEcGrtb2ie90DjZPx9Ef_7B-XQqo_XX3x6czMmUT7w@mail.gmail.com>
	<CAKGHGPTfEsUPa=-8ZggDSvYtGo7c2Zhk2NKC0V6a4Z4Axn30nw@mail.gmail.com>
	<CAKGHGPSTxdYVELsPjUVHvVSt8Na0NdaNhD73oCkWqThK5QBSSg@mail.gmail.com>
	<CADiQ+QBuui0tfCwHHmf=sMedcDcLKbd5MMfedzWhiUV+J+0QiQ@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
Message-ID: <CAKGHGPRYvyZxjSABMTc7B6M-XQBWZADY=6NopYi=Of5iOOfWpQ@mail.gmail.com>

2012/2/23 mark florisson <markflorisson88 at gmail.com>:
> On 23 February 2012 08:36, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/23 mark florisson <markflorisson88 at gmail.com>:
>>> On 23 February 2012 08:30, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2012/2/20 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> 2012/2/20 mark florisson <markflorisson88 at gmail.com>:
>>>>>> On 19 February 2012 10:16, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>> 2012/2/15 mark florisson <markflorisson88 at gmail.com>:
>>>>>>>> On 15 February 2012 15:45, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>>>> On 14 February 2012 21:33, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>> On Tue, Feb 14, 2012 at 1:09 PM, mark florisson
>>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>>> On 14 February 2012 17:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>>> On Tue, Feb 14, 2012 at 7:49 AM, mark florisson
>>>>>>>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>>>>>>>> On 14 February 2012 07:07, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>>>>> On Sun, Feb 12, 2012 at 12:53 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>>>>> 2012/2/12 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>>>>>>>>>>>>> 2012/2/11 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>>>>> All of Sage passes except for one test:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> sage -t ?devel/sage/sage/misc/sageinspect.py
>>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>>> line 970:
>>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(bernstein_polynomial_factory_ratlist.coeffs_bitsize)
>>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)
>>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>>> File "/levi/scratch/robertwb/hudson/sage-4.8/devel/sage-main/sage/misc/sageinspect.py",
>>>>>>>>>>>>>>>>> line 973:
>>>>>>>>>>>>>>>>> ? ?sage: sage_getargspec(BooleanMonomialMonoid.gen)
>>>>>>>>>>>>>>>>> Expected:
>>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=(0,))
>>>>>>>>>>>>>>>>> Got:
>>>>>>>>>>>>>>>>> ? ?ArgSpec(args=['self', 'i'], varargs=None, keywords=None, defaults=())
>>>>>>>>>>>>>>>>> **********************************************************************
>>>>>>>>>>>>>>>>> 1 items had failures:
>>>>>>>>>>>>>>>>> ? 2 of ?31 in __main__.example_21
>>>>>>>>>>>>>>>>> ***Test Failed*** 2 failures.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Any ideas why this would have changed?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CyFunction now provides its own code object. So inspect.getargs() is
>>>>>>>>>>>>>>>> called instead of
>>>>>>>>>>>>>>>> inspect.ArgSpec(*_sage_getargspec_cython(sage_getsource(obj))). It
>>>>>>>>>>>>>>>> seems like func.func_defaults should be implemented.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've created a pull request:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/cython/cython/pull/88
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks! The only other thing I can think of was a question of using
>>>>>>>>>>>>>> caching to mitigate the longer compile times, but I can't remember if
>>>>>>>>>>>>>> this was resolved.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The compiler has like 2 or 3 seconds of constant overhead if you use
>>>>>>>>>>>>> memoryviews.
>>>>>>>>>>>>
>>>>>>>>>>>> That'd be nice to cut down, but certainly not a blocker.
>>>>>>>>>>>>
>>>>>>>>>>>>>> As I'm going to be MIA any day now, someone else should take up the
>>>>>>>>>>>>>> banner to push this long awaited release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Missing in action"? Are you planning to desert? :) I can't find any
>>>>>>>>>>>>> relevant abbreviation, but I think I know what it means,
>>>>>>>>>>>>> congratulations in advance.
>>>>>>>>>>>>
>>>>>>>>>>>> Twin boys coming any day now!
>>>>>>>>>>>
>>>>>>>>>>> And the Cython team just keeps on growing!
>>>>>>>>>>
>>>>>>>>>> :)
>>>>>>>>>>
>>>>>>>>>>>>> Stefan, you have been involved the longest, would you feel up to the
>>>>>>>>>>>>> task? You probably have the best understanding and experience with any
>>>>>>>>>>>>> issues (no pressure :). Otherwise I could have a try...
>>>>>>>>>>>>
>>>>>>>>>>>> It's pretty easy. Once the defaults change is in it's probably worth
>>>>>>>>>>>> cutting a beta or release candidate to email to dev/users, and if
>>>>>>>>>>>> there's no blocking feedback you go ahead and push it out (basically
>>>>>>>>>>>> writing up the release notes on the wiki, cleaning up trac, tagging
>>>>>>>>>>>> the repository, making sure everything we care about on hudson is
>>>>>>>>>>>> still passing, uploading to pypi and the website (the sdist tarball),
>>>>>>>>>>>> emailing our lists and python-announce, re-building and updating the
>>>>>>>>>>>> pointer to the documentation, ...) If it goes on for a while it's
>>>>>>>>>>>> worth making/using a release branch on github.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the summary, I'm sure I would have missed one or two :) Ok,
>>>>>>>>>>> I'll volunteer then. Maybe I can create a beta somewhere next week and
>>>>>>>>>>> then we can see the community tear it apart.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> - Robert
>>>>>>>>>> _______________________________________________
>>>>>>>>>> cython-devel mailing list
>>>>>>>>>> cython-devel at python.org
>>>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>>
>>>>>>>>> Sorry, my previous email with attachment bounced. Here goes.
>>>>>>>>>
>>>>>>>>> I'm getting a substantial amount of failing tests on MSVC,
>>>>>>>>> https://gist.github.com/1836766. I think most complex number tests are
>>>>>>>>> failing because they cast
>>>>>>>>> a struct of a certain type to itself like ((struct_A) my_struct_A),
>>>>>>>>> which MSVC doesn't allow.
>>>>>>>>>
>>>>>>>>> Some tests seem to fail because they can't be imported: "compiling (c)
>>>>>>>>> and running numpy_parallel: ImportError: No module named
>>>>>>>>> numpy_parallel".
>>>>>>>>>
>>>>>>>>> And then there is a huge number of permission errors: WindowsError:
>>>>>>>>> [Error 5] Access is denied:
>>>>>>>>> 'c:\\Users\\mark\\cython\\BUILD\\compile\\cpp\\libc_math.pyd' . Maybe
>>>>>>>>> something is broken in the test runner (or in my setup somehow)?
>>>>>>>>
>>>>>>>> The pasted output is a little munged because it was redirected to a
>>>>>>>> log (and stdout is probably block buffering, something we could also
>>>>>>>> fix to line buffering).
>>>>>>>
>>>>>>> I've merged cydefaults branch and now sage-tests is blue.
>>>>>>
>>>>>> So, if the defaults are literals you build a tuple and set them on the
>>>>>> function, but if they are not literals you save everything in a struct
>>>>>> and use a callback that builds a tuple from the elements of that
>>>>>> struct, correct? Why can't you always just build a tuple, i.e., why do
>>>>>> you need the callback to build the tuple?
>>>>>>
>>>>>
>>>>> So if defaults are literals const tuple is created once at constant
>>>>> initialization. ?Since CyFunction.defaults are already there (remember
>>>>> http://trac.cython.org/cython_trac/ticket/674) I've decided to avoid
>>>>> defaults tuple initialization at function create time. Instead I've
>>>>> introduced constructor (defaults_getter) it's run only once and caches
>>>>> result in CyFunction.defaults_tuple.
>>>>>
>>>>> ps: We should wait with release until pyregr tests issue is solved.
>>>>>
>>>>
>>>> We can also fix this ticket before release
>>>> http://trac.cython.org/cython_trac/ticket/761
>>>
>>> Good idea. I think the ticket should read 'sys.path' instead of
>>> PYTHONPATH, though.
>>>
>>
>> Yeah, I think the fix is trivial we should prepend (or append?)
>> sys.path to cython includes
>>
>
> I think append, you'd want local things to override installed things
> along sys.path.
>

Yeah, right.


-- 
vitja.

From stefan_ml at behnel.de  Thu Feb 23 16:43:19 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Feb 2012 16:43:19 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
Message-ID: <4F465E97.8080606@behnel.de>

mark florisson, 23.02.2012 09:38:
> On 23 February 2012 08:36, Vitja Makarov wrote:
>> 2012/2/23 mark florisson:
>>> On 23 February 2012 08:30, Vitja Makarov wrote:
>>>> We can also fix this ticket before release
>>>> http://trac.cython.org/cython_trac/ticket/761
>>>
>>> Good idea. I think the ticket should read 'sys.path' instead of
>>> PYTHONPATH, though.
>>
>> Yeah, I think the fix is trivial we should prepend (or append?)
>> sys.path to cython includes
> 
> I think append, you'd want local things to override installed things
> along sys.path.

I think it's

1) user provided directories

2) Cython provided directories

3) sys.path

or would you swap 2) and 3) ?

Stefan

From markflorisson88 at gmail.com  Thu Feb 23 17:34:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 23 Feb 2012 16:34:22 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <4F465E97.8080606@behnel.de>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
Message-ID: <CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>

On 23 February 2012 15:43, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 23.02.2012 09:38:
>> On 23 February 2012 08:36, Vitja Makarov wrote:
>>> 2012/2/23 mark florisson:
>>>> On 23 February 2012 08:30, Vitja Makarov wrote:
>>>>> We can also fix this ticket before release
>>>>> http://trac.cython.org/cython_trac/ticket/761
>>>>
>>>> Good idea. I think the ticket should read 'sys.path' instead of
>>>> PYTHONPATH, though.
>>>
>>> Yeah, I think the fix is trivial we should prepend (or append?)
>>> sys.path to cython includes
>>
>> I think append, you'd want local things to override installed things
>> along sys.path.
>
> I think it's
>
> 1) user provided directories
>
> 2) Cython provided directories
>
> 3) sys.path

I think this is the most sensible order, yes. And the current
directory would also come before anything else, as always.

> or would you swap 2) and 3) ?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Thu Feb 23 21:52:04 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 24 Feb 2012 00:52:04 +0400
Subject: [Cython] cython tests and py3k hash randomization
Message-ID: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>

Recent py3k version has new feature "hash randomization" it solves
some security issues.
But has some drawbacks, for instance, dict.items() order is now
unknown. So it breaks
randomly some doctests that rely on exact order of dict items.

vitja at mchome:~/python$ ./py3k/bin/python -c  "print({'a':1, 'b':2})"
{'b': 2, 'a': 1}
vitja at mchome:~/python$ ./py3k/bin/python -c  "print({'a':1, 'b':2})"
{'a': 1, 'b': 2}

As a workaround we can set PYTHONHASHSEED environment variable to zero
for all cyhon-*-tests targets

This also affects generated code it internally depends on order of
items returned by dict.items()

-- 
vitja.

From dalcinl at gmail.com  Thu Feb 23 22:00:14 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Feb 2012 18:00:14 -0300
Subject: [Cython] Broken C-API generation for ext modules
Message-ID: <CAEcYPwDEsdrqocRXjkb9Z_0QkgiyngCOcn8QwFTH5KigyAmmrA@mail.gmail.com>

The commit below from Stefan broke C-API generation for extension
modules. The problem is that the code of __Pyx_ImportModule() and
__Pyx_ImportType() depeds on the #definition of
__Pyx_PyIdentifier_FromString , and such #define is not emitted in
C-API headers.

Stefan, given that __Pyx_PyIdentifier_FromString() is only used in
__Pyx_ImportModule() and __Pyx_ImportType(), what's your opinion about
just rolling back your commit?


commit 500f9a40f5ad441c2c204d076cfc4f82a41d531b
Author: Stefan Behnel <scoder at users.berlios.de>
Date:   Sat Sep 10 00:14:05 2011 +0200

    minor code simplification in utility functions by using macro for
Py2/3 dependent PyString/PyUnicode_FromString() calls


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Thu Feb 23 22:39:13 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 23 Feb 2012 21:39:13 +0000
Subject: [Cython] cython tests and py3k hash randomization
In-Reply-To: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
References: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
Message-ID: <CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>

On 23 February 2012 20:52, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Recent py3k version has new feature "hash randomization" it solves
> some security issues.
> But has some drawbacks, for instance, dict.items() order is now
> unknown. So it breaks
> randomly some doctests that rely on exact order of dict items.
>
> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
> {'b': 2, 'a': 1}
> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
> {'a': 1, 'b': 2}
>
> As a workaround we can set PYTHONHASHSEED environment variable to zero
> for all cyhon-*-tests targets
>
> This also affects generated code it internally depends on order of
> items returned by dict.items()

Any code or test that relies on dictionary order is wrong, really. So
I assume any pyregr issues will be fixed by the CPython test suite? If
there are any such failing tests in Cython we should simply fix them.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Fri Feb 24 06:24:12 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 24 Feb 2012 09:24:12 +0400
Subject: [Cython] cython tests and py3k hash randomization
In-Reply-To: <CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>
References: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
	<CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>
Message-ID: <CAKGHGPR3Wo4bycZGe1_RBuzQC-jnj+bd1p=Q5NF46rYhBSvmLg@mail.gmail.com>

2012/2/24 mark florisson <markflorisson88 at gmail.com>:
> On 23 February 2012 20:52, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> Recent py3k version has new feature "hash randomization" it solves
>> some security issues.
>> But has some drawbacks, for instance, dict.items() order is now
>> unknown. So it breaks
>> randomly some doctests that rely on exact order of dict items.
>>
>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>> {'b': 2, 'a': 1}
>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>> {'a': 1, 'b': 2}
>>
>> As a workaround we can set PYTHONHASHSEED environment variable to zero
>> for all cyhon-*-tests targets
>>
>> This also affects generated code it internally depends on order of
>> items returned by dict.items()
>
> Any code or test that relies on dictionary order is wrong, really. So
> I assume any pyregr issues will be fixed by the CPython test suite? If
> there are any such failing tests in Cython we should simply fix them.
>

Yes, you're right but I'm not sure how many tests may be broken. I
don't think we want to fix
them before release. Now I've added

# Disable python hash randomization
export PYTHONHASHSEED=0

to cython-devel-tests and it worked.

It's not that easy to fix, here is simple doctest:

def test_foo():
    """
    >>> test_foo()
    {'a': 1, 'b': 2}
    """
    return {'a': 1, 'b': 2}

You can't use dicts in doctests anymore there are few options instead:
1. Compare sorted items(), e.g. sorted(test_foo().items())
2. Inplace compare test_foo() == {...}

-- 
vitja.

From stefan_ml at behnel.de  Fri Feb 24 09:42:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Feb 2012 09:42:26 +0100
Subject: [Cython] cython tests and py3k hash randomization
In-Reply-To: <CAKGHGPR3Wo4bycZGe1_RBuzQC-jnj+bd1p=Q5NF46rYhBSvmLg@mail.gmail.com>
References: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
	<CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>
	<CAKGHGPR3Wo4bycZGe1_RBuzQC-jnj+bd1p=Q5NF46rYhBSvmLg@mail.gmail.com>
Message-ID: <4F474D72.7050504@behnel.de>

Vitja Makarov, 24.02.2012 06:24:
> 2012/2/24 mark florisson <markflorisson88 at gmail.com>:
>> On 23 February 2012 20:52, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> Recent py3k version has new feature "hash randomization" it solves
>>> some security issues.
>>> But has some drawbacks, for instance, dict.items() order is now
>>> unknown. So it breaks
>>> randomly some doctests that rely on exact order of dict items.
>>>
>>> vitja at mchome:~/python$ ./py3k/bin/python -c  "print({'a':1, 'b':2})"
>>> {'b': 2, 'a': 1}
>>> vitja at mchome:~/python$ ./py3k/bin/python -c  "print({'a':1, 'b':2})"
>>> {'a': 1, 'b': 2}
>>>
>>> As a workaround we can set PYTHONHASHSEED environment variable to zero
>>> for all cyhon-*-tests targets
>>>
>>> This also affects generated code it internally depends on order of
>>> items returned by dict.items()
>>
>> Any code or test that relies on dictionary order is wrong, really. So
>> I assume any pyregr issues will be fixed by the CPython test suite?

Definitely.


>> If there are any such failing tests in Cython we should simply fix them.
> 
> Yes, you're right but I'm not sure how many tests may be broken. I
> don't think we want to fix them before release. Now I've added
> 
> # Disable python hash randomization
> export PYTHONHASHSEED=0
> 
> to cython-devel-tests and it worked.

That should only be a work-around until all tests are fixed, though.


> It's not that easy to fix, here is simple doctest:
> 
> def test_foo():
>     """
>     >>> test_foo()
>     {'a': 1, 'b': 2}
>     """
>     return {'a': 1, 'b': 2}
> 
> You can't use dicts in doctests anymore there are few options instead:
> 1. Compare sorted items(), e.g. sorted(test_foo().items())
> 2. Inplace compare test_foo() == {...}

I always use 1) because it gives you better test failure output in doctests.

Stefan

From vitja.makarov at gmail.com  Fri Feb 24 10:07:59 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 24 Feb 2012 13:07:59 +0400
Subject: [Cython] cython tests and py3k hash randomization
In-Reply-To: <4F474D72.7050504@behnel.de>
References: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
	<CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>
	<CAKGHGPR3Wo4bycZGe1_RBuzQC-jnj+bd1p=Q5NF46rYhBSvmLg@mail.gmail.com>
	<4F474D72.7050504@behnel.de>
Message-ID: <CAKGHGPRJjcFFVUPXrF1Jm+3cY=QNkoq0mfQ6bbrk5eP4kc9BPA@mail.gmail.com>

2012/2/24 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 24.02.2012 06:24:
>> 2012/2/24 mark florisson <markflorisson88 at gmail.com>:
>>> On 23 February 2012 20:52, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> Recent py3k version has new feature "hash randomization" it solves
>>>> some security issues.
>>>> But has some drawbacks, for instance, dict.items() order is now
>>>> unknown. So it breaks
>>>> randomly some doctests that rely on exact order of dict items.
>>>>
>>>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>>>> {'b': 2, 'a': 1}
>>>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>>>> {'a': 1, 'b': 2}
>>>>
>>>> As a workaround we can set PYTHONHASHSEED environment variable to zero
>>>> for all cyhon-*-tests targets
>>>>
>>>> This also affects generated code it internally depends on order of
>>>> items returned by dict.items()
>>>
>>> Any code or test that relies on dictionary order is wrong, really. So
>>> I assume any pyregr issues will be fixed by the CPython test suite?
>
> Definitely.
>

Ok, I'll take a look.

>
>>> If there are any such failing tests in Cython we should simply fix them.
>>
>> Yes, you're right but I'm not sure how many tests may be broken. I
>> don't think we want to fix them before release. Now I've added
>>
>> # Disable python hash randomization
>> export PYTHONHASHSEED=0
>>
>> to cython-devel-tests and it worked.
>
> That should only be a work-around until all tests are fixed, though.
>
>
>> It's not that easy to fix, here is simple doctest:
>>
>> def test_foo():
>> ? ? """
>> ? ? >>> test_foo()
>> ? ? {'a': 1, 'b': 2}
>> ? ? """
>> ? ? return {'a': 1, 'b': 2}
>>
>> You can't use dicts in doctests anymore there are few options instead:
>> 1. Compare sorted items(), e.g. sorted(test_foo().items())
>> 2. Inplace compare test_foo() == {...}
>
> I always use 1) because it gives you better test failure output in doctests.
>



-- 
vitja.

From sable at users.sourceforge.net  Fri Feb 24 10:25:34 2012
From: sable at users.sourceforge.net (=?ISO-8859-1?Q?S=E9bastien_Sabl=E9_Sabl=E9?=)
Date: Fri, 24 Feb 2012 10:25:34 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
Message-ID: <CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>

Hi,

could you please also look at incorporating the following patch before
releasing 0.16? (if it has not already been merged)

https://github.com/cython/cython/pull/67

It has been more or less validated, but a test case is needed.

This patch makes using C++ templates much more convenient with Cython.

Currently I have to use hacks like the following which looks ugly and make
the code less readable:
ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData<float>"

Also thank you for all the work done on Cython, I have been using it (and
Pyrex before) intensively  for more than 6 years now, and it makes
integrating Python and C/C++ really convenient.

Thanks in advance

S?bastien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120224/5f0e7d3b/attachment.html>

From stefan_ml at behnel.de  Fri Feb 24 12:00:23 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Feb 2012 12:00:23 +0100
Subject: [Cython] Broken C-API generation for ext modules
In-Reply-To: <CAEcYPwDEsdrqocRXjkb9Z_0QkgiyngCOcn8QwFTH5KigyAmmrA@mail.gmail.com>
References: <CAEcYPwDEsdrqocRXjkb9Z_0QkgiyngCOcn8QwFTH5KigyAmmrA@mail.gmail.com>
Message-ID: <4F476DC7.6090805@behnel.de>

Lisandro Dalcin, 23.02.2012 22:00:
> The commit below from Stefan broke C-API generation for extension
> modules. The problem is that the code of __Pyx_ImportModule() and
> __Pyx_ImportType() depeds on the #definition of
> __Pyx_PyIdentifier_FromString , and such #define is not emitted in
> C-API headers.
> 
> Stefan, given that __Pyx_PyIdentifier_FromString() is only used in
> __Pyx_ImportModule() and __Pyx_ImportType(), what's your opinion about
> just rolling back your commit?
> 
> 
> commit 500f9a40f5ad441c2c204d076cfc4f82a41d531b
> Author: Stefan Behnel
> Date:   Sat Sep 10 00:14:05 2011 +0200

That's impressively old for a broken feature.


>     minor code simplification in utility functions by using macro for
> Py2/3 dependent PyString/PyUnicode_FromString() calls

https://github.com/cython/cython/commit/5a31a3d8d38d9d266886916432f1ebe621a2bc69

I pushed a fix here. Looks like the capi tests didn't detect this, though...

Stefan

From dalcinl at gmail.com  Fri Feb 24 16:07:12 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Feb 2012 12:07:12 -0300
Subject: [Cython] Broken C-API generation for ext modules
In-Reply-To: <4F476DC7.6090805@behnel.de>
References: <CAEcYPwDEsdrqocRXjkb9Z_0QkgiyngCOcn8QwFTH5KigyAmmrA@mail.gmail.com>
	<4F476DC7.6090805@behnel.de>
Message-ID: <CAEcYPwDUKU3D5t2-BA4FDN+fjLOpMTRC9e5haKJgLynyTkZ5CA@mail.gmail.com>

On 24 February 2012 08:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 23.02.2012 22:00:
>> The commit below from Stefan broke C-API generation for extension
>> modules. The problem is that the code of __Pyx_ImportModule() and
>> __Pyx_ImportType() depeds on the #definition of
>> __Pyx_PyIdentifier_FromString , and such #define is not emitted in
>> C-API headers.
>>
>> Stefan, given that __Pyx_PyIdentifier_FromString() is only used in
>> __Pyx_ImportModule() and __Pyx_ImportType(), what's your opinion about
>> just rolling back your commit?
>>
>>
>> commit 500f9a40f5ad441c2c204d076cfc4f82a41d531b
>> Author: Stefan Behnel
>> Date: ? Sat Sep 10 00:14:05 2011 +0200
>
> That's impressively old for a broken feature.
>

Indeed. I do not test for this feature on a regular basis in my
projects, and I've been using latest Cython release in production
runs.

>
>> ? ? minor code simplification in utility functions by using macro for
>> Py2/3 dependent PyString/PyUnicode_FromString() calls
>
> https://github.com/cython/cython/commit/5a31a3d8d38d9d266886916432f1ebe621a2bc69
>
> I pushed a fix here. Looks like the capi tests didn't detect this, though...
>

Thanks, I've improved the tests here:
https://github.com/cython/cython/commit/6f343d9c82e611af5e437641583bdb88a926af93

BTW, if you run

$ python runtests.py --no-cleanup module_api

and next look at the generated api header:

$ emacs BUILD/build/module_api/a_api.h

you will notice that there are a lot of blank lines between groups of
declarations and definitions. I guess this originates at
UtilityCodeBase.load_utilities_from_file(), but could not figure out
how to fix it... could you take a look?


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From vitja.makarov at gmail.com  Fri Feb 24 17:56:38 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 24 Feb 2012 20:56:38 +0400
Subject: [Cython] jenkins is down
Message-ID: <CAKGHGPQYB9F8vT1AEasiba8qjvW732ud5Wv9ds-9afbx8zRM4w@mail.gmail.com>

Jenkins app is down now. Before it got down it was raising exceptions.

Can some one restart jenkins?

-- 
vitja.

From stefan_ml at behnel.de  Fri Feb 24 18:11:42 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Feb 2012 18:11:42 +0100
Subject: [Cython] jenkins is down
In-Reply-To: <CAKGHGPQYB9F8vT1AEasiba8qjvW732ud5Wv9ds-9afbx8zRM4w@mail.gmail.com>
References: <CAKGHGPQYB9F8vT1AEasiba8qjvW732ud5Wv9ds-9afbx8zRM4w@mail.gmail.com>
Message-ID: <4F47C4CE.5080603@behnel.de>

Vitja Makarov, 24.02.2012 17:56:
> Jenkins app is down now. Before it got down it was raising exceptions.

Ah, sorry, should have posted a short message. The machine it's running on
(sage.math) has serious problems at the moment, so Jenkins will be down
until someone over there manages to fix the server.

Stefan

From stefan_ml at behnel.de  Sat Feb 25 07:46:08 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Feb 2012 07:46:08 +0100
Subject: [Cython] jenkins is down
In-Reply-To: <4F47C4CE.5080603@behnel.de>
References: <CAKGHGPQYB9F8vT1AEasiba8qjvW732ud5Wv9ds-9afbx8zRM4w@mail.gmail.com>
	<4F47C4CE.5080603@behnel.de>
Message-ID: <4F4883B0.3010604@behnel.de>

Stefan Behnel, 24.02.2012 18:11:
> Vitja Makarov, 24.02.2012 17:56:
>> Jenkins app is down now. Before it got down it was raising exceptions.
> 
> Ah, sorry, should have posted a short message. The machine it's running on
> (sage.math) has serious problems at the moment, so Jenkins will be down
> until someone over there manages to fix the server.

Jenkins is up and running again.

Stefan

From vitja.makarov at gmail.com  Sat Feb 25 10:29:43 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 25 Feb 2012 13:29:43 +0400
Subject: [Cython] jenkins is down
In-Reply-To: <4F4883B0.3010604@behnel.de>
References: <CAKGHGPQYB9F8vT1AEasiba8qjvW732ud5Wv9ds-9afbx8zRM4w@mail.gmail.com>
	<4F47C4CE.5080603@behnel.de> <4F4883B0.3010604@behnel.de>
Message-ID: <CAKGHGPQDifdTebc6gNUus-=hHdqx-M_8fyH=AV8c8o2N6enQhg@mail.gmail.com>

2012/2/25 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 24.02.2012 18:11:
>> Vitja Makarov, 24.02.2012 17:56:
>>> Jenkins app is down now. Before it got down it was raising exceptions.
>>
>> Ah, sorry, should have posted a short message. The machine it's running on
>> (sage.math) has serious problems at the moment, so Jenkins will be down
>> until someone over there manages to fix the server.
>
> Jenkins is up and running again.
>

Cool!

-- 
vitja.

From vitja.makarov at gmail.com  Sat Feb 25 12:12:18 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 25 Feb 2012 15:12:18 +0400
Subject: [Cython] cython tests and py3k hash randomization
In-Reply-To: <CAKGHGPRJjcFFVUPXrF1Jm+3cY=QNkoq0mfQ6bbrk5eP4kc9BPA@mail.gmail.com>
References: <CAKGHGPQBiz0=xMR1iDt8ByQ3aZ5=prJDLRS5AR7-yup+TBKN5w@mail.gmail.com>
	<CANg26EUGk-oJDviqhgBPGPaLBKsOSGOK1PnrMKyqX2E1cKb3hQ@mail.gmail.com>
	<CAKGHGPR3Wo4bycZGe1_RBuzQC-jnj+bd1p=Q5NF46rYhBSvmLg@mail.gmail.com>
	<4F474D72.7050504@behnel.de>
	<CAKGHGPRJjcFFVUPXrF1Jm+3cY=QNkoq0mfQ6bbrk5eP4kc9BPA@mail.gmail.com>
Message-ID: <CAKGHGPTBp-8CiGBTeOWKNz9MYCHsPOsVQNPVWL=tkSvstwTzzQ@mail.gmail.com>

2012/2/24 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/24 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 24.02.2012 06:24:
>>> 2012/2/24 mark florisson <markflorisson88 at gmail.com>:
>>>> On 23 February 2012 20:52, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> Recent py3k version has new feature "hash randomization" it solves
>>>>> some security issues.
>>>>> But has some drawbacks, for instance, dict.items() order is now
>>>>> unknown. So it breaks
>>>>> randomly some doctests that rely on exact order of dict items.
>>>>>
>>>>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>>>>> {'b': 2, 'a': 1}
>>>>> vitja at mchome:~/python$ ./py3k/bin/python -c ?"print({'a':1, 'b':2})"
>>>>> {'a': 1, 'b': 2}
>>>>>
>>>>> As a workaround we can set PYTHONHASHSEED environment variable to zero
>>>>> for all cyhon-*-tests targets
>>>>>
>>>>> This also affects generated code it internally depends on order of
>>>>> items returned by dict.items()
>>>>
>>>> Any code or test that relies on dictionary order is wrong, really. So
>>>> I assume any pyregr issues will be fixed by the CPython test suite?
>>
>> Definitely.
>>
>
> Ok, I'll take a look.
>
>>
>>>> If there are any such failing tests in Cython we should simply fix them.
>>>
>>> Yes, you're right but I'm not sure how many tests may be broken. I
>>> don't think we want to fix them before release. Now I've added
>>>
>>> # Disable python hash randomization
>>> export PYTHONHASHSEED=0
>>>
>>> to cython-devel-tests and it worked.
>>
>> That should only be a work-around until all tests are fixed, though.
>>
>>
>>> It's not that easy to fix, here is simple doctest:
>>>
>>> def test_foo():
>>> ? ? """
>>> ? ? >>> test_foo()
>>> ? ? {'a': 1, 'b': 2}
>>> ? ? """
>>> ? ? return {'a': 1, 'b': 2}
>>>
>>> You can't use dicts in doctests anymore there are few options instead:
>>> 1. Compare sorted items(), e.g. sorted(test_foo().items())
>>> 2. Inplace compare test_foo() == {...}
>>
>> I always use 1) because it gives you better test failure output in doctests.
>>
>

I fixed tests.

-- 
vitja.

From stefan_ml at behnel.de  Sat Feb 25 16:35:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Feb 2012 16:35:28 +0100
Subject: [Cython] memoryview test warnings
Message-ID: <4F48FFC0.7050103@behnel.de>

Hi,

I get these warnings in the tests:

"""
compiling (cpp) and running memoryview ...

memoryview.cpp: In function ?PyObject*
__pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*,
PyObject*, PyObject*)?:
memoryview.cpp:19703: warning: comparison between signed and unsigned
integer expressions

memoryview.cpp: At global scope:
memoryview.cpp:10210: warning: ?__Pyx_memviewslice
__pyx_f_10memoryview_func()? defined but not used

compiling (cpp) and running memslice ...

memslice.cpp: In function ?PyObject*
__pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*,
PyObject*, PyObject*)?:
memslice.cpp:39266: warning: comparison between signed and unsigned integer
expressions
"""

Would be nice if they could be fixed for the release.

Also, the memslice test writes output to the test log:

"""
acquired default
acquired Global_A
"""

Normally, all test output should get collected by the doctest runner.

Stefan

From markflorisson88 at gmail.com  Sat Feb 25 17:05:26 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 25 Feb 2012 16:05:26 +0000
Subject: [Cython] memoryview test warnings
In-Reply-To: <4F48FFC0.7050103@behnel.de>
References: <4F48FFC0.7050103@behnel.de>
Message-ID: <CANg26EU+k1DVnxX0896Uhitc_zgyfqJ+xoJEbTk2FoC9a+LdHg@mail.gmail.com>

On 25 February 2012 15:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I get these warnings in the tests:
>
> """
> compiling (cpp) and running memoryview ...
>
> memoryview.cpp: In function ?PyObject*
> __pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*,
> PyObject*, PyObject*)?:
> memoryview.cpp:19703: warning: comparison between signed and unsigned
> integer expressions

This warning is incorrect as the code checks for 'if my_signed_value >
0 and my_signed_value == my_unsigned_value', but the C compiler can't
figure that out.

> memoryview.cpp: At global scope:
> memoryview.cpp:10210: warning: ?__Pyx_memviewslice
> __pyx_f_10memoryview_func()? defined but not used

This one can be fixed, just an unused function.

> compiling (cpp) and running memslice ...
>
> memslice.cpp: In function ?PyObject*
> __pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*,
> PyObject*, PyObject*)?:
> memslice.cpp:39266: warning: comparison between signed and unsigned integer
> expressions
> """
>
> Would be nice if they could be fixed for the release.
>
> Also, the memslice test writes output to the test log:
>
> """
> acquired default
> acquired Global_A
> """

I'm not sure these can be removed. If you pass in a name to a
mockbuffer it will print when it is acquired and released. Any
accidental release of say, a default value in a test function would
result in extra output, thus failing the test and preventing
regressions. These values cannot be part of a global doctest, one
tests for a global variable and the other for a default.

> Normally, all test output should get collected by the doctest runner.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sat Feb 25 17:29:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 25 Feb 2012 16:29:22 +0000
Subject: [Cython] 0.16 release
In-Reply-To: <CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
Message-ID: <CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>

2012/2/24 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
> Hi,
>
> could you please also look at incorporating the following patch before
> releasing 0.16? (if it has not already been merged)
>
> https://github.com/cython/cython/pull/67
>
> It has been more or less validated, but a test case is needed.
>
> This patch makes using C++ templates much more convenient with Cython.
>
> Currently I have to use hacks like the following which looks ugly and make
> the code less readable:
> ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData<float>"
>
> Also thank you for all the work done on Cython, I have been using it (and
> Pyrex before) intensively? for more than 6 years now, and it makes
> integrating Python and C/C++ really convenient.
>
> Thanks in advance
>
> S?bastien
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Ok I merged it and added a test. I also fixed a lot of tests to run
under MSVC on windows. I'm thinking to merge
https://github.com/cython/cython/pull/77, see if everything still
passes on Jenkins, and then pushing out a beta release for 0.16. I
created some release notes, please feel free to add to the page
(especially to the feature and improvements lists), they might be
incomplete: http://wiki.cython.org/ReleaseNotes-0.16

Are there any other last-minute bug fixes pending?

From markflorisson88 at gmail.com  Sun Feb 26 13:08:46 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 26 Feb 2012 12:08:46 +0000
Subject: [Cython] Cython 0.16 Beta 0
Message-ID: <CANg26EXhkDzL6=RXFhgqyVJMFTfoSmKpra4HGsv+e9f58MKaJQ@mail.gmail.com>

The Cython team is pleased to announce the first beta release for the
upcoming release of Cython 0.16. A tarball can be grabbed from here:
http://cython.org/release/Cython-0.16.beta0.tar.gz. This release comes
with several great new features such as better function introspection,
super without arguments, fused types, memoryviews and more, as well as
many bug fixes. For a complete overview see
http://wiki.cython.org/ReleaseNotes-0.16.

This beta release means a feature freeze for 0.16. Have a try everyone
and remember, bonus points to anyone who can break things!

Mark

From sable at users.sourceforge.net  Mon Feb 27 10:29:41 2012
From: sable at users.sourceforge.net (=?ISO-8859-1?Q?S=E9bastien_Sabl=E9_Sabl=E9?=)
Date: Mon, 27 Feb 2012 10:29:41 +0100
Subject: [Cython] 0.16 release
In-Reply-To: <CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
References: <CANg26EUyg0z4iKAT+HU11GsDr1qwqT=8eB+62Mk07x7uwD0==w@mail.gmail.com>
	<CANg26EU4R-ujsJwH2qSG4LoKvmrz_XmE-hokmWRC5C7ZJkgC9Q@mail.gmail.com>
	<CADiQ+QB6FZsWj4ASkqhPW3vN2H9J51T2q_3YSY4_PSkMoT41HQ@mail.gmail.com>
	<CANg26EUSSyy-NGLCLqzAOMVZpkPM4qbJaF1V=V8ybP9g63uJgQ@mail.gmail.com>
	<CADiQ+QB8f09ZHnJ4d=boZvgkp0s171=k85uzZc0Fcv=Z9CXUoQ@mail.gmail.com>
	<CANg26EXxWYauzd-2xLDtst8A6sWEVNvfmkrYYPcmoZbpNx_LrA@mail.gmail.com>
	<CANg26EVEoZBBAj_GMBjEy3BEsLvmqm8f_NVPcwpu1VXoJOYh5g@mail.gmail.com>
	<CAKGHGPTqAzG7tpnckEuhn9K=w3oUXUZkAh2PZq5xdbZ2DGD3+A@mail.gmail.com>
	<CANg26EXbeE-o=Htsvf2BPdyjU7TSUnR189DnscoNqGg0JsJz_A@mail.gmail.com>
	<CAKGHGPT25cv5CnqmQ0csBCqkNgNuupWunzyHs7x4ML52akMw+g@mail.gmail.com>
	<CAKGHGPS2t=vyuwp4o+zRXUB5hem8_zsNxiqbbLisFQtqf6hiRg@mail.gmail.com>
	<CANg26EV+4QXiNaKNaJeFE=-228VbtnFdQQ1s7ba2n=SBb+h2AQ@mail.gmail.com>
	<CAKGHGPR7nrwVYfj=70B-UyHcKwC-U6NH7GhWBXqROKUidkt8rw@mail.gmail.com>
	<CANg26EUiBZQsQOz+B3eC+Fp7z+Yf1CmSu1pjD7i9S3Y89Nh=dw@mail.gmail.com>
	<4F465E97.8080606@behnel.de>
	<CANg26EXoQzS_6giBO-X4+rp7OWWsHhGGo93-NFuaq2o_T-3ogg@mail.gmail.com>
	<CAJqKL-s7vdqZKjHZTomeR8F8njDVhjPV5Pm4e1Z3BvRziMXfqA@mail.gmail.com>
	<CANg26EXLnupiYfOtXO+TL1gLvvk=1o1BU_sZ3NobB3S=3hx=4Q@mail.gmail.com>
Message-ID: <CAJqKL-tpLwxmL2ttyZ=gyTzcNLUSAWd3GH=2xW20=U-OHWqwwQ@mail.gmail.com>

Great, thanks!

2012/2/25 mark florisson <markflorisson88 at gmail.com>

> 2012/2/24 S?bastien Sabl? Sabl? <sable at users.sourceforge.net>:
> > Hi,
> >
> > could you please also look at incorporating the following patch before
> > releasing 0.16? (if it has not already been merged)
> >
> > https://github.com/cython/cython/pull/67
> >
> > It has been more or less validated, but a test case is needed.
> >
> > This patch makes using C++ templates much more convenient with Cython.
> >
> > Currently I have to use hacks like the following which looks ugly and
> make
> > the code less readable:
> > ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData<float>"
> >
> > Also thank you for all the work done on Cython, I have been using it (and
> > Pyrex before) intensively  for more than 6 years now, and it makes
> > integrating Python and C/C++ really convenient.
> >
> > Thanks in advance
> >
> > S?bastien
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
>
> Ok I merged it and added a test. I also fixed a lot of tests to run
> under MSVC on windows. I'm thinking to merge
> https://github.com/cython/cython/pull/77, see if everything still
> passes on Jenkins, and then pushing out a beta release for 0.16. I
> created some release notes, please feel free to add to the page
> (especially to the feature and improvements lists), they might be
> incomplete: http://wiki.cython.org/ReleaseNotes-0.16
>
> Are there any other last-minute bug fixes pending?
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120227/30035b07/attachment.html>

From stefan_ml at behnel.de  Tue Feb 28 10:54:17 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 10:54:17 +0100
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <4F4CA003.7080407@behnel.de>
References: <4F4CA003.7080407@behnel.de>
Message-ID: <4F4CA449.4000108@behnel.de>

I'm going to reimplement this, but not for 0.16 anymore, I'd say.

-------- Original-Message --------
Betreff: Re: [cython-users] What's up with PyEval_InitThreads() in python 2.7?

Mike Cui, 28.02.2012 10:18:
>> Thanks for the test code, you hadn't mentioned that you use a "with gil"
>> block. Could you try the latest github version of Cython?
>>
> 
> Ahh, much better!
> 
>   #if CYTHON_REFNANNY
>   #ifdef WITH_THREAD
>   __pyx_gilstate_save = PyGILState_Ensure();
>   #endif
>   #endif /* CYTHON_REFNANNY */
>   __Pyx_RefNannySetupContext("callback");
>   #if CYTHON_REFNANNY
>   #ifdef WITH_THREAD
>   PyGILState_Release(__pyx_gilstate_save);
>   #endif
>   #endif /* CYTHON_REFNANNY */


Hmm, thanks for posting this - it can be further improved. There's no
reason for code bloat here, it should all just go into the
__Pyx_RefNannySetupContext() macro.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 11:16:16 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 10:16:16 +0000
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <4F4CA449.4000108@behnel.de>
References: <4F4CA003.7080407@behnel.de>
	<4F4CA449.4000108@behnel.de>
Message-ID: <CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>

On 28 February 2012 09:54, Stefan Behnel <stefan_ml at behnel.de> wrote:
> I'm going to reimplement this, but not for 0.16 anymore, I'd say.

That's ok, I fixed it to not acquire the GIL seeing that control flow
obsoletes None initialization. So you might as well move it into the
setup function if you care, the thing is that that would needlessly
acquire the GIL for the common case (when you have the GIL) in the
tests, so it might slow them down. It would be better to create a
__Pyx_RefNannySetupContextNogil() function wrapper. If you're not
running the code as a test the preprocessor would filter out this
bloat though, so it's really not a very big deal anyway.

> -------- Original-Message --------
> Betreff: Re: [cython-users] What's up with PyEval_InitThreads() in python 2.7?
>
> Mike Cui, 28.02.2012 10:18:
>>> Thanks for the test code, you hadn't mentioned that you use a "with gil"
>>> block. Could you try the latest github version of Cython?
>>>
>>
>> Ahh, much better!
>>
>> ? #if CYTHON_REFNANNY
>> ? #ifdef WITH_THREAD
>> ? __pyx_gilstate_save = PyGILState_Ensure();
>> ? #endif
>> ? #endif /* CYTHON_REFNANNY */
>> ? __Pyx_RefNannySetupContext("callback");
>> ? #if CYTHON_REFNANNY
>> ? #ifdef WITH_THREAD
>> ? PyGILState_Release(__pyx_gilstate_save);
>> ? #endif
>> ? #endif /* CYTHON_REFNANNY */
>
>
> Hmm, thanks for posting this - it can be further improved. There's no
> reason for code bloat here, it should all just go into the
> __Pyx_RefNannySetupContext() macro.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Feb 28 11:25:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 11:25:54 +0100
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
Message-ID: <4F4CABB2.5000603@behnel.de>

mark florisson, 28.02.2012 11:16:
> On 28 February 2012 09:54, Stefan Behnel wrote:
>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
> 
> That's ok, I fixed it to not acquire the GIL seeing that control flow
> obsoletes None initialization. So you might as well move it into the
> setup function if you care, the thing is that that would needlessly
> acquire the GIL for the common case (when you have the GIL) in the
> tests, so it might slow them down. It would be better to create a
> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
> running the code as a test the preprocessor would filter out this
> bloat though, so it's really not a very big deal anyway.

I was going to pass a constant flag into the macro that would let the C
compiler do the right thing:

"""
#ifdef WITH_THREAD
  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
          if (acquire_gil) { \
              PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
              PyGILState_Release(__pyx_gilstate_save); \
          } else { \
              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
          }
#else
  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
          __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
#endif
"""

That also gets rid of the need to declare the "save" variable independently.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 11:28:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 10:28:43 +0000
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <4F4CABB2.5000603@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
Message-ID: <CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>

On 28 February 2012 10:25, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 11:16:
>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>
>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>> obsoletes None initialization. So you might as well move it into the
>> setup function if you care, the thing is that that would needlessly
>> acquire the GIL for the common case (when you have the GIL) in the
>> tests, so it might slow them down. It would be better to create a
>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>> running the code as a test the preprocessor would filter out this
>> bloat though, so it's really not a very big deal anyway.
>
> I was going to pass a constant flag into the macro that would let the C
> compiler do the right thing:
>
> """
> #ifdef WITH_THREAD
> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
> ? ? ? ? ?if (acquire_gil) { \
> ? ? ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
> ? ? ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save); \
> ? ? ? ? ?} else { \
> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
> ? ? ? ? ?}
> #else
> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
> ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
> #endif
> """
>
> That also gets rid of the need to declare the "save" variable independently.

I don't think that will work, I think the teardown re-uses the save variable.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Feb 28 11:53:19 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 11:53:19 +0100
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
Message-ID: <4F4CB21F.7060300@behnel.de>

mark florisson, 28.02.2012 11:28:
> On 28 February 2012 10:25, Stefan Behnel wrote:
>> mark florisson, 28.02.2012 11:16:
>>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>>
>>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>>> obsoletes None initialization. So you might as well move it into the
>>> setup function if you care, the thing is that that would needlessly
>>> acquire the GIL for the common case (when you have the GIL) in the
>>> tests, so it might slow them down. It would be better to create a
>>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>>> running the code as a test the preprocessor would filter out this
>>> bloat though, so it's really not a very big deal anyway.
>>
>> I was going to pass a constant flag into the macro that would let the C
>> compiler do the right thing:
>>
>> """
>> #ifdef WITH_THREAD
>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>          if (acquire_gil) { \
>>              PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>              PyGILState_Release(__pyx_gilstate_save); \
>>          } else { \
>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>          }
>> #else
>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>          __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
>> #endif
>> """
>>
>> That also gets rid of the need to declare the "save" variable independently.
> 
> I don't think that will work, I think the teardown re-uses the save variable.

Well, it doesn't *have* to be the same variable, though.

Speaking of that, BTW, there are a couple of places in the code (Nodes.py,
from line 1500 on) that release the GIL, and some of them, specifically
some error cases, look like they are using the wrong conditions to decide
what they need to do in order to achieve that. I think this is most easily
fixed by using the above kind of macro also for the refnanny cleanup call.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 12:20:58 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 11:20:58 +0000
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <4F4CB21F.7060300@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
Message-ID: <CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>

On 28 February 2012 10:53, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 11:28:
>> On 28 February 2012 10:25, Stefan Behnel wrote:
>>> mark florisson, 28.02.2012 11:16:
>>>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>>>
>>>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>>>> obsoletes None initialization. So you might as well move it into the
>>>> setup function if you care, the thing is that that would needlessly
>>>> acquire the GIL for the common case (when you have the GIL) in the
>>>> tests, so it might slow them down. It would be better to create a
>>>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>>>> running the code as a test the preprocessor would filter out this
>>>> bloat though, so it's really not a very big deal anyway.
>>>
>>> I was going to pass a constant flag into the macro that would let the C
>>> compiler do the right thing:
>>>
>>> """
>>> #ifdef WITH_THREAD
>>> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>> ? ? ? ? ?if (acquire_gil) { \
>>> ? ? ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
>>> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>> ? ? ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save); \
>>> ? ? ? ? ?} else { \
>>> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>> ? ? ? ? ?}
>>> #else
>>> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>> ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
>>> #endif
>>> """
>>>
>>> That also gets rid of the need to declare the "save" variable independently.
>>
>> I don't think that will work, I think the teardown re-uses the save variable.
>
> Well, it doesn't *have* to be the same variable, though.
>
> Speaking of that, BTW, there are a couple of places in the code (Nodes.py,
> from line 1500 on) that release the GIL, and some of them,

Sorry, do you mean acquire?

> specifically
> some error cases, look like they are using the wrong conditions to decide
> what they need to do in order to achieve that. I think this is most easily
> fixed by using the above kind of macro also for the refnanny cleanup call.

If you could be more specific as to how the conditions might be wrong,
that would be helpful. The conditions don't just pertain to refnanny,
nogil functions with with gil blocks and object arguments may also
incref their arguments (admittedly, it doesn't check for the borrowed
case, and always assumes the "incref to obtain a new reference" case).

The same goes for cleanup, it acquires the GIL to clean up any
temporaries and new references to function arguments. But there is
indeed room for optimization, e.g. if there are no new references to
object arguments and no error, then the GIL doesn't need to be
acquired. It's all rather subtle and tricky, but I think I wrote
decent tests at the time (but could always use more if the behaviour
were to be changed).
The cleanup could be further improved if each statement or each block
would do its own local cleanup. So any object temporaries would be
cleaned up before leaving the with gil block, etc. Anyway, changing
anything more than just the refnanny macros or functions would
probably have to wait until 0.16.1.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Feb 28 14:50:45 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 14:50:45 +0100
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
Message-ID: <4F4CDBB5.9040203@behnel.de>

mark florisson, 28.02.2012 12:20:
> On 28 February 2012 10:53, Stefan Behnel wrote:
>> mark florisson, 28.02.2012 11:28:
>>> On 28 February 2012 10:25, Stefan Behnel wrote:
>>>> mark florisson, 28.02.2012 11:16:
>>>>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>>>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>>>>
>>>>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>>>>> obsoletes None initialization. So you might as well move it into the
>>>>> setup function if you care, the thing is that that would needlessly
>>>>> acquire the GIL for the common case (when you have the GIL) in the
>>>>> tests, so it might slow them down. It would be better to create a
>>>>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>>>>> running the code as a test the preprocessor would filter out this
>>>>> bloat though, so it's really not a very big deal anyway.
>>>>
>>>> I was going to pass a constant flag into the macro that would let the C
>>>> compiler do the right thing:
>>>>
>>>> """
>>>> #ifdef WITH_THREAD
>>>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>          if (acquire_gil) { \
>>>>              PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
>>>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>              PyGILState_Release(__pyx_gilstate_save); \
>>>>          } else { \
>>>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>          }
>>>> #else
>>>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>          __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
>>>> #endif
>>>> """
>>>>
>>>> That also gets rid of the need to declare the "save" variable independently.
>>>
>>> I don't think that will work, I think the teardown re-uses the save variable.
>>
>> Well, it doesn't *have* to be the same variable, though.
>>
>> Speaking of that, BTW, there are a couple of places in the code (Nodes.py,
>> from line 1500 on) that release the GIL, and some of them,
> 
> Sorry, do you mean acquire?

No, I meant release, at the exit points.


>> specifically
>> some error cases, look like they are using the wrong conditions to decide
>> what they need to do in order to achieve that. I think this is most easily
>> fixed by using the above kind of macro also for the refnanny cleanup call.
> 
> If you could be more specific as to how the conditions might be wrong,
> that would be helpful.

I was thinking about the use of the "acquire_gil_for_refnanny_only" flag at
the end, but maybe I got confused by the immense complexity of the function
exit generation code, with its several places where the GIL can get
acquired. I think this is worth some refactoring...

In any case, the "acquire_gil_for_refnanny_only" flag can easily be used to
trigger the necessary acquire-release code in the refnanny macros (as in
the code above). That's at least a tiny first step to reducing this complexity.


> The conditions don't just pertain to refnanny,
> nogil functions with with gil blocks and object arguments may also
> incref their arguments

I'm well aware of that.


> The same goes for cleanup, it acquires the GIL to clean up any
> temporaries and new references to function arguments. But there is
> indeed room for optimization, e.g. if there are no new references to
> object arguments and no error, then the GIL doesn't need to be
> acquired. It's all rather subtle and tricky, but I think I wrote
> decent tests at the time (but could always use more if the behaviour
> were to be changed).

I'm not currently intending to change the behaviour, just the generated code.


> The cleanup could be further improved if each statement or each block
> would do its own local cleanup. So any object temporaries would be
> cleaned up before leaving the with gil block, etc. Anyway, changing
> anything more than just the refnanny macros or functions would
> probably have to wait until 0.16.1.

Agreed. I'll see how my changes turn out, and then we can decide when to
release them. Given that they don't really fix anything, just simplify the
C code, I don't see a compelling enough reason to get them in for 0.16.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 16:35:14 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 15:35:14 +0000
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <4F4CDBB5.9040203@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
Message-ID: <CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>

On 28 February 2012 13:50, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 12:20:
>> On 28 February 2012 10:53, Stefan Behnel wrote:
>>> mark florisson, 28.02.2012 11:28:
>>>> On 28 February 2012 10:25, Stefan Behnel wrote:
>>>>> mark florisson, 28.02.2012 11:16:
>>>>>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>>>>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>>>>>
>>>>>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>>>>>> obsoletes None initialization. So you might as well move it into the
>>>>>> setup function if you care, the thing is that that would needlessly
>>>>>> acquire the GIL for the common case (when you have the GIL) in the
>>>>>> tests, so it might slow them down. It would be better to create a
>>>>>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>>>>>> running the code as a test the preprocessor would filter out this
>>>>>> bloat though, so it's really not a very big deal anyway.
>>>>>
>>>>> I was going to pass a constant flag into the macro that would let the C
>>>>> compiler do the right thing:
>>>>>
>>>>> """
>>>>> #ifdef WITH_THREAD
>>>>> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>> ? ? ? ? ?if (acquire_gil) { \
>>>>> ? ? ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
>>>>> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>> ? ? ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save); \
>>>>> ? ? ? ? ?} else { \
>>>>> ? ? ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>> ? ? ? ? ?}
>>>>> #else
>>>>> ?#define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>> ? ? ? ? ?__pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
>>>>> #endif
>>>>> """
>>>>>
>>>>> That also gets rid of the need to declare the "save" variable independently.
>>>>
>>>> I don't think that will work, I think the teardown re-uses the save variable.
>>>
>>> Well, it doesn't *have* to be the same variable, though.
>>>
>>> Speaking of that, BTW, there are a couple of places in the code (Nodes.py,
>>> from line 1500 on) that release the GIL, and some of them,
>>
>> Sorry, do you mean acquire?
>
> No, I meant release, at the exit points.
>
>
>>> specifically
>>> some error cases, look like they are using the wrong conditions to decide
>>> what they need to do in order to achieve that. I think this is most easily
>>> fixed by using the above kind of macro also for the refnanny cleanup call.
>>
>> If you could be more specific as to how the conditions might be wrong,
>> that would be helpful.
>
> I was thinking about the use of the "acquire_gil_for_refnanny_only" flag at
> the end, but maybe I got confused by the immense complexity of the function
> exit generation code, with its several places where the GIL can get
> acquired. I think this is worth some refactoring...
>
> In any case, the "acquire_gil_for_refnanny_only" flag can easily be used to
> trigger the necessary acquire-release code in the refnanny macros (as in
> the code above). That's at least a tiny first step to reducing this complexity.
>

Basically, the cleanup code only needs a matching release because the
corresponding acquire is in EnsureGILNode, which wraps the function
body in case of a nogil function with a 'with gil' block. Any changes
to the conditions in FuncDefNode will have to be reflected by the code
that does that wrapping. Changing the refnanny macro for the cleanup
code will not avail anything, as the GIL is already ensured.

>> The conditions don't just pertain to refnanny,
>> nogil functions with with gil blocks and object arguments may also
>> incref their arguments
>
> I'm well aware of that.
>
>
>> The same goes for cleanup, it acquires the GIL to clean up any
>> temporaries and new references to function arguments. But there is
>> indeed room for optimization, e.g. if there are no new references to
>> object arguments and no error, then the GIL doesn't need to be
>> acquired. It's all rather subtle and tricky, but I think I wrote
>> decent tests at the time (but could always use more if the behaviour
>> were to be changed).
>
> I'm not currently intending to change the behaviour, just the generated code.
>
>
>> The cleanup could be further improved if each statement or each block
>> would do its own local cleanup. So any object temporaries would be
>> cleaned up before leaving the with gil block, etc. Anyway, changing
>> anything more than just the refnanny macros or functions would
>> probably have to wait until 0.16.1.
>
> Agreed. I'll see how my changes turn out, and then we can decide when to
> release them. Given that they don't really fix anything, just simplify the
> C code, I don't see a compelling enough reason to get them in for 0.16.

Sure, good luck :) I'll be moving for the next couple of days, so I'll
be rather preoccupied.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From dalcinl at gmail.com  Tue Feb 28 19:19:48 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Feb 2012 15:19:48 -0300
Subject: [Cython] __cinit__ swallowing exceptions
Message-ID: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>

This is something I really have no idea about how to fix, so I'll ask
any of you to do it.

How to reproduce. The quick example below should fail in the second to
last line in test_cinit.py, but it succeeds:

$ cat cinit.pyx
cdef class A:
    def __cinit__(self, A a=None):
        pass

$ cat test_cinit.py
import pyximport; pyximport.install()
from cinit import A
a = A(123)
print (a)

$ python test_cinit.py
/home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
warning: ?__pyx_clear_code_object_cache? defined but not used
[-Wunused-function]
<cinit.A object at 0x7fd28f1b7920>


I think the issue is bad code generation. Please try to follow the
flow below, when ArgTypeTest fails, then "goto  __pyx_L1_error" but
__pyx_r is never initialized to -1, so the function (accidentally)
returns 0 indicating success. BTW, GCC warns about __pyx_r used before
initialization is some other code of mine (more complex, involving
inheritance, and the C compiler inlining code), but not for this
simple example.


static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
PyObject *__pyx_args, PyObject *__pyx_kwds) {
...
  int __pyx_r;
...
  {
   ....  <arg unpacking code>....
  }
  goto __pyx_L4_argument_unpacking_done;
...
  __pyx_L4_argument_unpacking_done:;
  if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
__pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
__pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
...
  __pyx_L1_error:;
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}


BTW, valgrind is able to catch the issue:


$ touch cinit.pyx
$ CFLAGS=-O0 valgrind python test_cinit.py
==6735== Memcheck, a memory error detector
==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==6735== Command: python test_cinit.py
==6735==
/home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
warning: ?__pyx_clear_code_object_cache? defined but not used
[-Wunused-function]
==6735== Conditional jump or move depends on uninitialised value(s)
==6735==    at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
==6735==    by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CDE794: PyEval_EvalFrameEx (in
/usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CE15A4: PyEval_EvalCodeEx (in
/usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CFC7EF: PyRun_FileExFlags (in
/usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
/usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
==6735==    by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
==6735==
<cinit.A object at 0xec47430>
==6735==
==6735== HEAP SUMMARY:
==6735==     in use at exit: 8,870,441 bytes in 53,569 blocks
==6735==   total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
bytes allocated
==6735==
==6735== LEAK SUMMARY:
==6735==    definitely lost: 0 bytes in 0 blocks
==6735==    indirectly lost: 0 bytes in 0 blocks
==6735==      possibly lost: 2,319,018 bytes in 15,424 blocks
==6735==    still reachable: 6,551,423 bytes in 38,145 blocks
==6735==         suppressed: 0 bytes in 0 blocks
==6735== Rerun with --leak-check=full to see details of leaked memory
==6735==
==6735== For counts of detected and suppressed errors, rerun with: -v
==6735== Use --track-origins=yes to see where uninitialised values come from
==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From markflorisson88 at gmail.com  Tue Feb 28 19:51:21 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 18:51:21 +0000
Subject: [Cython] __cinit__ swallowing exceptions
In-Reply-To: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
References: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
Message-ID: <CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>

On 28 February 2012 18:19, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> This is something I really have no idea about how to fix, so I'll ask
> any of you to do it.
>
> How to reproduce. The quick example below should fail in the second to
> last line in test_cinit.py, but it succeeds:
>
> $ cat cinit.pyx
> cdef class A:
> ? ?def __cinit__(self, A a=None):
> ? ? ? ?pass
>
> $ cat test_cinit.py
> import pyximport; pyximport.install()
> from cinit import A
> a = A(123)
> print (a)
>
> $ python test_cinit.py
> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
> warning: ?__pyx_clear_code_object_cache? defined but not used
> [-Wunused-function]
> <cinit.A object at 0x7fd28f1b7920>
>
>
> I think the issue is bad code generation. Please try to follow the
> flow below, when ArgTypeTest fails, then "goto ?__pyx_L1_error" but
> __pyx_r is never initialized to -1, so the function (accidentally)
> returns 0 indicating success. BTW, GCC warns about __pyx_r used before
> initialization is some other code of mine (more complex, involving
> inheritance, and the C compiler inlining code), but not for this
> simple example.
>
>
> static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
> PyObject *__pyx_args, PyObject *__pyx_kwds) {
> ...
> ?int __pyx_r;
> ...
> ?{
> ? .... ?<arg unpacking code>....
> ?}
> ?goto __pyx_L4_argument_unpacking_done;
> ...
> ?__pyx_L4_argument_unpacking_done:;
> ?if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
> __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
> __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
> ...
> ?__pyx_L1_error:;
> ?__pyx_L0:;
> ?__Pyx_RefNannyFinishContext();
> ?return __pyx_r;
> }
>
>
> BTW, valgrind is able to catch the issue:
>
>
> $ touch cinit.pyx
> $ CFLAGS=-O0 valgrind python test_cinit.py
> ==6735== Memcheck, a memory error detector
> ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
> ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
> ==6735== Command: python test_cinit.py
> ==6735==
> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
> warning: ?__pyx_clear_code_object_cache? defined but not used
> [-Wunused-function]
> ==6735== Conditional jump or move depends on uninitialised value(s)
> ==6735== ? ?at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
> ==6735== ? ?by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CDE794: PyEval_EvalFrameEx (in
> /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CE15A4: PyEval_EvalCodeEx (in
> /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CFC7EF: PyRun_FileExFlags (in
> /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
> /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
> ==6735== ? ?by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
> ==6735==
> <cinit.A object at 0xec47430>
> ==6735==
> ==6735== HEAP SUMMARY:
> ==6735== ? ? in use at exit: 8,870,441 bytes in 53,569 blocks
> ==6735== ? total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
> bytes allocated
> ==6735==
> ==6735== LEAK SUMMARY:
> ==6735== ? ?definitely lost: 0 bytes in 0 blocks
> ==6735== ? ?indirectly lost: 0 bytes in 0 blocks
> ==6735== ? ? ?possibly lost: 2,319,018 bytes in 15,424 blocks
> ==6735== ? ?still reachable: 6,551,423 bytes in 38,145 blocks
> ==6735== ? ? ? ? suppressed: 0 bytes in 0 blocks
> ==6735== Rerun with --leak-check=full to see details of leaked memory
> ==6735==
> ==6735== For counts of detected and suppressed errors, rerun with: -v
> ==6735== Use --track-origins=yes to see where uninitialised values come from
> ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
>
>
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Thanks, I fixed it here: https://github.com/markflorisson88/cython

It should probably have more tests, also for other special methods.

From vitja.makarov at gmail.com  Tue Feb 28 19:57:28 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 28 Feb 2012 22:57:28 +0400
Subject: [Cython] __cinit__ swallowing exceptions
In-Reply-To: <CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>
References: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
	<CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>
Message-ID: <CAKGHGPQrYd8OVD6VEazEO2gwB4fQwD_nkMiEB-v4A9HcHNFoSg@mail.gmail.com>

2012/2/28 mark florisson <markflorisson88 at gmail.com>:
> On 28 February 2012 18:19, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> This is something I really have no idea about how to fix, so I'll ask
>> any of you to do it.
>>
>> How to reproduce. The quick example below should fail in the second to
>> last line in test_cinit.py, but it succeeds:
>>
>> $ cat cinit.pyx
>> cdef class A:
>> ? ?def __cinit__(self, A a=None):
>> ? ? ? ?pass
>>
>> $ cat test_cinit.py
>> import pyximport; pyximport.install()
>> from cinit import A
>> a = A(123)
>> print (a)
>>
>> $ python test_cinit.py
>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>> warning: ?__pyx_clear_code_object_cache? defined but not used
>> [-Wunused-function]
>> <cinit.A object at 0x7fd28f1b7920>
>>
>>
>> I think the issue is bad code generation. Please try to follow the
>> flow below, when ArgTypeTest fails, then "goto ?__pyx_L1_error" but
>> __pyx_r is never initialized to -1, so the function (accidentally)
>> returns 0 indicating success. BTW, GCC warns about __pyx_r used before
>> initialization is some other code of mine (more complex, involving
>> inheritance, and the C compiler inlining code), but not for this
>> simple example.
>>
>>
>> static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
>> PyObject *__pyx_args, PyObject *__pyx_kwds) {
>> ...
>> ?int __pyx_r;
>> ...
>> ?{
>> ? .... ?<arg unpacking code>....
>> ?}
>> ?goto __pyx_L4_argument_unpacking_done;
>> ...
>> ?__pyx_L4_argument_unpacking_done:;
>> ?if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
>> __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
>> __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
>> ...
>> ?__pyx_L1_error:;
>> ?__pyx_L0:;
>> ?__Pyx_RefNannyFinishContext();
>> ?return __pyx_r;
>> }
>>
>>
>> BTW, valgrind is able to catch the issue:
>>
>>
>> $ touch cinit.pyx
>> $ CFLAGS=-O0 valgrind python test_cinit.py
>> ==6735== Memcheck, a memory error detector
>> ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>> ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>> ==6735== Command: python test_cinit.py
>> ==6735==
>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>> warning: ?__pyx_clear_code_object_cache? defined but not used
>> [-Wunused-function]
>> ==6735== Conditional jump or move depends on uninitialised value(s)
>> ==6735== ? ?at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
>> ==6735== ? ?by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CDE794: PyEval_EvalFrameEx (in
>> /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CE15A4: PyEval_EvalCodeEx (in
>> /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CFC7EF: PyRun_FileExFlags (in
>> /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
>> /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
>> ==6735== ? ?by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
>> ==6735==
>> <cinit.A object at 0xec47430>
>> ==6735==
>> ==6735== HEAP SUMMARY:
>> ==6735== ? ? in use at exit: 8,870,441 bytes in 53,569 blocks
>> ==6735== ? total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
>> bytes allocated
>> ==6735==
>> ==6735== LEAK SUMMARY:
>> ==6735== ? ?definitely lost: 0 bytes in 0 blocks
>> ==6735== ? ?indirectly lost: 0 bytes in 0 blocks
>> ==6735== ? ? ?possibly lost: 2,319,018 bytes in 15,424 blocks
>> ==6735== ? ?still reachable: 6,551,423 bytes in 38,145 blocks
>> ==6735== ? ? ? ? suppressed: 0 bytes in 0 blocks
>> ==6735== Rerun with --leak-check=full to see details of leaked memory
>> ==6735==
>> ==6735== For counts of detected and suppressed errors, rerun with: -v
>> ==6735== Use --track-origins=yes to see where uninitialised values come from
>> ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
>>
>>
>>
>> --
>> Lisandro Dalcin
>> ---------------
>> CIMEC (INTEC/CONICET-UNL)
>> Predio CONICET-Santa Fe
>> Colectora RN 168 Km 472, Paraje El Pozo
>> 3000 Santa Fe, Argentina
>> Tel: +54-342-4511594 (ext 1011)
>> Tel/Fax: +54-342-4511169
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Thanks, I fixed it here: https://github.com/markflorisson88/cython
>
> It should probably have more tests, also for other special methods.


This bug was introduced by DefNode refactoring.

Mark, your fix isn't correct __pyx_r should be set to error_value() on
error just like DefNode does.

-- 
vitja.

From markflorisson88 at gmail.com  Tue Feb 28 19:59:32 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 18:59:32 +0000
Subject: [Cython] __cinit__ swallowing exceptions
In-Reply-To: <CAKGHGPQrYd8OVD6VEazEO2gwB4fQwD_nkMiEB-v4A9HcHNFoSg@mail.gmail.com>
References: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
	<CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>
	<CAKGHGPQrYd8OVD6VEazEO2gwB4fQwD_nkMiEB-v4A9HcHNFoSg@mail.gmail.com>
Message-ID: <CANg26EVeTJXYsdnPhURjyTXuCNeCs5iowLm-CzX8oBJbmD6FfA@mail.gmail.com>

On 28 February 2012 18:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/2/28 mark florisson <markflorisson88 at gmail.com>:
>> On 28 February 2012 18:19, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>> This is something I really have no idea about how to fix, so I'll ask
>>> any of you to do it.
>>>
>>> How to reproduce. The quick example below should fail in the second to
>>> last line in test_cinit.py, but it succeeds:
>>>
>>> $ cat cinit.pyx
>>> cdef class A:
>>> ? ?def __cinit__(self, A a=None):
>>> ? ? ? ?pass
>>>
>>> $ cat test_cinit.py
>>> import pyximport; pyximport.install()
>>> from cinit import A
>>> a = A(123)
>>> print (a)
>>>
>>> $ python test_cinit.py
>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>> [-Wunused-function]
>>> <cinit.A object at 0x7fd28f1b7920>
>>>
>>>
>>> I think the issue is bad code generation. Please try to follow the
>>> flow below, when ArgTypeTest fails, then "goto ?__pyx_L1_error" but
>>> __pyx_r is never initialized to -1, so the function (accidentally)
>>> returns 0 indicating success. BTW, GCC warns about __pyx_r used before
>>> initialization is some other code of mine (more complex, involving
>>> inheritance, and the C compiler inlining code), but not for this
>>> simple example.
>>>
>>>
>>> static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
>>> PyObject *__pyx_args, PyObject *__pyx_kwds) {
>>> ...
>>> ?int __pyx_r;
>>> ...
>>> ?{
>>> ? .... ?<arg unpacking code>....
>>> ?}
>>> ?goto __pyx_L4_argument_unpacking_done;
>>> ...
>>> ?__pyx_L4_argument_unpacking_done:;
>>> ?if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
>>> __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
>>> __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
>>> ...
>>> ?__pyx_L1_error:;
>>> ?__pyx_L0:;
>>> ?__Pyx_RefNannyFinishContext();
>>> ?return __pyx_r;
>>> }
>>>
>>>
>>> BTW, valgrind is able to catch the issue:
>>>
>>>
>>> $ touch cinit.pyx
>>> $ CFLAGS=-O0 valgrind python test_cinit.py
>>> ==6735== Memcheck, a memory error detector
>>> ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>> ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>>> ==6735== Command: python test_cinit.py
>>> ==6735==
>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>> [-Wunused-function]
>>> ==6735== Conditional jump or move depends on uninitialised value(s)
>>> ==6735== ? ?at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
>>> ==6735== ? ?by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CDE794: PyEval_EvalFrameEx (in
>>> /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CE15A4: PyEval_EvalCodeEx (in
>>> /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CFC7EF: PyRun_FileExFlags (in
>>> /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
>>> /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
>>> ==6735== ? ?by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
>>> ==6735==
>>> <cinit.A object at 0xec47430>
>>> ==6735==
>>> ==6735== HEAP SUMMARY:
>>> ==6735== ? ? in use at exit: 8,870,441 bytes in 53,569 blocks
>>> ==6735== ? total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
>>> bytes allocated
>>> ==6735==
>>> ==6735== LEAK SUMMARY:
>>> ==6735== ? ?definitely lost: 0 bytes in 0 blocks
>>> ==6735== ? ?indirectly lost: 0 bytes in 0 blocks
>>> ==6735== ? ? ?possibly lost: 2,319,018 bytes in 15,424 blocks
>>> ==6735== ? ?still reachable: 6,551,423 bytes in 38,145 blocks
>>> ==6735== ? ? ? ? suppressed: 0 bytes in 0 blocks
>>> ==6735== Rerun with --leak-check=full to see details of leaked memory
>>> ==6735==
>>> ==6735== For counts of detected and suppressed errors, rerun with: -v
>>> ==6735== Use --track-origins=yes to see where uninitialised values come from
>>> ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
>>>
>>>
>>>
>>> --
>>> Lisandro Dalcin
>>> ---------------
>>> CIMEC (INTEC/CONICET-UNL)
>>> Predio CONICET-Santa Fe
>>> Colectora RN 168 Km 472, Paraje El Pozo
>>> 3000 Santa Fe, Argentina
>>> Tel: +54-342-4511594 (ext 1011)
>>> Tel/Fax: +54-342-4511169
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Thanks, I fixed it here: https://github.com/markflorisson88/cython
>>
>> It should probably have more tests, also for other special methods.
>
>
> This bug was introduced by DefNode refactoring.
>
> Mark, your fix isn't correct __pyx_r should be set to error_value() on
> error just like DefNode does.

Right, that would be work for all cases. Could you fix it and push to master?

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Tue Feb 28 20:05:14 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 28 Feb 2012 23:05:14 +0400
Subject: [Cython] __cinit__ swallowing exceptions
In-Reply-To: <CANg26EVeTJXYsdnPhURjyTXuCNeCs5iowLm-CzX8oBJbmD6FfA@mail.gmail.com>
References: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
	<CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>
	<CAKGHGPQrYd8OVD6VEazEO2gwB4fQwD_nkMiEB-v4A9HcHNFoSg@mail.gmail.com>
	<CANg26EVeTJXYsdnPhURjyTXuCNeCs5iowLm-CzX8oBJbmD6FfA@mail.gmail.com>
Message-ID: <CAKGHGPRD7=NpKQx2vsE9kkoPx=bR2-qwv+KVqKsqgoJnCPL0pg@mail.gmail.com>

2012/2/28 mark florisson <markflorisson88 at gmail.com>:
> On 28 February 2012 18:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/2/28 mark florisson <markflorisson88 at gmail.com>:
>>> On 28 February 2012 18:19, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>> This is something I really have no idea about how to fix, so I'll ask
>>>> any of you to do it.
>>>>
>>>> How to reproduce. The quick example below should fail in the second to
>>>> last line in test_cinit.py, but it succeeds:
>>>>
>>>> $ cat cinit.pyx
>>>> cdef class A:
>>>> ? ?def __cinit__(self, A a=None):
>>>> ? ? ? ?pass
>>>>
>>>> $ cat test_cinit.py
>>>> import pyximport; pyximport.install()
>>>> from cinit import A
>>>> a = A(123)
>>>> print (a)
>>>>
>>>> $ python test_cinit.py
>>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>>> [-Wunused-function]
>>>> <cinit.A object at 0x7fd28f1b7920>
>>>>
>>>>
>>>> I think the issue is bad code generation. Please try to follow the
>>>> flow below, when ArgTypeTest fails, then "goto ?__pyx_L1_error" but
>>>> __pyx_r is never initialized to -1, so the function (accidentally)
>>>> returns 0 indicating success. BTW, GCC warns about __pyx_r used before
>>>> initialization is some other code of mine (more complex, involving
>>>> inheritance, and the C compiler inlining code), but not for this
>>>> simple example.
>>>>
>>>>
>>>> static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
>>>> PyObject *__pyx_args, PyObject *__pyx_kwds) {
>>>> ...
>>>> ?int __pyx_r;
>>>> ...
>>>> ?{
>>>> ? .... ?<arg unpacking code>....
>>>> ?}
>>>> ?goto __pyx_L4_argument_unpacking_done;
>>>> ...
>>>> ?__pyx_L4_argument_unpacking_done:;
>>>> ?if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
>>>> __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
>>>> __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
>>>> ...
>>>> ?__pyx_L1_error:;
>>>> ?__pyx_L0:;
>>>> ?__Pyx_RefNannyFinishContext();
>>>> ?return __pyx_r;
>>>> }
>>>>
>>>>
>>>> BTW, valgrind is able to catch the issue:
>>>>
>>>>
>>>> $ touch cinit.pyx
>>>> $ CFLAGS=-O0 valgrind python test_cinit.py
>>>> ==6735== Memcheck, a memory error detector
>>>> ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>>> ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>>>> ==6735== Command: python test_cinit.py
>>>> ==6735==
>>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>>> [-Wunused-function]
>>>> ==6735== Conditional jump or move depends on uninitialised value(s)
>>>> ==6735== ? ?at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
>>>> ==6735== ? ?by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CDE794: PyEval_EvalFrameEx (in
>>>> /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CE15A4: PyEval_EvalCodeEx (in
>>>> /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CFC7EF: PyRun_FileExFlags (in
>>>> /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
>>>> /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
>>>> ==6735== ? ?by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
>>>> ==6735==
>>>> <cinit.A object at 0xec47430>
>>>> ==6735==
>>>> ==6735== HEAP SUMMARY:
>>>> ==6735== ? ? in use at exit: 8,870,441 bytes in 53,569 blocks
>>>> ==6735== ? total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
>>>> bytes allocated
>>>> ==6735==
>>>> ==6735== LEAK SUMMARY:
>>>> ==6735== ? ?definitely lost: 0 bytes in 0 blocks
>>>> ==6735== ? ?indirectly lost: 0 bytes in 0 blocks
>>>> ==6735== ? ? ?possibly lost: 2,319,018 bytes in 15,424 blocks
>>>> ==6735== ? ?still reachable: 6,551,423 bytes in 38,145 blocks
>>>> ==6735== ? ? ? ? suppressed: 0 bytes in 0 blocks
>>>> ==6735== Rerun with --leak-check=full to see details of leaked memory
>>>> ==6735==
>>>> ==6735== For counts of detected and suppressed errors, rerun with: -v
>>>> ==6735== Use --track-origins=yes to see where uninitialised values come from
>>>> ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
>>>>
>>>>
>>>>
>>>> --
>>>> Lisandro Dalcin
>>>> ---------------
>>>> CIMEC (INTEC/CONICET-UNL)
>>>> Predio CONICET-Santa Fe
>>>> Colectora RN 168 Km 472, Paraje El Pozo
>>>> 3000 Santa Fe, Argentina
>>>> Tel: +54-342-4511594 (ext 1011)
>>>> Tel/Fax: +54-342-4511169
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Thanks, I fixed it here: https://github.com/markflorisson88/cython
>>>
>>> It should probably have more tests, also for other special methods.
>>
>>
>> This bug was introduced by DefNode refactoring.
>>
>> Mark, your fix isn't correct __pyx_r should be set to error_value() on
>> error just like DefNode does.
>
> Right, that would be work for all cases. Could you fix it and push to master?
>

Sure.

-- 
vitja.

From stefan_ml at behnel.de  Tue Feb 28 20:14:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 20:14:14 +0100
Subject: [Cython] [cython-users] What's up with PyEval_InitThreads() in
 python 2.7?
In-Reply-To: <CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
Message-ID: <4F4D2786.4050304@behnel.de>

mark florisson, 28.02.2012 16:35:
> On 28 February 2012 13:50, Stefan Behnel wrote:
>> mark florisson, 28.02.2012 12:20:
>>> On 28 February 2012 10:53, Stefan Behnel wrote:
>>>> mark florisson, 28.02.2012 11:28:
>>>>> On 28 February 2012 10:25, Stefan Behnel wrote:
>>>>>> mark florisson, 28.02.2012 11:16:
>>>>>>> On 28 February 2012 09:54, Stefan Behnel wrote:
>>>>>>>> I'm going to reimplement this, but not for 0.16 anymore, I'd say.
>>>>>>>
>>>>>>> That's ok, I fixed it to not acquire the GIL seeing that control flow
>>>>>>> obsoletes None initialization. So you might as well move it into the
>>>>>>> setup function if you care, the thing is that that would needlessly
>>>>>>> acquire the GIL for the common case (when you have the GIL) in the
>>>>>>> tests, so it might slow them down.

Regarding this bit, we might just use a local temp variable to remember
when we have acquired the GIL already. The C compiler should be able to
figure out its value at each point in the code and drop unnecessary GIL
handling code accordingly.


>>>>>>> It would be better to create a
>>>>>>> __Pyx_RefNannySetupContextNogil() function wrapper. If you're not
>>>>>>> running the code as a test the preprocessor would filter out this
>>>>>>> bloat though, so it's really not a very big deal anyway.
>>>>>>
>>>>>> I was going to pass a constant flag into the macro that would let the C
>>>>>> compiler do the right thing:
>>>>>>
>>>>>> """
>>>>>> #ifdef WITH_THREAD
>>>>>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>>>          if (acquire_gil) { \
>>>>>>              PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure(); \
>>>>>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>>>              PyGILState_Release(__pyx_gilstate_save); \
>>>>>>          } else { \
>>>>>>              __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...) \
>>>>>>          }
>>>>>> #else
>>>>>>  #define __Pyx_RefNannySetupContext(name, acquire_gil) \
>>>>>>          __pyx_refnanny = __Pyx_RefNanny->SetupContext((name), ...)
>>>>>> #endif
>>>>>> """
>>>>>>
>>>>>> That also gets rid of the need to declare the "save" variable independently.
>>>>>
>>>>> I don't think that will work, I think the teardown re-uses the save variable.
>>>>
>>>> Well, it doesn't *have* to be the same variable, though.
>>>>
>>>> Speaking of that, BTW, there are a couple of places in the code (Nodes.py,
>>>> from line 1500 on) that release the GIL, and some of them, specifically
>>>> some error cases, look like they are using the wrong conditions to decide
>>>> what they need to do in order to achieve that. I think this is most easily
>>>> fixed by using the above kind of macro also for the refnanny cleanup call.
>>>
>>> If you could be more specific as to how the conditions might be wrong,
>>> that would be helpful.
>>
>> I was thinking about the use of the "acquire_gil_for_refnanny_only" flag at
>> the end, but maybe I got confused by the immense complexity of the function
>> exit generation code, with its several places where the GIL can get
>> acquired. I think this is worth some refactoring...
>>
>> In any case, the "acquire_gil_for_refnanny_only" flag can easily be used to
>> trigger the necessary acquire-release code in the refnanny macros (as in
>> the code above). That's at least a tiny first step to reducing this complexity.
> 
> Basically, the cleanup code only needs a matching release because the
> corresponding acquire is in EnsureGILNode, which wraps the function
> body in case of a nogil function with a 'with gil' block. Any changes
> to the conditions in FuncDefNode will have to be reflected by the code
> that does that wrapping.

Hmm, that sounds a bit fragile, though. I think I'll leave it with the
change of moving the entry code into the macro, just to drop the C code
overhead a little. That was my primary intention anyway.


> Changing the refnanny macro for the cleanup
> code will not avail anything, as the GIL is already ensured.

Right, I noticed that, too.

Stefan

From vitja.makarov at gmail.com  Tue Feb 28 20:35:54 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 28 Feb 2012 23:35:54 +0400
Subject: [Cython] __cinit__ swallowing exceptions
In-Reply-To: <CAKGHGPRD7=NpKQx2vsE9kkoPx=bR2-qwv+KVqKsqgoJnCPL0pg@mail.gmail.com>
References: <CAEcYPwAsg3CHViUwf-=1eAF1az5-ekBrBR7OedzqTJm4cE9vZQ@mail.gmail.com>
	<CANg26EX_kGPHwd02UOVNX67u0QWX1tqK3wTo7BTf_uwiS=m8hA@mail.gmail.com>
	<CAKGHGPQrYd8OVD6VEazEO2gwB4fQwD_nkMiEB-v4A9HcHNFoSg@mail.gmail.com>
	<CANg26EVeTJXYsdnPhURjyTXuCNeCs5iowLm-CzX8oBJbmD6FfA@mail.gmail.com>
	<CAKGHGPRD7=NpKQx2vsE9kkoPx=bR2-qwv+KVqKsqgoJnCPL0pg@mail.gmail.com>
Message-ID: <CAKGHGPRr5ux-h5aDER2mJFWVJPXNYKBg6z1CqJO+X1g9+mkPdw@mail.gmail.com>

2012/2/28 Vitja Makarov <vitja.makarov at gmail.com>:
> 2012/2/28 mark florisson <markflorisson88 at gmail.com>:
>> On 28 February 2012 18:57, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2012/2/28 mark florisson <markflorisson88 at gmail.com>:
>>>> On 28 February 2012 18:19, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>> This is something I really have no idea about how to fix, so I'll ask
>>>>> any of you to do it.
>>>>>
>>>>> How to reproduce. The quick example below should fail in the second to
>>>>> last line in test_cinit.py, but it succeeds:
>>>>>
>>>>> $ cat cinit.pyx
>>>>> cdef class A:
>>>>> ? ?def __cinit__(self, A a=None):
>>>>> ? ? ? ?pass
>>>>>
>>>>> $ cat test_cinit.py
>>>>> import pyximport; pyximport.install()
>>>>> from cinit import A
>>>>> a = A(123)
>>>>> print (a)
>>>>>
>>>>> $ python test_cinit.py
>>>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>>>> [-Wunused-function]
>>>>> <cinit.A object at 0x7fd28f1b7920>
>>>>>
>>>>>
>>>>> I think the issue is bad code generation. Please try to follow the
>>>>> flow below, when ArgTypeTest fails, then "goto ?__pyx_L1_error" but
>>>>> __pyx_r is never initialized to -1, so the function (accidentally)
>>>>> returns 0 indicating success. BTW, GCC warns about __pyx_r used before
>>>>> initialization is some other code of mine (more complex, involving
>>>>> inheritance, and the C compiler inlining code), but not for this
>>>>> simple example.
>>>>>
>>>>>
>>>>> static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self,
>>>>> PyObject *__pyx_args, PyObject *__pyx_kwds) {
>>>>> ...
>>>>> ?int __pyx_r;
>>>>> ...
>>>>> ?{
>>>>> ? .... ?<arg unpacking code>....
>>>>> ?}
>>>>> ?goto __pyx_L4_argument_unpacking_done;
>>>>> ...
>>>>> ?__pyx_L4_argument_unpacking_done:;
>>>>> ?if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a),
>>>>> __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0];
>>>>> __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
>>>>> ...
>>>>> ?__pyx_L1_error:;
>>>>> ?__pyx_L0:;
>>>>> ?__Pyx_RefNannyFinishContext();
>>>>> ?return __pyx_r;
>>>>> }
>>>>>
>>>>>
>>>>> BTW, valgrind is able to catch the issue:
>>>>>
>>>>>
>>>>> $ touch cinit.pyx
>>>>> $ CFLAGS=-O0 valgrind python test_cinit.py
>>>>> ==6735== Memcheck, a memory error detector
>>>>> ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>>>> ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>>>>> ==6735== Command: python test_cinit.py
>>>>> ==6735==
>>>>> /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13:
>>>>> warning: ?__pyx_clear_code_object_cache? defined but not used
>>>>> [-Wunused-function]
>>>>> ==6735== Conditional jump or move depends on uninitialised value(s)
>>>>> ==6735== ? ?at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548)
>>>>> ==6735== ? ?by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CDE794: PyEval_EvalFrameEx (in
>>>>> /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CE15A4: PyEval_EvalCodeEx (in
>>>>> /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CFC7EF: PyRun_FileExFlags (in
>>>>> /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in
>>>>> /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0)
>>>>> ==6735== ? ?by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so)
>>>>> ==6735==
>>>>> <cinit.A object at 0xec47430>
>>>>> ==6735==
>>>>> ==6735== HEAP SUMMARY:
>>>>> ==6735== ? ? in use at exit: 8,870,441 bytes in 53,569 blocks
>>>>> ==6735== ? total heap usage: 391,334 allocs, 337,765 frees, 94,515,287
>>>>> bytes allocated
>>>>> ==6735==
>>>>> ==6735== LEAK SUMMARY:
>>>>> ==6735== ? ?definitely lost: 0 bytes in 0 blocks
>>>>> ==6735== ? ?indirectly lost: 0 bytes in 0 blocks
>>>>> ==6735== ? ? ?possibly lost: 2,319,018 bytes in 15,424 blocks
>>>>> ==6735== ? ?still reachable: 6,551,423 bytes in 38,145 blocks
>>>>> ==6735== ? ? ? ? suppressed: 0 bytes in 0 blocks
>>>>> ==6735== Rerun with --leak-check=full to see details of leaked memory
>>>>> ==6735==
>>>>> ==6735== For counts of detected and suppressed errors, rerun with: -v
>>>>> ==6735== Use --track-origins=yes to see where uninitialised values come from
>>>>> ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lisandro Dalcin
>>>>> ---------------
>>>>> CIMEC (INTEC/CONICET-UNL)
>>>>> Predio CONICET-Santa Fe
>>>>> Colectora RN 168 Km 472, Paraje El Pozo
>>>>> 3000 Santa Fe, Argentina
>>>>> Tel: +54-342-4511594 (ext 1011)
>>>>> Tel/Fax: +54-342-4511169
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>> Thanks, I fixed it here: https://github.com/markflorisson88/cython
>>>>
>>>> It should probably have more tests, also for other special methods.
>>>
>>>
>>> This bug was introduced by DefNode refactoring.
>>>
>>> Mark, your fix isn't correct __pyx_r should be set to error_value() on
>>> error just like DefNode does.
>>
>> Right, that would be work for all cases. Could you fix it and push to master?
>>
>
> Sure.
>

I fixed it here:
https://github.com/cython/cython/commit/bef95224ad130b4b4680c283c12bea7be79328e5

Thanks!

-- 
vitja.

From stefan_ml at behnel.de  Tue Feb 28 20:58:23 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 20:58:23 +0100
Subject: [Cython] GIL handling C code (was: [cython-users] What's up
 with PyEval_InitThreads() in python 2.7?)
In-Reply-To: <CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
Message-ID: <4F4D31DF.4020306@behnel.de>

mark florisson, 28.02.2012 16:35:
> Basically, the cleanup code only needs a matching release because the
> corresponding acquire is in EnsureGILNode, which wraps the function
> body in case of a nogil function with a 'with gil' block. Any changes
> to the conditions in FuncDefNode will have to be reflected by the code
> that does that wrapping. Changing the refnanny macro for the cleanup
> code will not avail anything, as the GIL is already ensured.

Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
test is being duplicated. I noticed this when I moved the refnanny's GIL
state into a block local variable and that broke the C code. Basically, the
with-gil block had declared the variable in its own block, but was then
trying to access that variable in a second finally clause, further down and
outside of the with-gil block.

Looks like a bug to me.

Stefan

From stefan_ml at behnel.de  Tue Feb 28 21:19:11 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 21:19:11 +0100
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D31DF.4020306@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
Message-ID: <4F4D36BF.4040000@behnel.de>

Stefan Behnel, 28.02.2012 20:58:
> mark florisson, 28.02.2012 16:35:
>> Basically, the cleanup code only needs a matching release because the
>> corresponding acquire is in EnsureGILNode, which wraps the function
>> body in case of a nogil function with a 'with gil' block. Any changes
>> to the conditions in FuncDefNode will have to be reflected by the code
>> that does that wrapping. Changing the refnanny macro for the cleanup
>> code will not avail anything, as the GIL is already ensured.
> 
> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
> test is being duplicated. I noticed this when I moved the refnanny's GIL
> state into a block local variable and that broke the C code. Basically, the
> with-gil block had declared the variable in its own block, but was then
> trying to access that variable in a second finally clause, further down and
> outside of the with-gil block.

Hmm, guess I got confused again...

So, the code is this:

"""
       /*finally:*/ {
          int __pyx_why;
          __pyx_why = 0; goto __pyx_L8;
          __pyx_L7: __pyx_why = 4; goto __pyx_L8;
          __pyx_L8:;
          #ifdef WITH_THREAD
          PyGILState_Release(__pyx_gilstate_save);
          #endif
          switch (__pyx_why) {
            case 4: goto __pyx_L4;
          }
        }
    }
  }
  /*finally:*/ {
    int __pyx_why;
    __pyx_why = 0; goto __pyx_L5;
    __pyx_L4: __pyx_why = 4; goto __pyx_L5;
    __pyx_L5:;
    #ifdef WITH_THREAD
    __pyx_gilstate_save = PyGILState_Ensure();
    #endif
    switch (__pyx_why) {
      case 4: goto __pyx_L1_error;
    }
  }

  goto __pyx_L0;
  __pyx_L1_error:;
  __Pyx_XDECREF(__pyx_t_1);
  __Pyx_XDECREF(__pyx_t_2);
  __Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
__pyx_clineno, __pyx_lineno, __pyx_filename);
  __pyx_L0:;
  #ifdef WITH_THREAD
  PyGILState_Release(__pyx_gilstate_save);
  #endif
}
"""

The first "finally" block is inside of the with-gil block, whereas the
second is outside. In the second, the GIL is reacquired, and I guess that's
for cleaning up temps in the error case, right? I see the problem that it
can't be acquired after jumping to the exit labels (error/return) in the
current code layout, but wouldn't it make sense to generate this code
instead (starting at the second finally clause):

"""
  /*finally:*/ {
    int __pyx_why;
    __pyx_why = 0; goto __pyx_L5;
    __pyx_L4: __pyx_why = 4; goto __pyx_L5;
    __pyx_L5:;
    switch (__pyx_why) {
      case 4: goto __pyx_L1_error;
    }
  }

  goto __pyx_L0;
  __pyx_L1_error:;
  #ifdef WITH_THREAD
  __pyx_gilstate_save = PyGILState_Ensure();
  #endif
  __Pyx_XDECREF(__pyx_t_1);
  __Pyx_XDECREF(__pyx_t_2);
  __Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
__pyx_clineno, __pyx_lineno, __pyx_filename);
  #ifdef WITH_THREAD
  PyGILState_Release(__pyx_gilstate_save);
  #endif
  return;   /* <- error return here! */
  __pyx_L0:;
}
"""

Meaning: split the return code paths and let them independently acquire the
GIL if needed? That would at least relieve the outer finally from having to
care about the GIL and having to rely on "someone" to declare the GIL
variable and eventually release it.

Or, alternatively, add an additional exit path to the finally block that
jumps to a label that acquires the GIL before going on to the original
label. Something along those lines...

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 21:20:17 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 20:20:17 +0000
Subject: [Cython] GIL handling C code (was: [cython-users] What's up
 with PyEval_InitThreads() in python 2.7?)
In-Reply-To: <4F4D31DF.4020306@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
Message-ID: <CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>

On 28 February 2012 19:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 16:35:
>> Basically, the cleanup code only needs a matching release because the
>> corresponding acquire is in EnsureGILNode, which wraps the function
>> body in case of a nogil function with a 'with gil' block. Any changes
>> to the conditions in FuncDefNode will have to be reflected by the code
>> that does that wrapping. Changing the refnanny macro for the cleanup
>> code will not avail anything, as the GIL is already ensured.
>
> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
> test is being duplicated. I noticed this when I moved the refnanny's GIL
> state into a block local variable and that broke the C code. Basically, the
> with-gil block had declared the variable in its own block, but was then
> trying to access that variable in a second finally clause, further down and
> outside of the with-gil block.
>
> Looks like a bug to me.

That's not a bug, that's how it is implemented. At setup, a variable
__pyx_gilstate_save is declared, which is also used for teardown. Any
with GIL blocks use C block scoping to do the same thing. The with
block is itself a try/finally, so you get a try/finally wrapped in a
try/finally. The code uses try/finally for it's way of trapping
control flow, allowing some code to execute and resuming control flow
afterwards.


> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb 28 22:08:34 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:08:34 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D36BF.4040000@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de> <4F4D36BF.4040000@behnel.de>
Message-ID: <CANg26EVbUbsoHWF+7Jcex1Ksc24O5AFvtAt5Q-f4D+gSjLh=4Q@mail.gmail.com>

On 28 February 2012 20:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 28.02.2012 20:58:
>> mark florisson, 28.02.2012 16:35:
>>> Basically, the cleanup code only needs a matching release because the
>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>> body in case of a nogil function with a 'with gil' block. Any changes
>>> to the conditions in FuncDefNode will have to be reflected by the code
>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>> code will not avail anything, as the GIL is already ensured.
>>
>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>> state into a block local variable and that broke the C code. Basically, the
>> with-gil block had declared the variable in its own block, but was then
>> trying to access that variable in a second finally clause, further down and
>> outside of the with-gil block.
>
> Hmm, guess I got confused again...
>
> So, the code is this:
>
> """
> ? ? ? /*finally:*/ {
> ? ? ? ? ?int __pyx_why;
> ? ? ? ? ?__pyx_why = 0; goto __pyx_L8;
> ? ? ? ? ?__pyx_L7: __pyx_why = 4; goto __pyx_L8;
> ? ? ? ? ?__pyx_L8:;
> ? ? ? ? ?#ifdef WITH_THREAD
> ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save);
> ? ? ? ? ?#endif
> ? ? ? ? ?switch (__pyx_why) {
> ? ? ? ? ? ?case 4: goto __pyx_L4;
> ? ? ? ? ?}
> ? ? ? ?}
> ? ?}
> ?}
> ?/*finally:*/ {
> ? ?int __pyx_why;
> ? ?__pyx_why = 0; goto __pyx_L5;
> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
> ? ?__pyx_L5:;
> ? ?#ifdef WITH_THREAD
> ? ?__pyx_gilstate_save = PyGILState_Ensure();
> ? ?#endif
> ? ?switch (__pyx_why) {
> ? ? ?case 4: goto __pyx_L1_error;
> ? ?}
> ?}
>
> ?goto __pyx_L0;
> ?__pyx_L1_error:;
> ?__Pyx_XDECREF(__pyx_t_1);
> ?__Pyx_XDECREF(__pyx_t_2);
> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
> __pyx_clineno, __pyx_lineno, __pyx_filename);
> ?__pyx_L0:;
> ?#ifdef WITH_THREAD
> ?PyGILState_Release(__pyx_gilstate_save);
> ?#endif
> }
> """
>
> The first "finally" block is inside of the with-gil block, whereas the
> second is outside. In the second, the GIL is reacquired, and I guess that's
> for cleaning up temps in the error case, right? I see the problem that it
> can't be acquired after jumping to the exit labels (error/return) in the
> current code layout, but wouldn't it make sense to generate this code
> instead (starting at the second finally clause):

This code could certainly be optimized, I just never got around to
implementing it.

> """
> ?/*finally:*/ {
> ? ?int __pyx_why;
> ? ?__pyx_why = 0; goto __pyx_L5;
> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
> ? ?__pyx_L5:;
> ? ?switch (__pyx_why) {
> ? ? ?case 4: goto __pyx_L1_error;
> ? ?}
> ?}
>
> ?goto __pyx_L0;
> ?__pyx_L1_error:;
> ?#ifdef WITH_THREAD
> ?__pyx_gilstate_save = PyGILState_Ensure();
> ?#endif
> ?__Pyx_XDECREF(__pyx_t_1);
> ?__Pyx_XDECREF(__pyx_t_2);
> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
> __pyx_clineno, __pyx_lineno, __pyx_filename);
> ?#ifdef WITH_THREAD
> ?PyGILState_Release(__pyx_gilstate_save);
> ?#endif
> ?return; ? /* <- error return here! */
> ?__pyx_L0:;
> }
> """
>
> Meaning: split the return code paths and let them independently acquire the
> GIL if needed? That would at least relieve the outer finally from having to
> care about the GIL and having to rely on "someone" to declare the GIL
> variable and eventually release it.

(Sorry, dinner). In this case, that would be valid. Normally though,
the exception case should fall through to the normal case, so I think
what you'd want is

    ...
    goto L0;
L1:
    PyGILState_Ensure();
    /* error code */
    __pyx_r = ...;
    goto L2;
L0:
    PyGILState_Ensure();
goto L2:
    /* normal code */
    return __pyx_r;
}

If the entire body is a 'with gil' block, it'd probably be easiest to
transform the nogil function into a 'with gil' function. If you want
to optimize for "paths going from with gil blocks directly into
function cleanup code" (i.e., no intermediate nogil try/finally or
nested with gil blocks), then you could use two more labels in the
code above that bypass the acquire.

> Or, alternatively, add an additional exit path to the finally block that
> jumps to a label that acquires the GIL before going on to the original
> label. Something along those lines...
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Feb 28 22:08:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 22:08:54 +0100
Subject: [Cython] GIL handling C code
In-Reply-To: <CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
	<CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
Message-ID: <4F4D4266.2070109@behnel.de>

mark florisson, 28.02.2012 21:20:
> On 28 February 2012 19:58, Stefan Behnel wrote:
>> mark florisson, 28.02.2012 16:35:
>>> Basically, the cleanup code only needs a matching release because the
>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>> body in case of a nogil function with a 'with gil' block. Any changes
>>> to the conditions in FuncDefNode will have to be reflected by the code
>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>> code will not avail anything, as the GIL is already ensured.
>>
>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>> state into a block local variable and that broke the C code. Basically, the
>> with-gil block had declared the variable in its own block, but was then
>> trying to access that variable in a second finally clause, further down and
>> outside of the with-gil block.
>>
>> Looks like a bug to me.
> 
> That's not a bug, that's how it is implemented. At setup, a variable
> __pyx_gilstate_save is declared, which is also used for teardown. Any
> with GIL blocks use C block scoping to do the same thing. The with
> block is itself a try/finally, so you get a try/finally wrapped in a
> try/finally. The code uses try/finally for it's way of trapping
> control flow, allowing some code to execute and resuming control flow
> afterwards.

Ok, so what really got me confused is that the code used the variable
"acquire_gil_for_refnanny_only" in places that didn't have anything to do
with the refnanny. Similarly, "acquire_gil_for_var_decls_only" was used for
cleanup even though the GIL had already been released if this flag was set,
way before the end of the function.

I think I fixed both issues in the patch I attached. At least, it still
passes the gil related tests and doesn't raise any C compiler warnings
about the GIL state variable being unused.

Does this look about right?

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: refnanny_rework.patch
Type: text/x-patch
Size: 3897 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120228/4f43784a/attachment.bin>

From markflorisson88 at gmail.com  Tue Feb 28 22:09:15 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:09:15 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <CANg26EVbUbsoHWF+7Jcex1Ksc24O5AFvtAt5Q-f4D+gSjLh=4Q@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de> <4F4D36BF.4040000@behnel.de>
	<CANg26EVbUbsoHWF+7Jcex1Ksc24O5AFvtAt5Q-f4D+gSjLh=4Q@mail.gmail.com>
Message-ID: <CANg26EVRmBF-2=SYpF7_s_tkqV9_Km2tW-Woy_MVVmH+LPDVfQ@mail.gmail.com>

On 28 February 2012 21:08, mark florisson <markflorisson88 at gmail.com> wrote:
> On 28 February 2012 20:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Stefan Behnel, 28.02.2012 20:58:
>>> mark florisson, 28.02.2012 16:35:
>>>> Basically, the cleanup code only needs a matching release because the
>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>> code will not avail anything, as the GIL is already ensured.
>>>
>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>> state into a block local variable and that broke the C code. Basically, the
>>> with-gil block had declared the variable in its own block, but was then
>>> trying to access that variable in a second finally clause, further down and
>>> outside of the with-gil block.
>>
>> Hmm, guess I got confused again...
>>
>> So, the code is this:
>>
>> """
>> ? ? ? /*finally:*/ {
>> ? ? ? ? ?int __pyx_why;
>> ? ? ? ? ?__pyx_why = 0; goto __pyx_L8;
>> ? ? ? ? ?__pyx_L7: __pyx_why = 4; goto __pyx_L8;
>> ? ? ? ? ?__pyx_L8:;
>> ? ? ? ? ?#ifdef WITH_THREAD
>> ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save);
>> ? ? ? ? ?#endif
>> ? ? ? ? ?switch (__pyx_why) {
>> ? ? ? ? ? ?case 4: goto __pyx_L4;
>> ? ? ? ? ?}
>> ? ? ? ?}
>> ? ?}
>> ?}
>> ?/*finally:*/ {
>> ? ?int __pyx_why;
>> ? ?__pyx_why = 0; goto __pyx_L5;
>> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
>> ? ?__pyx_L5:;
>> ? ?#ifdef WITH_THREAD
>> ? ?__pyx_gilstate_save = PyGILState_Ensure();
>> ? ?#endif
>> ? ?switch (__pyx_why) {
>> ? ? ?case 4: goto __pyx_L1_error;
>> ? ?}
>> ?}
>>
>> ?goto __pyx_L0;
>> ?__pyx_L1_error:;
>> ?__Pyx_XDECREF(__pyx_t_1);
>> ?__Pyx_XDECREF(__pyx_t_2);
>> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>> ?__pyx_L0:;
>> ?#ifdef WITH_THREAD
>> ?PyGILState_Release(__pyx_gilstate_save);
>> ?#endif
>> }
>> """
>>
>> The first "finally" block is inside of the with-gil block, whereas the
>> second is outside. In the second, the GIL is reacquired, and I guess that's
>> for cleaning up temps in the error case, right? I see the problem that it
>> can't be acquired after jumping to the exit labels (error/return) in the
>> current code layout, but wouldn't it make sense to generate this code
>> instead (starting at the second finally clause):
>
> This code could certainly be optimized, I just never got around to
> implementing it.
>
>> """
>> ?/*finally:*/ {
>> ? ?int __pyx_why;
>> ? ?__pyx_why = 0; goto __pyx_L5;
>> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
>> ? ?__pyx_L5:;
>> ? ?switch (__pyx_why) {
>> ? ? ?case 4: goto __pyx_L1_error;
>> ? ?}
>> ?}
>>
>> ?goto __pyx_L0;
>> ?__pyx_L1_error:;
>> ?#ifdef WITH_THREAD
>> ?__pyx_gilstate_save = PyGILState_Ensure();
>> ?#endif
>> ?__Pyx_XDECREF(__pyx_t_1);
>> ?__Pyx_XDECREF(__pyx_t_2);
>> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>> ?#ifdef WITH_THREAD
>> ?PyGILState_Release(__pyx_gilstate_save);
>> ?#endif
>> ?return; ? /* <- error return here! */
>> ?__pyx_L0:;
>> }
>> """
>>
>> Meaning: split the return code paths and let them independently acquire the
>> GIL if needed? That would at least relieve the outer finally from having to
>> care about the GIL and having to rely on "someone" to declare the GIL
>> variable and eventually release it.
>
> (Sorry, dinner). In this case, that would be valid. Normally though,
> the exception case should fall through to the normal case, so I think
> what you'd want is
>
> ? ?...
> ? ?goto L0;
> L1:
> ? ?PyGILState_Ensure();
> ? ?/* error code */
> ? ?__pyx_r = ...;
> ? ?goto L2;
> L0:
> ? ?PyGILState_Ensure();
> goto L2:
> ? ?/* normal code */
> ? ?return __pyx_r;
> }

And there should be a release just before the return of course :)

> If the entire body is a 'with gil' block, it'd probably be easiest to
> transform the nogil function into a 'with gil' function. If you want
> to optimize for "paths going from with gil blocks directly into
> function cleanup code" (i.e., no intermediate nogil try/finally or
> nested with gil blocks), then you could use two more labels in the
> code above that bypass the acquire.
>
>> Or, alternatively, add an additional exit path to the finally block that
>> jumps to a label that acquires the GIL before going on to the original
>> label. Something along those lines...
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Feb 28 22:19:11 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:19:11 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D4266.2070109@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
	<CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
	<4F4D4266.2070109@behnel.de>
Message-ID: <CANg26EXG=EywYwA4uiqtFmgR8RFbOqZE19a_gXiLHV+0aaNxDw@mail.gmail.com>

On 28 February 2012 21:08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 21:20:
>> On 28 February 2012 19:58, Stefan Behnel wrote:
>>> mark florisson, 28.02.2012 16:35:
>>>> Basically, the cleanup code only needs a matching release because the
>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>> code will not avail anything, as the GIL is already ensured.
>>>
>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>> state into a block local variable and that broke the C code. Basically, the
>>> with-gil block had declared the variable in its own block, but was then
>>> trying to access that variable in a second finally clause, further down and
>>> outside of the with-gil block.
>>>
>>> Looks like a bug to me.
>>
>> That's not a bug, that's how it is implemented. At setup, a variable
>> __pyx_gilstate_save is declared, which is also used for teardown. Any
>> with GIL blocks use C block scoping to do the same thing. The with
>> block is itself a try/finally, so you get a try/finally wrapped in a
>> try/finally. The code uses try/finally for it's way of trapping
>> control flow, allowing some code to execute and resuming control flow
>> afterwards.
>
> Ok, so what really got me confused is that the code used the variable
> "acquire_gil_for_refnanny_only" in places that didn't have anything to do
> with the refnanny. Similarly, "acquire_gil_for_var_decls_only" was used for
> cleanup even though the GIL had already been released if this flag was set,
> way before the end of the function.
>
> I think I fixed both issues in the patch I attached. At least, it still
> passes the gil related tests and doesn't raise any C compiler warnings
> about the GIL state variable being unused.
>
> Does this look about right?

It looks right to me, yeah. (<pedantic>I prefer to format bools
directly with %d, as they are a subclass of int anyway</pedantic>). I
think in general the EnsureGILNode should have been mentioned in the
code generation function of FuncDefNode, which makes it easier to
figure out what is going on. The documentation is currently at the
wrapping site in AnalyseDeclarationsTransform.

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Tue Feb 28 22:22:19 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:22:19 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D4266.2070109@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
	<CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
	<4F4D4266.2070109@behnel.de>
Message-ID: <CANg26EURk4-myU--SL5tD1g5k_KRxoGhP1hjppktmhOcWDrRiQ@mail.gmail.com>

On 28 February 2012 21:08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 21:20:
>> On 28 February 2012 19:58, Stefan Behnel wrote:
>>> mark florisson, 28.02.2012 16:35:
>>>> Basically, the cleanup code only needs a matching release because the
>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>> code will not avail anything, as the GIL is already ensured.
>>>
>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>> state into a block local variable and that broke the C code. Basically, the
>>> with-gil block had declared the variable in its own block, but was then
>>> trying to access that variable in a second finally clause, further down and
>>> outside of the with-gil block.
>>>
>>> Looks like a bug to me.
>>
>> That's not a bug, that's how it is implemented. At setup, a variable
>> __pyx_gilstate_save is declared, which is also used for teardown. Any
>> with GIL blocks use C block scoping to do the same thing. The with
>> block is itself a try/finally, so you get a try/finally wrapped in a
>> try/finally. The code uses try/finally for it's way of trapping
>> control flow, allowing some code to execute and resuming control flow
>> afterwards.
>
> Ok, so what really got me confused is that the code used the variable
> "acquire_gil_for_refnanny_only" in places that didn't have anything to do
> with the refnanny. Similarly, "acquire_gil_for_var_decls_only" was used for
> cleanup even though the GIL had already been released if this flag was set,
> way before the end of the function.

Yeah those names weren't very apt, they are useful for the setup code
but don't make sense for the teardown code :)

> I think I fixed both issues in the patch I attached. At least, it still
> passes the gil related tests and doesn't raise any C compiler warnings
> about the GIL state variable being unused.
>
> Does this look about right?
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From stefan_ml at behnel.de  Tue Feb 28 22:22:23 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 22:22:23 +0100
Subject: [Cython] GIL handling C code
In-Reply-To: <CANg26EVRmBF-2=SYpF7_s_tkqV9_Km2tW-Woy_MVVmH+LPDVfQ@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de> <4F4D36BF.4040000@behnel.de>
	<CANg26EVbUbsoHWF+7Jcex1Ksc24O5AFvtAt5Q-f4D+gSjLh=4Q@mail.gmail.com>
	<CANg26EVRmBF-2=SYpF7_s_tkqV9_Km2tW-Woy_MVVmH+LPDVfQ@mail.gmail.com>
Message-ID: <4F4D458F.3000602@behnel.de>

mark florisson, 28.02.2012 22:09:
> On 28 February 2012 21:08, mark florisson wrote:
>> On 28 February 2012 20:19, Stefan Behnel wrote:
>>> Stefan Behnel, 28.02.2012 20:58:
>>>> mark florisson, 28.02.2012 16:35:
>>>>> Basically, the cleanup code only needs a matching release because the
>>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>>> code will not avail anything, as the GIL is already ensured.
>>>>
>>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>>> state into a block local variable and that broke the C code. Basically, the
>>>> with-gil block had declared the variable in its own block, but was then
>>>> trying to access that variable in a second finally clause, further down and
>>>> outside of the with-gil block.
>>>
>>> Hmm, guess I got confused again...
>>>
>>> So, the code is this:
>>>
>>> """
>>>       /*finally:*/ {
>>>          int __pyx_why;
>>>          __pyx_why = 0; goto __pyx_L8;
>>>          __pyx_L7: __pyx_why = 4; goto __pyx_L8;
>>>          __pyx_L8:;
>>>          #ifdef WITH_THREAD
>>>          PyGILState_Release(__pyx_gilstate_save);
>>>          #endif
>>>          switch (__pyx_why) {
>>>            case 4: goto __pyx_L4;
>>>          }
>>>        }
>>>    }
>>>  }
>>>  /*finally:*/ {
>>>    int __pyx_why;
>>>    __pyx_why = 0; goto __pyx_L5;
>>>    __pyx_L4: __pyx_why = 4; goto __pyx_L5;
>>>    __pyx_L5:;
>>>    #ifdef WITH_THREAD
>>>    __pyx_gilstate_save = PyGILState_Ensure();
>>>    #endif
>>>    switch (__pyx_why) {
>>>      case 4: goto __pyx_L1_error;
>>>    }
>>>  }
>>>
>>>  goto __pyx_L0;
>>>  __pyx_L1_error:;
>>>  __Pyx_XDECREF(__pyx_t_1);
>>>  __Pyx_XDECREF(__pyx_t_2);
>>>  __Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>>>  __pyx_L0:;
>>>  #ifdef WITH_THREAD
>>>  PyGILState_Release(__pyx_gilstate_save);
>>>  #endif
>>> }
>>> """
>>>
>>> The first "finally" block is inside of the with-gil block, whereas the
>>> second is outside. In the second, the GIL is reacquired, and I guess that's
>>> for cleaning up temps in the error case, right? I see the problem that it
>>> can't be acquired after jumping to the exit labels (error/return) in the
>>> current code layout, but wouldn't it make sense to generate this code
>>> instead (starting at the second finally clause):
>>
>> This code could certainly be optimized, I just never got around to
>> implementing it.
>>
>>> """
>>>  /*finally:*/ {
>>>    int __pyx_why;
>>>    __pyx_why = 0; goto __pyx_L5;
>>>    __pyx_L4: __pyx_why = 4; goto __pyx_L5;
>>>    __pyx_L5:;
>>>    switch (__pyx_why) {
>>>      case 4: goto __pyx_L1_error;
>>>    }
>>>  }
>>>
>>>  goto __pyx_L0;
>>>  __pyx_L1_error:;
>>>  #ifdef WITH_THREAD
>>>  __pyx_gilstate_save = PyGILState_Ensure();
>>>  #endif
>>>  __Pyx_XDECREF(__pyx_t_1);
>>>  __Pyx_XDECREF(__pyx_t_2);
>>>  __Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>>>  #ifdef WITH_THREAD
>>>  PyGILState_Release(__pyx_gilstate_save);
>>>  #endif
>>>  return;   /* <- error return here! */
>>>  __pyx_L0:;
>>> }
>>> """
>>>
>>> Meaning: split the return code paths and let them independently acquire the
>>> GIL if needed? That would at least relieve the outer finally from having to
>>> care about the GIL and having to rely on "someone" to declare the GIL
>>> variable and eventually release it.
>>
>> In this case, that would be valid. Normally though,
>> the exception case should fall through to the normal case, so I think
>> what you'd want is
>>
>>    ...
>>    goto L0;
>> L1:
>>    PyGILState_Ensure();
>>    /* error code */
>>    __pyx_r = ...;
>>    goto L2;
>> L0:
>>    PyGILState_Ensure();
>> L2:
>>    /* normal code */
>>    return __pyx_r;
>> }
> 
> And there should be a release just before the return of course :)

Yes, I think that's much better than acquiring the GIL in some node in the
syntax tree and then releasing it in a completely different place at
function exit.

(IIRC, the C compiler won't allow the definition of the GIL state variable
to occur at the end in this case because there'd be code paths that use it
that start at a label after the declaration, so it would still have to be
declared at the beginning. Fine with me.)


>> If the entire body is a 'with gil' block, it'd probably be easiest to
>> transform the nogil function into a 'with gil' function.

Right.


>> If you want
>> to optimize for "paths going from with gil blocks directly into
>> function cleanup code" (i.e., no intermediate nogil try/finally or
>> nested with gil blocks), then you could use two more labels in the
>> code above that bypass the acquire.

In any case, jumping through labels is the normal way we do these things in
Cython, so this is worth cleaning up. And if we do all of this in the main
function code generation, it's much easier to figure out on what code paths
(i.e. at what labels) the GIL is really required for cleanup.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 22:31:30 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:31:30 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D458F.3000602@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de> <4F4D36BF.4040000@behnel.de>
	<CANg26EVbUbsoHWF+7Jcex1Ksc24O5AFvtAt5Q-f4D+gSjLh=4Q@mail.gmail.com>
	<CANg26EVRmBF-2=SYpF7_s_tkqV9_Km2tW-Woy_MVVmH+LPDVfQ@mail.gmail.com>
	<4F4D458F.3000602@behnel.de>
Message-ID: <CANg26EVbZmx5LdABc=e+BJuMSL+J3NrM8EcRsCBLbmJfpgAS8w@mail.gmail.com>

On 28 February 2012 21:22, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 22:09:
>> On 28 February 2012 21:08, mark florisson wrote:
>>> On 28 February 2012 20:19, Stefan Behnel wrote:
>>>> Stefan Behnel, 28.02.2012 20:58:
>>>>> mark florisson, 28.02.2012 16:35:
>>>>>> Basically, the cleanup code only needs a matching release because the
>>>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>>>> code will not avail anything, as the GIL is already ensured.
>>>>>
>>>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>>>> state into a block local variable and that broke the C code. Basically, the
>>>>> with-gil block had declared the variable in its own block, but was then
>>>>> trying to access that variable in a second finally clause, further down and
>>>>> outside of the with-gil block.
>>>>
>>>> Hmm, guess I got confused again...
>>>>
>>>> So, the code is this:
>>>>
>>>> """
>>>> ? ? ? /*finally:*/ {
>>>> ? ? ? ? ?int __pyx_why;
>>>> ? ? ? ? ?__pyx_why = 0; goto __pyx_L8;
>>>> ? ? ? ? ?__pyx_L7: __pyx_why = 4; goto __pyx_L8;
>>>> ? ? ? ? ?__pyx_L8:;
>>>> ? ? ? ? ?#ifdef WITH_THREAD
>>>> ? ? ? ? ?PyGILState_Release(__pyx_gilstate_save);
>>>> ? ? ? ? ?#endif
>>>> ? ? ? ? ?switch (__pyx_why) {
>>>> ? ? ? ? ? ?case 4: goto __pyx_L4;
>>>> ? ? ? ? ?}
>>>> ? ? ? ?}
>>>> ? ?}
>>>> ?}
>>>> ?/*finally:*/ {
>>>> ? ?int __pyx_why;
>>>> ? ?__pyx_why = 0; goto __pyx_L5;
>>>> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
>>>> ? ?__pyx_L5:;
>>>> ? ?#ifdef WITH_THREAD
>>>> ? ?__pyx_gilstate_save = PyGILState_Ensure();
>>>> ? ?#endif
>>>> ? ?switch (__pyx_why) {
>>>> ? ? ?case 4: goto __pyx_L1_error;
>>>> ? ?}
>>>> ?}
>>>>
>>>> ?goto __pyx_L0;
>>>> ?__pyx_L1_error:;
>>>> ?__Pyx_XDECREF(__pyx_t_1);
>>>> ?__Pyx_XDECREF(__pyx_t_2);
>>>> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>>>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>>>> ?__pyx_L0:;
>>>> ?#ifdef WITH_THREAD
>>>> ?PyGILState_Release(__pyx_gilstate_save);
>>>> ?#endif
>>>> }
>>>> """
>>>>
>>>> The first "finally" block is inside of the with-gil block, whereas the
>>>> second is outside. In the second, the GIL is reacquired, and I guess that's
>>>> for cleaning up temps in the error case, right? I see the problem that it
>>>> can't be acquired after jumping to the exit labels (error/return) in the
>>>> current code layout, but wouldn't it make sense to generate this code
>>>> instead (starting at the second finally clause):
>>>
>>> This code could certainly be optimized, I just never got around to
>>> implementing it.
>>>
>>>> """
>>>> ?/*finally:*/ {
>>>> ? ?int __pyx_why;
>>>> ? ?__pyx_why = 0; goto __pyx_L5;
>>>> ? ?__pyx_L4: __pyx_why = 4; goto __pyx_L5;
>>>> ? ?__pyx_L5:;
>>>> ? ?switch (__pyx_why) {
>>>> ? ? ?case 4: goto __pyx_L1_error;
>>>> ? ?}
>>>> ?}
>>>>
>>>> ?goto __pyx_L0;
>>>> ?__pyx_L1_error:;
>>>> ?#ifdef WITH_THREAD
>>>> ?__pyx_gilstate_save = PyGILState_Ensure();
>>>> ?#endif
>>>> ?__Pyx_XDECREF(__pyx_t_1);
>>>> ?__Pyx_XDECREF(__pyx_t_2);
>>>> ?__Pyx_WriteUnraisable("with_gil.void_nogil_ignore_exception",
>>>> __pyx_clineno, __pyx_lineno, __pyx_filename);
>>>> ?#ifdef WITH_THREAD
>>>> ?PyGILState_Release(__pyx_gilstate_save);
>>>> ?#endif
>>>> ?return; ? /* <- error return here! */
>>>> ?__pyx_L0:;
>>>> }
>>>> """
>>>>
>>>> Meaning: split the return code paths and let them independently acquire the
>>>> GIL if needed? That would at least relieve the outer finally from having to
>>>> care about the GIL and having to rely on "someone" to declare the GIL
>>>> variable and eventually release it.
>>>
>>> In this case, that would be valid. Normally though,
>>> the exception case should fall through to the normal case, so I think
>>> what you'd want is
>>>
>>> ? ?...
>>> ? ?goto L0;
>>> L1:
>>> ? ?PyGILState_Ensure();
>>> ? ?/* error code */
>>> ? ?__pyx_r = ...;
>>> ? ?goto L2;
>>> L0:
>>> ? ?PyGILState_Ensure();
>>> L2:
>>> ? ?/* normal code */
>>> ? ?return __pyx_r;
>>> }
>>
>> And there should be a release just before the return of course :)
>
> Yes, I think that's much better than acquiring the GIL in some node in the
> syntax tree and then releasing it in a completely different place at
> function exit.
>
> (IIRC, the C compiler won't allow the definition of the GIL state variable
> to occur at the end in this case because there'd be code paths that use it
> that start at a label after the declaration, so it would still have to be
> declared at the beginning. Fine with me.)
>
>
>>> If the entire body is a 'with gil' block, it'd probably be easiest to
>>> transform the nogil function into a 'with gil' function.
>
> Right.
>
>
>>> If you want
>>> to optimize for "paths going from with gil blocks directly into
>>> function cleanup code" (i.e., no intermediate nogil try/finally or
>>> nested with gil blocks), then you could use two more labels in the
>>> code above that bypass the acquire.
>
> In any case, jumping through labels is the normal way we do these things in
> Cython, so this is worth cleaning up. And if we do all of this in the main
> function code generation, it's much easier to figure out on what code paths
> (i.e. at what labels) the GIL is really required for cleanup.

A cleanup along with any optimizations would be great, +1 from me.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Tue Feb 28 22:38:31 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Feb 2012 22:38:31 +0100
Subject: [Cython] GIL handling C code
In-Reply-To: <CANg26EXG=EywYwA4uiqtFmgR8RFbOqZE19a_gXiLHV+0aaNxDw@mail.gmail.com>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
	<CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
	<4F4D4266.2070109@behnel.de>
	<CANg26EXG=EywYwA4uiqtFmgR8RFbOqZE19a_gXiLHV+0aaNxDw@mail.gmail.com>
Message-ID: <4F4D4957.4020800@behnel.de>

mark florisson, 28.02.2012 22:19:
> On 28 February 2012 21:08, Stefan Behnel wrote:
>> mark florisson, 28.02.2012 21:20:
>>> On 28 February 2012 19:58, Stefan Behnel wrote:
>>>> mark florisson, 28.02.2012 16:35:
>>>>> Basically, the cleanup code only needs a matching release because the
>>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>>> code will not avail anything, as the GIL is already ensured.
>>>>
>>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>>> state into a block local variable and that broke the C code. Basically, the
>>>> with-gil block had declared the variable in its own block, but was then
>>>> trying to access that variable in a second finally clause, further down and
>>>> outside of the with-gil block.
>>>>
>>>> Looks like a bug to me.
>>>
>>> That's not a bug, that's how it is implemented. At setup, a variable
>>> __pyx_gilstate_save is declared, which is also used for teardown. Any
>>> with GIL blocks use C block scoping to do the same thing. The with
>>> block is itself a try/finally, so you get a try/finally wrapped in a
>>> try/finally. The code uses try/finally for it's way of trapping
>>> control flow, allowing some code to execute and resuming control flow
>>> afterwards.
>>
>> Ok, so what really got me confused is that the code used the variable
>> "acquire_gil_for_refnanny_only" in places that didn't have anything to do
>> with the refnanny. Similarly, "acquire_gil_for_var_decls_only" was used for
>> cleanup even though the GIL had already been released if this flag was set,
>> way before the end of the function.
>>
>> I think I fixed both issues in the patch I attached. At least, it still
>> passes the gil related tests and doesn't raise any C compiler warnings
>> about the GIL state variable being unused.
>>
>> Does this look about right?
> 
> It looks right to me, yeah.

Ok. I think this looks simple enough to go into the release, whereas any
more advanced cleanup and optimisations should have their time to mature.
Would you object?


> (<pedantic>I prefer to format bools
> directly with %d, as they are a subclass of int anyway</pedantic>).

I don't. It works when they are really booleans, but in Python, many things
that are "true" or "false" aren't actually of type bool. When it comes to
writing out (user visible) data, it's always best to make it clear what the
intended output is, instead of relying on 'implementation details' and data
of unknown sources (or different purposes, as in this case).


> I think in general the EnsureGILNode should have been mentioned in the
> code generation function of FuncDefNode, which makes it easier to
> figure out what is going on. The documentation is currently at the
> wrapping site in AnalyseDeclarationsTransform.

I'll add a comment.

Stefan

From markflorisson88 at gmail.com  Tue Feb 28 22:44:13 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 28 Feb 2012 21:44:13 +0000
Subject: [Cython] GIL handling C code
In-Reply-To: <4F4D4957.4020800@behnel.de>
References: <4F4CA003.7080407@behnel.de> <4F4CA449.4000108@behnel.de>
	<CANg26EVCjVpz4-Ma8r=Hf4M11cNkQkiFAZ3ZPdMGsmFSd8rrbg@mail.gmail.com>
	<4F4CABB2.5000603@behnel.de>
	<CANg26EUPcNcisDJ8_-ZEPOUb+0d52c7et3GJx5iuyFm861L9JA@mail.gmail.com>
	<4F4CB21F.7060300@behnel.de>
	<CANg26EWktdToekPMx9Q=hpGK3ZgywSwtq35=9MnvXZxdF16EEw@mail.gmail.com>
	<4F4CDBB5.9040203@behnel.de>
	<CANg26EU70D4TZrJQORkNc+LB8m14jkJ9dHMEGY=PRq8P3KcfZw@mail.gmail.com>
	<4F4D31DF.4020306@behnel.de>
	<CANg26EUfvBdzzLrdacuAW=2UiWe+d6Q-B8-X_o3dA74Vo6BwHw@mail.gmail.com>
	<4F4D4266.2070109@behnel.de>
	<CANg26EXG=EywYwA4uiqtFmgR8RFbOqZE19a_gXiLHV+0aaNxDw@mail.gmail.com>
	<4F4D4957.4020800@behnel.de>
Message-ID: <CANg26EXZKiLr2tWMZK4DWP1o2=2uyi+Yma+f5Q85tioLc3Qu1A@mail.gmail.com>

On 28 February 2012 21:38, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.02.2012 22:19:
>> On 28 February 2012 21:08, Stefan Behnel wrote:
>>> mark florisson, 28.02.2012 21:20:
>>>> On 28 February 2012 19:58, Stefan Behnel wrote:
>>>>> mark florisson, 28.02.2012 16:35:
>>>>>> Basically, the cleanup code only needs a matching release because the
>>>>>> corresponding acquire is in EnsureGILNode, which wraps the function
>>>>>> body in case of a nogil function with a 'with gil' block. Any changes
>>>>>> to the conditions in FuncDefNode will have to be reflected by the code
>>>>>> that does that wrapping. Changing the refnanny macro for the cleanup
>>>>>> code will not avail anything, as the GIL is already ensured.
>>>>>
>>>>> Regarding the "with gil" code, ISTM that the "finally" code in the with_gil
>>>>> test is being duplicated. I noticed this when I moved the refnanny's GIL
>>>>> state into a block local variable and that broke the C code. Basically, the
>>>>> with-gil block had declared the variable in its own block, but was then
>>>>> trying to access that variable in a second finally clause, further down and
>>>>> outside of the with-gil block.
>>>>>
>>>>> Looks like a bug to me.
>>>>
>>>> That's not a bug, that's how it is implemented. At setup, a variable
>>>> __pyx_gilstate_save is declared, which is also used for teardown. Any
>>>> with GIL blocks use C block scoping to do the same thing. The with
>>>> block is itself a try/finally, so you get a try/finally wrapped in a
>>>> try/finally. The code uses try/finally for it's way of trapping
>>>> control flow, allowing some code to execute and resuming control flow
>>>> afterwards.
>>>
>>> Ok, so what really got me confused is that the code used the variable
>>> "acquire_gil_for_refnanny_only" in places that didn't have anything to do
>>> with the refnanny. Similarly, "acquire_gil_for_var_decls_only" was used for
>>> cleanup even though the GIL had already been released if this flag was set,
>>> way before the end of the function.
>>>
>>> I think I fixed both issues in the patch I attached. At least, it still
>>> passes the gil related tests and doesn't raise any C compiler warnings
>>> about the GIL state variable being unused.
>>>
>>> Does this look about right?
>>
>> It looks right to me, yeah.
>
> Ok. I think this looks simple enough to go into the release, whereas any
> more advanced cleanup and optimisations should have their time to mature.
> Would you object?
>

Sure, that's fine with me.

>> (<pedantic>I prefer to format bools
>> directly with %d, as they are a subclass of int anyway</pedantic>).
>
> I don't. It works when they are really booleans, but in Python, many things
> that are "true" or "false" aren't actually of type bool. When it comes to
> writing out (user visible) data, it's always best to make it clear what the
> intended output is, instead of relying on 'implementation details' and data
> of unknown sources (or different purposes, as in this case).
>
>
>> I think in general the EnsureGILNode should have been mentioned in the
>> code generation function of FuncDefNode, which makes it easier to
>> figure out what is going on. The documentation is currently at the
>> wrapping site in AnalyseDeclarationsTransform.
>
> I'll add a comment.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Wed Feb 29 15:41:29 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Feb 2012 15:41:29 +0100
Subject: [Cython] [cython-users] cleaning up in a module
In-Reply-To: <4F4E2B4E.4060808@behnel.de>
References: <1330522212.3713.21.camel@farnsworth> <4F4E296F.3070603@behnel.de>
	<4F4E2B4E.4060808@behnel.de>
Message-ID: <4F4E3919.4040204@behnel.de>

Stefan Behnel, 29.02.2012 14:42:
> Stefan Behnel, 29.02.2012 14:34:
>> Henry Gomersall, 29.02.2012 14:30:
>>> What's the preferred way to clean up a C library when a module is
>>> deleted?
>>
>> PEP 3121:
>>
>> http://www.python.org/dev/peps/pep-3121/
>>
>> However, given that CPython doesn't currently support unloading extension
>> modules, your question is rather hypothetical.
> 
> Oh, what you *can* do, however, is register an "atexit" function that does
> the cleanup when the interpreter terminates.

Speaking of which, what about allowing users to implement a function

cdef void __dealloc__()

at the module level, which would then be called by the module cleanup
function (if generated) before running the rest of the cleanup code?

That would allow for an easier transition of user code towards a better
support of PEP 3121, specifically m_clear().

http://trac.cython.org/cython_trac/ticket/218

Stefan

From d.s.seljebotn at astro.uio.no  Wed Feb 29 18:06:42 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Feb 2012 09:06:42 -0800
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
Message-ID: <4F4E5B22.8010701@astro.uio.no>

I'm wondering what the best course of action for deprecating the shape 
field in numpy.pxd is.

The thing is, currently "shape" really gets in the way. In most 
situations it is OK with slow access to shape through the Python layer, 
and "arr.shape[0]" is often just fine, but currently one is in a 
situation where one must either write "(<object>arr).shape[0])" or 
"np.PyArray_DIMS(arr)[0]", or be faced with code that isn't 
forward-compatible with NumPy.

It would really be good to do the transition as fast as possible, so 
that all Cython code eventually becomes ready for upcoming NumPy releases.

The simplest change to make would be to simply remove all the ndarray 
fields from numpy.pxd, and inform about the alternatives in the release 
notes. That could be done in time for 0.16.

The alternatives of sounding sane deprecation warnings in just the right 
places takes more work. I can't work on that myself until PyCon 
sprints... perhaps put out 0.16.1 with just this change after PyCon 
sprints though...

Dag

From stefan_ml at behnel.de  Wed Feb 29 18:42:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Feb 2012 18:42:27 +0100
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4E5B22.8010701@astro.uio.no>
References: <4F4E5B22.8010701@astro.uio.no>
Message-ID: <4F4E6383.8030908@behnel.de>

Dag Sverre Seljebotn, 29.02.2012 18:06:
> I'm wondering what the best course of action for deprecating the shape
> field in numpy.pxd is.
> 
> The thing is, currently "shape" really gets in the way. In most situations
> it is OK with slow access to shape through the Python layer, and
> "arr.shape[0]" is often just fine, but currently one is in a situation
> where one must either write "(<object>arr).shape[0])" or
> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
> forward-compatible with NumPy.

Can Cython emulate this at the C layer? And even your work-around for the
Python object access looks more like a Cython bug to me. I wouldn't know
why that can't "just work". It usually works for other undeclared Python
attributes of "anything", so it might just as well be made to work here.


> It would really be good to do the transition as fast as possible, so that
> all Cython code eventually becomes ready for upcoming NumPy releases.

But it previously worked, right? It's just no longer supported in newer
NumPy versions IIUC? If that's the case, deleting it would break otherwise
working code. No-one forces you to switch to the latest NumPy version,
after all, and certainly not right now. A warning is much better.


> The simplest change to make would be to simply remove all the ndarray
> fields from numpy.pxd, and inform about the alternatives in the release
> notes. That could be done in time for 0.16.
> 
> The alternatives of sounding sane deprecation warnings in just the right
> places takes more work. I can't work on that myself until PyCon sprints...
> perhaps put out 0.16.1 with just this change after PyCon sprints though...

Personally, I don't think this is time critical.

Stefan

From d.s.seljebotn at astro.uio.no  Wed Feb 29 18:57:13 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Feb 2012 09:57:13 -0800
Subject: [Cython] Cython 0.16 and ndarray fields deprecation
In-Reply-To: <4F4E6383.8030908@behnel.de>
References: <4F4E5B22.8010701@astro.uio.no> <4F4E6383.8030908@behnel.de>
Message-ID: <4F4E66F9.9000303@astro.uio.no>

On 02/29/2012 09:42 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 29.02.2012 18:06:
>> I'm wondering what the best course of action for deprecating the shape
>> field in numpy.pxd is.
>>
>> The thing is, currently "shape" really gets in the way. In most situations
>> it is OK with slow access to shape through the Python layer, and
>> "arr.shape[0]" is often just fine, but currently one is in a situation
>> where one must either write "(<object>arr).shape[0])" or
>> "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't
>> forward-compatible with NumPy.
>
> Can Cython emulate this at the C layer? And even your work-around for the
> Python object access looks more like a Cython bug to me. I wouldn't know
> why that can't "just work". It usually works for other undeclared Python
> attributes of "anything", so it might just as well be made to work here.

Well, the problem is that shape is currently declared as a C field. It 
is also available as a Python attribute. Usually the user doesn't care 
which one is used, but the C field is declared for the few cases where 
access is speed-critical.

Though even with current NumPy, I find myself doing "print 
(<object>arr).shape" in order to get a tuple rather than a Py_ssize_t*...

>> It would really be good to do the transition as fast as possible, so that
>> all Cython code eventually becomes ready for upcoming NumPy releases.
>
> But it previously worked, right? It's just no longer supported in newer
> NumPy versions IIUC? If that's the case, deleting it would break otherwise
> working code. No-one forces you to switch to the latest NumPy version,
> after all, and certainly not right now. A warning is much better.

It previously worked, but it turns out that it was always frowned-upon. 
I didn't know that when I added the fields, and it was a convenient way 
of speeding things up...

Dag