From dg.gmane at thesamovar.net  Sun Dec  1 15:53:14 2013
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Sun, 1 Dec 2013 20:53:14 +0000 (UTC)
Subject: [Numpy-discussion] -ffast-math
References: <loom.20131129T211145-800@post.gmane.org>
	<5299CCE9.90602@googlemail.com>
Message-ID: <loom.20131201T214042-47@post.gmane.org>

Julian Taylor <jtaylor.debian <at> googlemail.com> writes:
> can you show the code that is slow in numpy?
> which version of gcc and libc are you using?
> with gcc 4.8 it uses the glibc 2.17 sin/cos with fast-math, so there
> should be no difference.

In trying to write some simple code to demonstrate it, I realised it was
weirdly more complicated than I thought. Previously I had been comparing
numpy against weave on a complicated expression, namely a*sin(2.0*freq*pi*t)
+ b + v*exp(-dt/tau) + (-a*sin(2.0*freq*pi*t) - b)*exp(-dt/tau). Doing that
with weave and no -ffast-math took the same time as numpy approximately, but
with weave and -ffast-math it was about 30x faster. Here only a and v are
arrays. Since numpy and weave with no -ffast-math took about the same time I
assumed it wasn't memory bound but to do with the -ffast-math.

Here's the demo code (you might need to comment a couple of lines out if you
want to actually run it, since it also tests a couple of things that depend
on a library):

http://bit.ly/IziH8H

However, when I did a simple example that just computed y=sin(x) for arrays
x and y, I found that numpy and weave without -ffast-math took about the
same time, but weave with -ffast-math was significantly slower than numpy!
My take home message from this: optimisation is weird. Could it be that
-ffast-math and -O3 allow SSE instructions and that there is some overhead
to this that makes it worth it for a complex expression but not for a simple
expression?

Here's the code for the simple example (doesn't have any dependencies):

http://bit.ly/18wdCKY

For reference, I'm on a newish 64 bit windows machine running 32 bit Python
2.7.3, gcc version 4.5.2, numpy 1.8.0 installed from binaries.

Dan



From jtaylor.debian at googlemail.com  Sun Dec  1 16:30:35 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 01 Dec 2013 22:30:35 +0100
Subject: [Numpy-discussion] -ffast-math
In-Reply-To: <loom.20131201T214042-47@post.gmane.org>
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>
	<loom.20131201T214042-47@post.gmane.org>
Message-ID: <529BAA7B.3070206@googlemail.com>

On 01.12.2013 21:53, Dan Goodman wrote:
> Julian Taylor <jtaylor.debian <at> googlemail.com> writes:
>> can you show the code that is slow in numpy?
>> which version of gcc and libc are you using?
>> with gcc 4.8 it uses the glibc 2.17 sin/cos with fast-math, so there
>> should be no difference.
> 
> In trying to write some simple code to demonstrate it, I realised it was
> weirdly more complicated than I thought. Previously I had been comparing
> numpy against weave on a complicated expression, namely a*sin(2.0*freq*pi*t)
> + b + v*exp(-dt/tau) + (-a*sin(2.0*freq*pi*t) - b)*exp(-dt/tau). Doing that
> with weave and no -ffast-math took the same time as numpy approximately, but
> with weave and -ffast-math it was about 30x faster. Here only a and v are
> arrays. Since numpy and weave with no -ffast-math took about the same time I
> assumed it wasn't memory bound but to do with the -ffast-math.
> 

this should be the code:

int N = _N;
for(int _idx=0; _idx<N; _idx++)
{
    double a = _array_neurongroup_a[_idx];
    double v = _array_neurongroup_v[_idx];
    double _v = a*sin(2.0*freq*pi*t) + b + v*exp(-dt/tau) +
(-a*sin(2.0*freq*pi*t) - b)*exp(-dt/tau);
    v = _v;
    _array_neurongroup_v[_idx] = v;
}


your sin and exp calls are loop invariants, they do not depend on the
loop iterable.
This allows to move the expensive functions out of the loop and only
leave some simple arithmetic in the body.

Unfortunately ieee754 floating point (gcc's default mode) does not allow
this type of transformation, they are not associative, you have special
values to propagate and sticky exceptions to preserve, set errno, etc.
All this prevents gcc from doing this in its default mode.
-ffast-math tells it to ignore all these things and just make it fast,
so it will do the loop invariant transformation in this case.

In this case setting -fno-math-errno, which disables taking care of
setting errno as the C standard requires, seems to be enough.

In pure numpy you have to do these types of transformations yourself as
cpython has no optimizer which does this type of loop invariant
optimizations.


From dg.gmane at thesamovar.net  Sun Dec  1 16:59:26 2013
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Sun, 1 Dec 2013 21:59:26 +0000 (UTC)
Subject: [Numpy-discussion] -ffast-math
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>
	<loom.20131201T214042-47@post.gmane.org>
	<529BAA7B.3070206@googlemail.com>
Message-ID: <loom.20131201T225602-140@post.gmane.org>

Julian Taylor <jtaylor.debian <at> googlemail.com> writes:
> your sin and exp calls are loop invariants, they do not depend on the
> loop iterable.
> This allows to move the expensive functions out of the loop and only
> leave some simple arithmetic in the body.

Ahhhh! I feel extremely stupid for not realising this! Thanks Julian.

Any thoughts on why using -ffast-math it actually goes slower for just doing
sin(x)?



From jtaylor.debian at googlemail.com  Sun Dec  1 17:16:40 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 01 Dec 2013 23:16:40 +0100
Subject: [Numpy-discussion] -ffast-math
In-Reply-To: <loom.20131201T225602-140@post.gmane.org>
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>	<loom.20131201T214042-47@post.gmane.org>	<529BAA7B.3070206@googlemail.com>
	<loom.20131201T225602-140@post.gmane.org>
Message-ID: <529BB548.6040806@googlemail.com>

On 01.12.2013 22:59, Dan Goodman wrote:
> Julian Taylor <jtaylor.debian <at> googlemail.com> writes:
>> your sin and exp calls are loop invariants, they do not depend on the
>> loop iterable.
>> This allows to move the expensive functions out of the loop and only
>> leave some simple arithmetic in the body.
> 
> Ahhhh! I feel extremely stupid for not realising this! Thanks Julian.
> 
> Any thoughts on why using -ffast-math it actually goes slower for just doing
> sin(x)?
> 

no on my linux machine ffast-math is a little faster:
numpy: 311 ms
weave_slow: 291 ms
weave_fast: 262 ms


here is a pure numpy version of your calculation which only performs 3
times worse than weave:

def timefunc_numpy2(a, v):
    ext = exp(-dt/tau)
    sit = sin(2.0*freq*pi*t)
    bs = 20000
    for i in range(0, N, bs):
        ab = a[i:i+bs]
        vb = v[i:i+bs]
        absit = ab*sit + b
        vb *= ext
        vb += absit
        vb -= absit*ext

it works by replacing temporaries with inplace operations and blocks the
operations to be more memory cache friendlier.
using numexpr should give you similar results.


From dg.gmane at thesamovar.net  Sun Dec  1 18:01:31 2013
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Sun, 1 Dec 2013 23:01:31 +0000 (UTC)
Subject: [Numpy-discussion] -ffast-math
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>	<loom.20131201T214042-47@post.gmane.org>	<529BAA7B.3070206@googlemail.com>
	<loom.20131201T225602-140@post.gmane.org>
	<529BB548.6040806@googlemail.com>
Message-ID: <loom.20131202T000131-727@post.gmane.org>

Julian Taylor <jtaylor.debian <at> googlemail.com> writes:

> 
> On 01.12.2013 22:59, Dan Goodman wrote:
> > Julian Taylor <jtaylor.debian <at> googlemail.com> writes:
> >> your sin and exp calls are loop invariants, they do not depend on the
> >> loop iterable.
> >> This allows to move the expensive functions out of the loop and only
> >> leave some simple arithmetic in the body.
> > 
> > Ahhhh! I feel extremely stupid for not realising this! Thanks Julian.
> > 
> > Any thoughts on why using -ffast-math it actually goes slower for just doing
> > sin(x)?
> > 
> 
> no on my linux machine ffast-math is a little faster:
> numpy: 311 ms
> weave_slow: 291 ms
> weave_fast: 262 ms

Maybe something to do with my older version of gcc (4.5)?

> here is a pure numpy version of your calculation which only performs 3
> times worse than weave:
> 
> def timefunc_numpy2(a, v):
>     ext = exp(-dt/tau)
>     sit = sin(2.0*freq*pi*t)
>     bs = 20000
>     for i in range(0, N, bs):
>         ab = a[i:i+bs]
>         vb = v[i:i+bs]
>         absit = ab*sit + b
>         vb *= ext
>         vb += absit
>         vb -= absit*ext
> 
> it works by replacing temporaries with inplace operations and blocks the
> operations to be more memory cache friendlier.
> using numexpr should give you similar results.

I was working on something similar without the blocking and also got good
results. Actually, your version with blocking doesn't give me as good
performance on my machine, it's around 6x slower than weave. I tried
different sizes for the block size but couldn't improve much on that. Using
this unblocked code:

def timefunc_numpy_smart():
    _sin_term = sin(2.0*freq*pi*t)
    _exp_term = exp(-dt/tau)
    _a_term = (_sin_term-_sin_term*_exp_term)
    _v = v
    _v *= _exp_term
    _v += a*_a_term
    _v += -b*_exp_term + b

I got around 5x slower. Using numexpr 'dumbly' (i.e. just putting the
expression in directly) was slower than the function above, but doing a
hybrid between the two approaches worked well:

def timefunc_numexpr_smart():
    _sin_term = sin(2.0*freq*pi*t)
    _exp_term = exp(-dt/tau)
    _a_term = (_sin_term-_sin_term*_exp_term)
    _const_term = -b*_exp_term + b
    v[:] = numexpr.evaluate('a*_a_term+v*_exp_term+_const_term')
    #numexpr.evaluate('a*_a_term+v*_exp_term+_const_term', out=v)

This was about 3.5x slower than weave. If I used the commented out final
line then it was only 1.5x slower than weave, but it also gives wrong
results. I reported this as a bug in numexpr a long time ago but I guess it
hasn't been fixed yet (or maybe I didn't upgrade my version recently).

Dan



From dg.gmane at thesamovar.net  Sun Dec  1 18:14:57 2013
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Sun, 1 Dec 2013 23:14:57 +0000 (UTC)
Subject: [Numpy-discussion] -ffast-math
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>	<loom.20131201T214042-47@post.gmane.org>	<529BAA7B.3070206@googlemail.com>
	<loom.20131201T225602-140@post.gmane.org>
	<529BB548.6040806@googlemail.com>
	<loom.20131202T000131-727@post.gmane.org>
Message-ID: <loom.20131202T001342-893@post.gmane.org>

Dan Goodman <dg.gmane <at> thesamovar.net> writes:
...
> I got around 5x slower. Using numexpr 'dumbly' (i.e. just putting the
> expression in directly) was slower than the function above, but doing a
> hybrid between the two approaches worked well:
> 
> def timefunc_numexpr_smart():
>     _sin_term = sin(2.0*freq*pi*t)
>     _exp_term = exp(-dt/tau)
>     _a_term = (_sin_term-_sin_term*_exp_term)
>     _const_term = -b*_exp_term + b
>     v[:] = numexpr.evaluate('a*_a_term+v*_exp_term+_const_term')
>     #numexpr.evaluate('a*_a_term+v*_exp_term+_const_term', out=v)
> 
> This was about 3.5x slower than weave. If I used the commented out final
> line then it was only 1.5x slower than weave, but it also gives wrong
> results. I reported this as a bug in numexpr a long time ago but I guess it
> hasn't been fixed yet (or maybe I didn't upgrade my version recently).

I just upgraded numexpr to 2.2 where they did fix this bug, and now the
'smart' numexpr version runs exactly as fast as weave (so I guess there were
some performance enhancements in numexpr as well).

Fantastic!

Dan



From mailinglists at xgm.de  Mon Dec  2 11:52:07 2013
From: mailinglists at xgm.de (Florian Lindner)
Date: Mon, 02 Dec 2013 17:52:07 +0100
Subject: [Numpy-discussion] Joining lists to an array
Message-ID: <6972186.imR98MDMjN@horus>

Hello,

I have this piece of example code

import random, numpy as np

y = []
doc_all = []
# da = np.zeros(2)
for i in range(4): 
    docs = range(random.randint(1, 10))
    y += [i]*len(docs)
    doc_all += docs
    # np.append(da, np.column_stack((docs, y)), axis=0)
    
data = np.array([doc_all, y]).transpose()

y and docs are lists that are created in the loop body and all joined 
together. From these two long lists an array is created at the end.

At the end data has a shape like (28, 2).

Is there a way I can do this more elegantly using numpy or scipy tricks? I was 
working on something like the two lines I commented out, but ...

Thanks,
Florian


From daniele at grinta.net  Mon Dec  2 12:44:25 2013
From: daniele at grinta.net (Daniele Nicolodi)
Date: Mon, 02 Dec 2013 18:44:25 +0100
Subject: [Numpy-discussion] math.fsum() like ufunc for numpy
Message-ID: <529CC6F9.2060106@grinta.net>

Hello,

there would be interest in adding a floating point accurate summation
function like Python's math.fsum() in the form of an ufunc to NumPy?

I had a look at the algorithm
(http://code.activestate.com/recipes/393090-binary-floating-point-summation-accurate-to-full-p/)
and it looks quite straightforward to implement. I can try to submit a
patch for it.

Cheers,
Daniele


From njs at pobox.com  Mon Dec  2 13:07:02 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Dec 2013 10:07:02 -0800
Subject: [Numpy-discussion] math.fsum() like ufunc for numpy
In-Reply-To: <529CC6F9.2060106@grinta.net>
References: <529CC6F9.2060106@grinta.net>
Message-ID: <CAPJVwBkmGn6z082gM=v_JUej2D_5C5F-3wMUvKdxjdangaKP6Q@mail.gmail.com>

I think that would be great. Technically what you'd want is a "gufunc".

-n

On Mon, Dec 2, 2013 at 9:44 AM, Daniele Nicolodi <daniele at grinta.net> wrote:
> Hello,
>
> there would be interest in adding a floating point accurate summation
> function like Python's math.fsum() in the form of an ufunc to NumPy?
>
> I had a look at the algorithm
> (http://code.activestate.com/recipes/393090-binary-floating-point-summation-accurate-to-full-p/)
> and it looks quite straightforward to implement. I can try to submit a
> patch for it.
>
> Cheers,
> Daniele
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


From jtaylor.debian at googlemail.com  Mon Dec  2 13:11:25 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 02 Dec 2013 19:11:25 +0100
Subject: [Numpy-discussion] math.fsum() like ufunc for numpy
In-Reply-To: <CAPJVwBkmGn6z082gM=v_JUej2D_5C5F-3wMUvKdxjdangaKP6Q@mail.gmail.com>
References: <529CC6F9.2060106@grinta.net>
	<CAPJVwBkmGn6z082gM=v_JUej2D_5C5F-3wMUvKdxjdangaKP6Q@mail.gmail.com>
Message-ID: <529CCD4D.70506@googlemail.com>

related this PR attempts to improve the accuracy of summation:
https://github.com/numpy/numpy/pull/3685
but math.fsum gives the exact result so it would a valuable ufunc even
when that PR is merged.

python3.4 will have yet another accurate summation in the statistics module:
http://www.python.org/dev/peps/pep-0450/

On 02.12.2013 19:07, Nathaniel Smith wrote:
> I think that would be great. Technically what you'd want is a "gufunc".
> 
> -n
> 
> On Mon, Dec 2, 2013 at 9:44 AM, Daniele Nicolodi <daniele at grinta.net> wrote:
>> Hello,
>>
>> there would be interest in adding a floating point accurate summation
>> function like Python's math.fsum() in the form of an ufunc to NumPy?
>>
>> I had a look at the algorithm
>> (http://code.activestate.com/recipes/393090-binary-floating-point-summation-accurate-to-full-p/)
>> and it looks quite straightforward to implement. I can try to submit a
>> patch for it.
>>
>> Cheers,
>> Daniele
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 



From ndbecker2 at gmail.com  Mon Dec  2 14:08:52 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Dec 2013 14:08:52 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
Message-ID: <l7ilrp$jie$1@ger.gmane.org>

This is np 1.8.0 on fedora x86_64:

In [5]: x =np.array ((1,))

In [6]: x.shape
Out[6]: (1,)

In [7]: x.strides
Out[7]: (9223372036854775807,)



From davidmenhur at gmail.com  Mon Dec  2 14:14:13 2013
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Mon, 2 Dec 2013 20:14:13 +0100
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7ilrp$jie$1@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>
Message-ID: <CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>

I get:

In [4]: x.strides
Out[4]: (8,)

Same architecture and OS, Numpy installed via Pip on Python 2.7.5.


On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:

> This is np 1.8.0 on fedora x86_64:
>
> In [5]: x =np.array ((1,))
>
> In [6]: x.shape
> Out[6]: (1,)
>
> In [7]: x.strides
> Out[7]: (9223372036854775807,)
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131202/9c0297f1/attachment.html>

From ndbecker2 at gmail.com  Mon Dec  2 14:18:02 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Dec 2013 14:18:02 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
Message-ID: <l7imd0$jie$2@ger.gmane.org>

I built using:

CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py 
install --user


a?id wrote:

> I get:
> 
> In [4]: x.strides
> Out[4]: (8,)
> 
> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
> 
> 
> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
> 
>> This is np 1.8.0 on fedora x86_64:
>>
>> In [5]: x =np.array ((1,))
>>
>> In [6]: x.shape
>> Out[6]: (1,)
>>
>> In [7]: x.strides
>> Out[7]: (9223372036854775807,)
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>




From nouiz at nouiz.org  Mon Dec  2 14:19:07 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 2 Dec 2013 14:19:07 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
Message-ID: <CADKKbtjeGHEFiLe=-xiL-XHFeL9o=VUCcNoCM7SLfWq7CMMAWA@mail.gmail.com>

There is a way to compile NumPy to use strange strides for dimension
with shape of 1.

This is done to help developer test their code to don't rely on this.
There was never a warranty to the value of strides in that cases. Most
of the time, it was the same, but in some cases, it was different.

Using such strange strides will cause segfault if you use them, so it
allow to see if you rely on them.

In Theano, we did some assertion on strides and checked them for
optimized call to blas. So we will need to change some code to support
this.

But I don't those strange strides should happen in the wild. Did you
installed NumPy manually?

Fred


On Mon, Dec 2, 2013 at 2:14 PM, Da?id <davidmenhur at gmail.com> wrote:
> I get:
>
> In [4]: x.strides
> Out[4]: (8,)
>
> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>
>
> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>
>> This is np 1.8.0 on fedora x86_64:
>>
>> In [5]: x =np.array ((1,))
>>
>> In [6]: x.shape
>> Out[6]: (1,)
>>
>> In [7]: x.strides
>> Out[7]: (9223372036854775807,)
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From nouiz at nouiz.org  Mon Dec  2 14:19:22 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 2 Dec 2013 14:19:22 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7imd0$jie$2@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
Message-ID: <CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>

It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.

Fred

On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I built using:
>
> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
> install --user
>
>
> a?id wrote:
>
>> I get:
>>
>> In [4]: x.strides
>> Out[4]: (8,)
>>
>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>>
>>
>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>
>>> This is np 1.8.0 on fedora x86_64:
>>>
>>> In [5]: x =np.array ((1,))
>>>
>>> In [6]: x.shape
>>> Out[6]: (1,)
>>>
>>> In [7]: x.strides
>>> Out[7]: (9223372036854775807,)
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From ndbecker2 at gmail.com  Mon Dec  2 14:35:36 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Dec 2013 14:35:36 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
Message-ID: <l7indt$8qh$1@ger.gmane.org>

I don't think that behavior is acceptable.

Fr?d?ric Bastien wrote:

> It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.
> 
> Fred
> 
> On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I built using:
>>
>> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
>> install --user
>>
>>
>> a?id wrote:
>>
>>> I get:
>>>
>>> In [4]: x.strides
>>> Out[4]: (8,)
>>>
>>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>>>
>>>
>>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>
>>>> This is np 1.8.0 on fedora x86_64:
>>>>
>>>> In [5]: x =np.array ((1,))
>>>>
>>>> In [6]: x.shape
>>>> Out[6]: (1,)
>>>>
>>>> In [7]: x.strides
>>>> Out[7]: (9223372036854775807,)
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From nouiz at nouiz.org  Mon Dec  2 14:44:55 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 2 Dec 2013 14:44:55 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7indt$8qh$1@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
Message-ID: <CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>

Just don't compile with NPY_RELAXED_STRIDES_CHECKING to have the old
behavior I think (which is an not always the same strides depending of
how it was created, I don't know if they changed that or not).

Do someone else recall the detail of this?

Fred

p.s. I didn't do this or asked for it. But this help test your
software to don't depend of the strides when shapes is 1.

On Mon, Dec 2, 2013 at 2:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I don't think that behavior is acceptable.
>
> Fr?d?ric Bastien wrote:
>
>> It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.
>>
>> Fred
>>
>> On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> I built using:
>>>
>>> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
>>> install --user
>>>
>>>
>>> a?id wrote:
>>>
>>>> I get:
>>>>
>>>> In [4]: x.strides
>>>> Out[4]: (8,)
>>>>
>>>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>>>>
>>>>
>>>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>
>>>>> This is np 1.8.0 on fedora x86_64:
>>>>>
>>>>> In [5]: x =np.array ((1,))
>>>>>
>>>>> In [6]: x.shape
>>>>> Out[6]: (1,)
>>>>>
>>>>> In [7]: x.strides
>>>>> Out[7]: (9223372036854775807,)
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From jtaylor.debian at googlemail.com  Mon Dec  2 14:48:42 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 02 Dec 2013 20:48:42 +0100
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7indt$8qh$1@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>	<l7imd0$jie$2@ger.gmane.org>	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
Message-ID: <529CE41A.7040602@googlemail.com>

I opened a ticket for it, though thinking about it, its probably
intentional be intentional to find code that assumes it can use the
strides to get the itemsize.
https://github.com/numpy/numpy/issues/4091


On 02.12.2013 20:35, Neal Becker wrote:
> I don't think that behavior is acceptable.
> 
> Fr?d?ric Bastien wrote:
> 
>> It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.
>>
>> Fred
>>
>> On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> I built using:
>>>
>>> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
>>> install --user
>>>
>>>
>>> a?id wrote:
>>>
>>>> I get:
>>>>
>>>> In [4]: x.strides
>>>> Out[4]: (8,)
>>>>
>>>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>>>>
>>>>
>>>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>
>>>>> This is np 1.8.0 on fedora x86_64:
>>>>>
>>>>> In [5]: x =np.array ((1,))
>>>>>
>>>>> In [6]: x.shape
>>>>> Out[6]: (1,)
>>>>>
>>>>> In [7]: x.strides
>>>>> Out[7]: (9223372036854775807,)
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 



From ndbecker2 at gmail.com  Mon Dec  2 14:51:25 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Dec 2013 14:51:25 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
Message-ID: <l7iobi$jfp$1@ger.gmane.org>

The software I'm using, which is 

https://github.com/ndarray/ndarray

does depend on this.  Am I the only one who thinks that this
behavior is not desirable?

Fr?d?ric Bastien wrote:

> Just don't compile with NPY_RELAXED_STRIDES_CHECKING to have the old
> behavior I think (which is an not always the same strides depending of
> how it was created, I don't know if they changed that or not).
> 
> Do someone else recall the detail of this?
> 
> Fred
> 
> p.s. I didn't do this or asked for it. But this help test your
> software to don't depend of the strides when shapes is 1.
> 
> On Mon, Dec 2, 2013 at 2:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I don't think that behavior is acceptable.
>>
>> Fr?d?ric Bastien wrote:
>>
>>> It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.
>>>
>>> Fred
>>>
>>> On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> I built using:
>>>>
>>>> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
>>>> install --user
>>>>
>>>>
>>>> a?id wrote:
>>>>
>>>>> I get:
>>>>>
>>>>> In [4]: x.strides
>>>>> Out[4]: (8,)
>>>>>
>>>>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
>>>>>
>>>>>
>>>>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>>
>>>>>> This is np 1.8.0 on fedora x86_64:
>>>>>>
>>>>>> In [5]: x =np.array ((1,))
>>>>>>
>>>>>> In [6]: x.shape
>>>>>> Out[6]: (1,)
>>>>>>
>>>>>> In [7]: x.strides
>>>>>> Out[7]: (9223372036854775807,)
>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From njs at pobox.com  Mon Dec  2 14:56:33 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Dec 2013 11:56:33 -0800
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7indt$8qh$1@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
Message-ID: <CAPJVwBkGjCD7MD7C_rypwGfxVpSOqRbf1r6L8U1e8tKDoAhkqA@mail.gmail.com>

On Mon, Dec 2, 2013 at 11:35 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I don't think that behavior is acceptable.

That's... too bad? I'm not sure what your objection actually is.

It's an intentional change (though disabled by default in 1.8), and a
necessary step to rationalizing our definition of contiguity and
stride handling in general, which has a number of benefits:
  http://docs.scipy.org/doc/numpy-dev/release.html#npy-relaxed-strides-checking
  http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memory-layout-of-an-ndarray

Why do you care about the stride of an array with only 1 element,
where by definition you never use the stride?

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


From sebastian at sipsolutions.net  Mon Dec  2 16:48:21 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 02 Dec 2013 22:48:21 +0100
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <l7iobi$jfp$1@ger.gmane.org>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
	<l7iobi$jfp$1@ger.gmane.org>
Message-ID: <1386020901.22572.7.camel@sebastian-laptop>

On Mon, 2013-12-02 at 14:51 -0500, Neal Becker wrote:
> The software I'm using, which is 
> 
> https://github.com/ndarray/ndarray
> 
> does depend on this.  Am I the only one who thinks that this
> behavior is not desirable?
> 

Well, this is not meant to be the way for a release version of numpy.
The rational was that since arbitrary strides *are* possible for such
arrays, creating them with arbitrary strides when you use
NPY_RELAXED_STRIDES_CHECKING=1 helps finding bugs.
Of course you are right to suppose that a *new* array should generally
have nice and clean strides, but if your code does work with arrays you
did not yourself create, it cannot make these assumptions. And I think
most code does this. And there are some advantages to ignoring such
strides for the contiguous flags.

If your arrays are contiguous, you don't really need the strides (use
the itemsize instead). How is ndarray broken by this?

- Sebastian


> Fr?d?ric Bastien wrote:
> 
> > Just don't compile with NPY_RELAXED_STRIDES_CHECKING to have the old
> > behavior I think (which is an not always the same strides depending of
> > how it was created, I don't know if they changed that or not).
> > 
> > Do someone else recall the detail of this?
> > 
> > Fred
> > 
> > p.s. I didn't do this or asked for it. But this help test your
> > software to don't depend of the strides when shapes is 1.
> > 
> > On Mon, Dec 2, 2013 at 2:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> >> I don't think that behavior is acceptable.
> >>
> >> Fr?d?ric Bastien wrote:
> >>
> >>> It is the NPY_RELAXED_STRIDES_CHECKING=1 flag that caused this.
> >>>
> >>> Fred
> >>>
> >>> On Mon, Dec 2, 2013 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> >>>> I built using:
> >>>>
> >>>> CFLAGS='-march=native -O3' NPY_RELAXED_STRIDES_CHECKING=1 python3 setup.py
> >>>> install --user
> >>>>
> >>>>
> >>>> a?id wrote:
> >>>>
> >>>>> I get:
> >>>>>
> >>>>> In [4]: x.strides
> >>>>> Out[4]: (8,)
> >>>>>
> >>>>> Same architecture and OS, Numpy installed via Pip on Python 2.7.5.
> >>>>>
> >>>>>
> >>>>> On 2 December 2013 20:08, Neal Becker <ndbecker2 at gmail.com> wrote:
> >>>>>
> >>>>>> This is np 1.8.0 on fedora x86_64:
> >>>>>>
> >>>>>> In [5]: x =np.array ((1,))
> >>>>>>
> >>>>>> In [6]: x.shape
> >>>>>> Out[6]: (1,)
> >>>>>>
> >>>>>> In [7]: x.strides
> >>>>>> Out[7]: (9223372036854775807,)
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> NumPy-Discussion mailing list
> >>>>>> NumPy-Discussion at scipy.org
> >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> NumPy-Discussion mailing list
> >>>> NumPy-Discussion at scipy.org
> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>> _______________________________________________
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From talljimbo at gmail.com  Mon Dec  2 18:15:50 2013
From: talljimbo at gmail.com (Jim Bosch)
Date: Mon, 2 Dec 2013 18:15:50 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <1386020901.22572.7.camel@sebastian-laptop>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
	<l7iobi$jfp$1@ger.gmane.org>
	<1386020901.22572.7.camel@sebastian-laptop>
Message-ID: <CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>

> If your arrays are contiguous, you don't really need the strides (use the
itemsize instead). How is ndarray broken by this?

ndarray is broken by this change because it expects the stride to be a
multiple of the itemsize (I think; I'm just looking at code here, as I
haven't had time to build NumPy 1.8 yet to test this); it has a slightly
more restricted model for what data can look like than NumPy has, and it's
easier to always just look at the stride for all sizes rather than
special-case for size=1.  I think that means the bug is ndarray's (indeed,
it's probably the kind of bug this new behavior was intended to catch, as I
should be handling the case of non-itemsize-multiple strides more
gracefully even when size > 1), and I'm working on a fix for it there now.

Thanks, Neil, for bringing this to my attention, and to all the NumPy dev's
for help in explaining what's going on.

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131202/18aab7e6/attachment.html>

From njs at pobox.com  Mon Dec  2 18:20:48 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Dec 2013 15:20:48 -0800
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
	<l7iobi$jfp$1@ger.gmane.org>
	<1386020901.22572.7.camel@sebastian-laptop>
	<CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>
Message-ID: <CAPJVwBnzztROo3FXcND3j5BT8MtJOZ8N1JhR4CQ92LS32v6wLg@mail.gmail.com>

On Mon, Dec 2, 2013 at 3:15 PM, Jim Bosch <talljimbo at gmail.com> wrote:
>> If your arrays are contiguous, you don't really need the strides (use the
>> itemsize instead). How is ndarray broken by this?
>
> ndarray is broken by this change because it expects the stride to be a
> multiple of the itemsize (I think; I'm just looking at code here, as I
> haven't had time to build NumPy 1.8 yet to test this); it has a slightly
> more restricted model for what data can look like than NumPy has, and it's
> easier to always just look at the stride for all sizes rather than
> special-case for size=1.

Note that arrays in which any dimension is 0 (i.e., 0 total elements)
can also have arbitrary strides with no consequence.

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


From sebastian at sipsolutions.net  Mon Dec  2 18:37:42 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 03 Dec 2013 00:37:42 +0100
Subject: [Numpy-discussion] nasty bug in 1.8.0??
In-Reply-To: <CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
	<l7iobi$jfp$1@ger.gmane.org>
	<1386020901.22572.7.camel@sebastian-laptop>
	<CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>
Message-ID: <1386027462.23220.9.camel@sebastian-laptop>

On Mon, 2013-12-02 at 18:15 -0500, Jim Bosch wrote:
> > If your arrays are contiguous, you don't really need the strides
> (use the itemsize instead). How is ndarray broken by this?
> 
> ndarray is broken by this change because it expects the stride to be a
> multiple of the itemsize (I think; I'm just looking at code here, as I
> haven't had time to build NumPy 1.8 yet to test this); it has a
> slightly more restricted model for what data can look like than NumPy
> has, and it's easier to always just look at the stride for all sizes
> rather than special-case for size=1.  I think that means the bug is
> ndarray's (indeed, it's probably the kind of bug this new behavior was
> intended to catch, as I should be handling the case of
> non-itemsize-multiple strides more gracefully even when size > 1), and
> I'm working on a fix for it there now.
> 
Most bugs I saw were just simply assuming:

arr.strides[-1] == arr.itemsize

when the array is C-contiguous, and could be fixed by just using
arr.itemsize...

Unless you need to calculate contiguous flags which are compatible to
NumPy with NPY_RELAXED_STRIDES_CHECKING (i.e. cython had this problem
since its memoryview would reject numpy's contiguous arrays as not
contiguous), you should not need to special case anything.

- Sebastian
> 
> Thanks, Neil, for bringing this to my attention, and to all the NumPy
> dev's for help in explaining what's going on.
> 
> 
> 
> Jim
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From ndbecker2 at gmail.com  Mon Dec  2 19:05:41 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Dec 2013 19:05:41 -0500
Subject: [Numpy-discussion] nasty bug in 1.8.0??
References: <l7ilrp$jie$1@ger.gmane.org>
	<CAJhcF=1v_1QXO-VtGVne6J_vW4s7SEfkmN-jLYkoJQ6rxX9r8g@mail.gmail.com>
	<l7imd0$jie$2@ger.gmane.org>
	<CADKKbtiYm2ZocYwYemCsB_gz=vCt2NxMAGD95TcYy+=JhU2vKg@mail.gmail.com>
	<l7indt$8qh$1@ger.gmane.org>
	<CADKKbtit7m+aixMgju+ihvm1W3=01LHmvveFxZ3jfqFP2wvVOQ@mail.gmail.com>
	<l7iobi$jfp$1@ger.gmane.org>
	<1386020901.22572.7.camel@sebastian-laptop>
	<CAN+0q24=x1mZbMCksosfV_jWd9Gpdn4CF=_6N9nbivtvJuUQZA@mail.gmail.com>
Message-ID: <l7j78b$6j5$1@ger.gmane.org>

Jim Bosch wrote:

>> If your arrays are contiguous, you don't really need the strides (use the
> itemsize instead). How is ndarray broken by this?
> 
> ndarray is broken by this change because it expects the stride to be a
> multiple of the itemsize (I think; I'm just looking at code here, as I
> haven't had time to build NumPy 1.8 yet to test this); it has a slightly
> more restricted model for what data can look like than NumPy has, and it's
> easier to always just look at the stride for all sizes rather than
> special-case for size=1.  I think that means the bug is ndarray's (indeed,
> it's probably the kind of bug this new behavior was intended to catch, as I
> should be handling the case of non-itemsize-multiple strides more
> gracefully even when size > 1), and I'm working on a fix for it there now.
> 
> Thanks, Neil, for bringing this to my attention, and to all the NumPy dev's
> for help in explaining what's going on.
> 
> Jim

The problem I encountered, is that canonical generic c++ code looks like:

template<typename in_t>
void F (in_t in) {
  int size = boost::size (in);
...

This fails when "in" is nd::Array<T,1,0>.  In that case, the iterator is 
strided_iterator.  And here, I find (via gdb), that stride==0.

The failure occurs here:

StridedIterator.h:

    template <typename U>
    int distance_to(StridedIterator<U> const & other) const {
        return std::distance(_data, other._data) / _stride; 
    }

How it happens that stride==0, and how to fix it, I don't know.




From francesc at continuum.io  Tue Dec  3 05:40:55 2013
From: francesc at continuum.io (Francesc Alted)
Date: Tue, 03 Dec 2013 11:40:55 +0100
Subject: [Numpy-discussion] -ffast-math
In-Reply-To: <loom.20131202T001342-893@post.gmane.org>
References: <loom.20131129T211145-800@post.gmane.org>	<5299CCE9.90602@googlemail.com>	<loom.20131201T214042-47@post.gmane.org>	<529BAA7B.3070206@googlemail.com>	<loom.20131201T225602-140@post.gmane.org>	<529BB548.6040806@googlemail.com>	<loom.20131202T000131-727@post.gmane.org>
	<loom.20131202T001342-893@post.gmane.org>
Message-ID: <529DB537.4090103@continuum.io>

On 12/2/13, 12:14 AM, Dan Goodman wrote:
> Dan Goodman <dg.gmane <at> thesamovar.net> writes:
> ...
>> I got around 5x slower. Using numexpr 'dumbly' (i.e. just putting the
>> expression in directly) was slower than the function above, but doing a
>> hybrid between the two approaches worked well:
>>
>> def timefunc_numexpr_smart():
>>      _sin_term = sin(2.0*freq*pi*t)
>>      _exp_term = exp(-dt/tau)
>>      _a_term = (_sin_term-_sin_term*_exp_term)
>>      _const_term = -b*_exp_term + b
>>      v[:] = numexpr.evaluate('a*_a_term+v*_exp_term+_const_term')
>>      #numexpr.evaluate('a*_a_term+v*_exp_term+_const_term', out=v)
>>
>> This was about 3.5x slower than weave. If I used the commented out final
>> line then it was only 1.5x slower than weave, but it also gives wrong
>> results. I reported this as a bug in numexpr a long time ago but I guess it
>> hasn't been fixed yet (or maybe I didn't upgrade my version recently).
> I just upgraded numexpr to 2.2 where they did fix this bug, and now the
> 'smart' numexpr version runs exactly as fast as weave (so I guess there were
> some performance enhancements in numexpr as well).

Err no, there have not been performance improvements in numexpr since 
2.0 (that I am aware of).  Maybe you are running in a multi-core machine 
now and you are seeing better speedup because of this?  Also, your 
expressions are made of transcendental functions, so linking numexpr 
with MKL could accelerate computations a good deal too.

-- 
Francesc Alted



From cmkleffner at gmail.com  Tue Dec  3 07:02:09 2013
From: cmkleffner at gmail.com (Carl Kleffner)
Date: Tue, 3 Dec 2013 13:02:09 +0100
Subject: [Numpy-discussion] how to use install_clib with setup.py?
Message-ID: <CAGGsPMydHuv9N8yk855-ZUt34_fgbh8N8UGp9v2BFxUnp_JU6w@mail.gmail.com>

I have a question concerning install_clib on windows.

What I want to do is to copy a dll (libopenblas.dll) to the numpy/core
folder with setup.py install. The path to the dll is given in site.cfg. The
dll itself is a external dependency.

Somewhere i found a reference to install_clib to copy external dll's, but
howto to apply this within setup.py?

Regards

Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131203/38248aec/attachment.html>

From nouiz at nouiz.org  Tue Dec  3 14:50:37 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 3 Dec 2013 14:50:37 -0500
Subject: [Numpy-discussion] Theano 0.6 released
Message-ID: <CADKKbtjc_Thd8u=SO134YtpVE0KpoFJYY1dWo-CBRp_hO6JNNA@mail.gmail.com>

What's New
----------

We recommend that everybody update to this version.


Highlights (since 0.6rc5):
 * Last release with support for Python 2.4 and 2.5.
 * We will try to release more frequently.
 * Fix crash/installation problems.
 * Use less memory for conv3d2d.

0.6rc4 skipped for a technical reason.

Highlights (since 0.6rc3):
 * Python 3.3 compatibility with buildbot test for it.
 * Full advanced indexing support.
 * Better Windows 64 bit support.
 * New profiler.
 * Better error messages that help debugging.
 * Better support for newer NumPy versions (remove useless warning/crash).
 * Faster optimization/compilation for big graph.
 * Move in Theano the Conv3d2d implementation.
 * Better SymPy/Theano bridge: Make an Theano op from SymPy expression
and use SymPy c code generator.
 * Bug fixes.

Change from 0.6rc5:
 * Fix crash when specifing march in cxxflags Theano flag. (Frederic
B., reported by FiReTiTi)
 * code cleanup (Jorg Bornschein)
 * Fix Canopy installation on windows when it was installed for all
users: Raingo
 * Fix Theano tests due to a scipy change. (Frederic B.)
 * Work around bug introduced in scipy dev 0.14. (Frederic B.)
 * Fix Theano tests following bugfix in SciPy. (Frederic B., reported
by Ziyuan Lin)
 * Add Theano flag cublas.lib (Misha Denil)
 * Make conv3d2d work more inplace (so less memory usage) (Frederic
B., repoted by Jean-Philippe Ouellet)

See https://pypi.python.org/pypi/Theano for more details.


Download and Install
--------------------

You can download Theano from http://pypi.python.org/pypi/Theano

Installation instructions are available at
http://deeplearning.net/software/theano/install.html

Description
-----------

Theano is a Python library that allows you to define, optimize, and
efficiently evaluate mathematical expressions involving
multi-dimensional arrays. It is built on top of NumPy. Theano
features:

 * tight integration with NumPy: a similar interface to NumPy's.
   numpy.ndarrays are also used internally in Theano-compiled functions.
 * transparent use of a GPU: perform data-intensive computations up to
   140x faster than on a CPU (support for float32 only).
 * efficient symbolic differentiation: Theano can compute derivatives
   for functions of one or many inputs.
 * speed and stability optimizations: avoid nasty bugs when computing
   expressions such as log(1+ exp(x)) for large values of x.
 * dynamic C code generation: evaluate expressions faster.
 * extensive unit-testing and self-verification: includes tools for
   detecting and diagnosing bugs and/or potential problems.

Theano has been powering large-scale computationally intensive
scientific research since 2007, but it is also approachable
enough to be used in the classroom (IFT6266 at the University of Montreal).

Resources
---------

About Theano:

http://deeplearning.net/software/theano/

Theano-related projects:

http://github.com/Theano/Theano/wiki/Related-projects

About NumPy:

http://numpy.scipy.org/

About SciPy:

http://www.scipy.org/

Machine Learning Tutorial with Theano on Deep Architectures:

http://deeplearning.net/tutorial/

Acknowledgments
---------------


I would like to thank all contributors of Theano. For this particular
release (since 0.5), many people have helped, notably:


Frederic Bastien
Pascal Lamblin
Ian Goodfellow
Olivier Delalleau
Razvan Pascanu
abalkin
Arnaud Bergeron
Nicolas Bouchard +
Jeremiah Lowin +
Matthew Rocklin
Eric Larsen +
James Bergstra
David Warde-Farley
John Salvatier +
Vivek Kulkarni +
Yann N. Dauphin
Ludwig Schmidt-Hackenberg +
Gabe Schwartz +
Rami Al-Rfou' +
Guillaume Desjardins
Caglar +
Sigurd Spieckermann +
Steven Pigeon +
Bogdan Budescu +
Jey Kottalam +
Mehdi Mirza +
Alexander Belopolsky +
Ethan Buchman +
Jason Yosinski
Nicolas Pinto +
Sina Honari +
Ben McCann +
Graham Taylor
Hani Almousli
Ilya Dyachenko +
Jan Schl?ter +
Jorg Bornschein +
Micky Latowicki +
Yaroslav Halchenko +
Eric Hunsberger +
Amir Elaguizy +
Hannes Schulz +
Huy Nguyen +
Ilan Schnell +
Li Yao
Misha Denil +
Robert Kern +
Sebastian Berg +
Vincent Dumoulin +
Wei Li +
XterNalz +


A total of 51 people contributed to this release.
People with a "+" by their names contributed a patch for the first time.

Also, thank you to all NumPy and Scipy developers as Theano builds on
their strengths.

All questions/comments are always welcome on the Theano
mailing-lists ( http://deeplearning.net/software/theano/#community )


From jslavin at cfa.harvard.edu  Thu Dec  5 11:04:19 2013
From: jslavin at cfa.harvard.edu (Slavin, Jonathan)
Date: Thu, 5 Dec 2013 11:04:19 -0500
Subject: [Numpy-discussion] no more search capability?
Message-ID: <CACcRS=ejWYUdR-fJtZvYG5ca-S3OMVMDjrUZhLRMMNz0-_6DsA@mail.gmail.com>

Hi all,

Although I like the look of the newly designed numpy/scipy web pages, I
have to say that I really miss the search capability.  Is there any motion
toward restoring that?

Jon
-- 
________________________________________________________
Jonathan D. Slavin                 Harvard-Smithsonian CfA
jslavin at cfa.harvard.edu       60 Garden Street, MS 83
phone: (617) 496-7981       Cambridge, MA 02138-1516
fax: (617) 496-7577            USA
________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/97690366/attachment.html>

From jslavin at cfa.harvard.edu  Thu Dec  5 13:30:47 2013
From: jslavin at cfa.harvard.edu (Slavin, Jonathan)
Date: Thu, 5 Dec 2013 13:30:47 -0500
Subject: [Numpy-discussion] no more search capability?
Message-ID: <CACcRS=dwi7_M5y1C07iK=pfo7TO=JUXABjgy0HuuTpbNg+LEpg@mail.gmail.com>

Answering part of my own question, I see that there is still a search
capability on one of the numpy web pages, but it's not where it used to be
and, in my opinion, is not easy to find.  There used to be a search box on
each web page of the numpy docs.

Jon

On Thu, Dec 5, 2013 at 1:00 PM, <numpy-discussion-request at scipy.org> wrote:

> Send NumPy-Discussion mailing list submissions to
>         numpy-discussion at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.scipy.org/mailman/listinfo/numpy-discussion
> or, via email, send a message with subject or body 'help' to
>         numpy-discussion-request at scipy.org
>
> You can reach the person managing the list at
>         numpy-discussion-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of NumPy-Discussion digest..."
>
> Today's Topics:
>
>    1. no more search capability? (Slavin, Jonathan)
>
>
> ---------- Forwarded message ----------
> From: "Slavin, Jonathan" <jslavin at cfa.harvard.edu>
> To: numpy-discussion at scipy.org
> Cc:
> Date: Thu, 5 Dec 2013 11:04:19 -0500
> Subject: [Numpy-discussion] no more search capability?
> Hi all,
>
> Although I like the look of the newly designed numpy/scipy web pages, I
> have to say that I really miss the search capability.  Is there any motion
> toward restoring that?
>
> Jon
> --
> ________________________________________________________
> Jonathan D. Slavin                 Harvard-Smithsonian CfA
> jslavin at cfa.harvard.edu       60 Garden Street, MS 83
> phone: (617) 496-7981       Cambridge, MA 02138-1516
> fax: (617) 496-7577            USA
> ________________________________________________________
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
________________________________________________________
Jonathan D. Slavin                 Harvard-Smithsonian CfA
jslavin at cfa.harvard.edu       60 Garden Street, MS 83
phone: (617) 496-7981       Cambridge, MA 02138-1516
fax: (617) 496-7577            USA
________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/c798625e/attachment.html>

From sebastian at sipsolutions.net  Thu Dec  5 17:37:19 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 05 Dec 2013 23:37:19 +0100
Subject: [Numpy-discussion] Deprecate boolean math operators?
Message-ID: <1386283039.23728.6.camel@sebastian-laptop>

Hey,

there was a discussion that for numpy booleans math operators +,-,* (and
the unary -), while defined, are not very helpful. I have set up a quick
PR with start (needs some fixes inside numpy still):

https://github.com/numpy/numpy/pull/4105

The idea is to deprecate these, since the binary operators |,^,| (and
the unary ~ even if it is weird) behave identical. This would not affect
sums of boolean arrays. For the moment I saw one "annoying" change in
numpy, and that is `abs(x - y)` being used for allclose and working
nicely currently.

- Sebastian



From ndarray at mac.com  Thu Dec  5 18:49:31 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Thu, 5 Dec 2013 18:49:31 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386283039.23728.6.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>
Message-ID: <CAP7h-xaqFDmVH=HknO18yfoq340QpyGO+LUFV=BY5=sJhXLbvw@mail.gmail.com>

On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> For the moment I saw one "annoying" change in
> numpy, and that is `abs(x - y)` being used for allclose and working
> nicely currently.
>

It would probably be an improvement if allclose returned all(x == y) unless
one of the arguments is inexact.  At the moment allclose() fails for char
arrays:

>>> allclose('abc', 'abc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numpy/core/numeric.py", line 2114, in allclose
    xinf = isinf(x)
TypeError: Not implemented for this type
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/248ad012/attachment.html>

From fmmirzaei at gmail.com  Thu Dec  5 22:14:01 2013
From: fmmirzaei at gmail.com (Faraz Mirzaei)
Date: Thu, 5 Dec 2013 19:14:01 -0800
Subject: [Numpy-discussion] surprising behavior of np.asarray on masked
	arrays
Message-ID: <CAJ0y5L_-b9U6gu2cZmMfwAzaYgLqWLg8UOP9iF5NzU90FWjqGw@mail.gmail.com>

Hi,

If I pass a masked array through np.asarray, I get original unmasked array.

Example:

test = np.array([[1, 0], [-1, 3]])

testMasked = ma.masked_less_equal(test, 0)


print testMasked

[[1 --]

 [-- 3]]


print testMasked.fill_value

999999


print np.asarray(testMasked)

[[ 1 0]

 [-1 3]]


Is this behavior intentional? How does the np.asarray access the original
masked values? Shouldn't the masked values be at least filled with
fill_value?


Thanks,


Faraz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/cf9143d9/attachment.html>

From josef.pktd at gmail.com  Thu Dec  5 22:33:02 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 5 Dec 2013 22:33:02 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386283039.23728.6.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>
Message-ID: <CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>

On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> Hey,
>
> there was a discussion that for numpy booleans math operators +,-,* (and
> the unary -), while defined, are not very helpful. I have set up a quick
> PR with start (needs some fixes inside numpy still):
>
> https://github.com/numpy/numpy/pull/4105
>
> The idea is to deprecate these, since the binary operators |,^,| (and
> the unary ~ even if it is weird) behave identical. This would not affect
> sums of boolean arrays. For the moment I saw one "annoying" change in
> numpy, and that is `abs(x - y)` being used for allclose and working
> nicely currently.

I like mask = mask1 * mask2

That's what I learned working my way through scipy.stats.distributions
a long time ago.

But the main thing is that we use boolean often as 0,1 integer array
in the actual calculations, and I only sometimes add the astype(int)

x[:, None] * (y[:, None] == np.unique(y))

I always thought booleans *are* just 0, 1 integers, until last time
there was the discussion we saw the weird + or - behavior.

We also use rescaling to (-1, 1) in statsmodels   y = mask * 2 - 1
(but maybe we convert to integer first)
My guess is that I only use multiplication heavily, where the boolean
is a dummy variable with 0 if male and 1 if female for example.

Nothing serious but nice not to have to worry about casting with
astype(int) first.

x[:, None] * (y[:, None] == np.unique(y)).astype(int)     (Is the
bracket at the right spot ?)

Josef


>
> - Sebastian
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From josef.pktd at gmail.com  Thu Dec  5 22:35:34 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 5 Dec 2013 22:35:34 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
Message-ID: <CAMMTP+AWA6ELejs9928L_y=uNSpfXgRYJgoutc2fZ_YYWO2VtA@mail.gmail.com>

On Thu, Dec 5, 2013 at 10:33 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> Hey,
>>
>> there was a discussion that for numpy booleans math operators +,-,* (and
>> the unary -), while defined, are not very helpful. I have set up a quick
>> PR with start (needs some fixes inside numpy still):
>>
>> https://github.com/numpy/numpy/pull/4105
>>
>> The idea is to deprecate these, since the binary operators |,^,| (and
>> the unary ~ even if it is weird) behave identical. This would not affect
>> sums of boolean arrays. For the moment I saw one "annoying" change in
>> numpy, and that is `abs(x - y)` being used for allclose and working
>> nicely currently.
>
> I like mask = mask1 * mask2
>
> That's what I learned working my way through scipy.stats.distributions
> a long time ago.
>
> But the main thing is that we use boolean often as 0,1 integer array
> in the actual calculations, and I only sometimes add the astype(int)
>
> x[:, None] * (y[:, None] == np.unique(y))
>
> I always thought booleans *are* just 0, 1 integers, until last time
> there was the discussion we saw the weird + or - behavior.
>
> We also use rescaling to (-1, 1) in statsmodels   y = mask * 2 - 1
> (but maybe we convert to integer first)
> My guess is that I only use multiplication heavily, where the boolean
> is a dummy variable with 0 if male and 1 if female for example.
>
> Nothing serious but nice not to have to worry about casting with
> astype(int) first.
>
> x[:, None] * (y[:, None] == np.unique(y)).astype(int)     (Is the
> bracket at the right spot ?)


what about np.dot,    np.dot(mask, x) which is the same as (mask * x).sum(0) ?

Josef

>
> Josef
>
>
>>
>> - Sebastian
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From ndarray at mac.com  Thu Dec  5 22:56:48 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Thu, 5 Dec 2013 22:56:48 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386283039.23728.6.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>
Message-ID: <CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>

On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg <sebastian at sipsolutions.net>
wrote:
> there was a discussion that for numpy booleans math operators +,-,* (and
> the unary -), while defined, are not very helpful.

It has been suggested at the Github that there is an area where it is
useful to have linear algebra operations like matrix multiplication to be
defined over a semiring:

http://en.wikipedia.org/wiki/Logical_matrix

This still does not justify having unary or binary -, so I suggest that we
first discuss deprecation of those.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/1cf81c51/attachment.html>

From ndarray at mac.com  Thu Dec  5 23:00:50 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Thu, 5 Dec 2013 23:00:50 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+AWA6ELejs9928L_y=uNSpfXgRYJgoutc2fZ_YYWO2VtA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<CAMMTP+AWA6ELejs9928L_y=uNSpfXgRYJgoutc2fZ_YYWO2VtA@mail.gmail.com>
Message-ID: <CAP7h-xY8g5jbmG7b-nAOmL5Zvh-TYOgLqUL+Bh5FUoOe9x_cXQ@mail.gmail.com>

On Thu, Dec 5, 2013 at 10:35 PM, <josef.pktd at gmail.com> wrote:

> what about np.dot,    np.dot(mask, x) which is the same as (mask *
> x).sum(0) ?


I am not sure which way your argument goes, but I don't think you would
find the following natural:

>>> x = array([True, True])
>>> dot(x,x)
True
>>> (x*x).sum()
2
>>> (x*x).sum(0)
2
>>> (x*x).sum(False)
2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/6c1f68a8/attachment.html>

From josef.pktd at gmail.com  Thu Dec  5 23:02:34 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 5 Dec 2013 23:02:34 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>
Message-ID: <CAMMTP+DnJS=ne-VCpRwaq=pJh48oOpPP0G7YX-5i1Waqm2zoYg@mail.gmail.com>

On Thu, Dec 5, 2013 at 10:56 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
> On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg <sebastian at sipsolutions.net>
> wrote:
>> there was a discussion that for numpy booleans math operators +,-,* (and
>> the unary -), while defined, are not very helpful.
>
> It has been suggested at the Github that there is an area where it is useful
> to have linear algebra operations like matrix multiplication to be defined
> over a semiring:
>
> http://en.wikipedia.org/wiki/Logical_matrix
>
> This still does not justify having unary or binary -, so I suggest that we
> first discuss deprecation of those.

Does it make sense to only remove - and maybe / ?

would python sum still work?   (I almost never use it.)

>>> sum(mask)
2
>>> sum(mask.tolist())
2

is accumulate the same as sum and would keep working?

>>> np.add.accumulate(mask)
array([0, 0, 0, 1, 2])


In operation with other dtypes, do they still dominate so these work?

>>> x / mask
array([0, 0, 0, 3, 4])
>>> x * 1. / mask
array([ nan,  inf,  inf,   3.,   4.])
>>> x**mask
array([1, 1, 1, 3, 4])
>>> mask - 5
array([-5, -5, -5, -4, -4])

Josef

>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From alan.isaac at gmail.com  Thu Dec  5 23:05:32 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Thu, 05 Dec 2013 23:05:32 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
Message-ID: <52A14D0C.1060209@gmail.com>

For + and * (and thus `dot`), this will "fix" something that is not broken.
It is in fact in conformance with a large literature on boolean arrays
and boolean matrices.  That not everyone pays attention to this literature
does not constitute a reason to break the extant, correct behavior.

I'm sure I cannot be the only one who has for years taught students
about Boolean matrices using NumPy, because of this correct behavior
of this dtype. (By correct, I mean in conformance with the literature.)

Alan Isaac


From ndarray at mac.com  Thu Dec  5 23:14:52 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Thu, 5 Dec 2013 23:14:52 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A14D0C.1060209@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
Message-ID: <CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>

On Thu, Dec 5, 2013 at 11:05 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> For + and * (and thus `dot`), this will "fix" something that is not broken.


+ and * are not broken - just redundant given | and &.

What is really broken is -, both unary and binary:

>>> int(np.bool_(0) - np.bool_(1))
1
>>> int(-np.bool_(0))
1

> I'm sure I cannot be the only one who has for years taught students
> about Boolean matrices using NumPy

(I would not be so sure:-)

In that experience, did you find minus to be as useful?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131205/34c04e5f/attachment.html>

From stefan at sun.ac.za  Thu Dec  5 23:22:18 2013
From: stefan at sun.ac.za (=?iso-8859-1?Q?St=E9fan?= van der Walt)
Date: Fri, 6 Dec 2013 06:22:18 +0200
Subject: [Numpy-discussion] surprising behavior of np.asarray on masked
 arrays
In-Reply-To: <CAJ0y5L_-b9U6gu2cZmMfwAzaYgLqWLg8UOP9iF5NzU90FWjqGw@mail.gmail.com>
References: <CAJ0y5L_-b9U6gu2cZmMfwAzaYgLqWLg8UOP9iF5NzU90FWjqGw@mail.gmail.com>
Message-ID: <20131206042218.GA11321@shinobi>

Hi Faraz

On Thu, 05 Dec 2013 19:14:01 -0800, Faraz Mirzaei wrote:
> If I pass a masked array through np.asarray, I get original unmasked array.

`asarray` disregards any information attached to the underlying ndarray by the
subclass.  To preserve the subclass, you'd need to use `asanyarray`.

The only functions that are aware of masked arrays live inside of `np.ma`, so
you can also use `np.ma.asarray`.

Which behavior in particular would you like to see, since I presume you can
already get hold of the filled array, should you want to?

St?fan



From josef.pktd at gmail.com  Thu Dec  5 23:28:47 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 5 Dec 2013 23:28:47 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xY8g5jbmG7b-nAOmL5Zvh-TYOgLqUL+Bh5FUoOe9x_cXQ@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<CAMMTP+AWA6ELejs9928L_y=uNSpfXgRYJgoutc2fZ_YYWO2VtA@mail.gmail.com>
	<CAP7h-xY8g5jbmG7b-nAOmL5Zvh-TYOgLqUL+Bh5FUoOe9x_cXQ@mail.gmail.com>
Message-ID: <CAMMTP+ABosuf0Ph-84uHG2H+nPG9LFeBDO=VTrRxPLOQ_EfzLA@mail.gmail.com>

On Thu, Dec 5, 2013 at 11:00 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
>
> On Thu, Dec 5, 2013 at 10:35 PM, <josef.pktd at gmail.com> wrote:
>>
>> what about np.dot,    np.dot(mask, x) which is the same as (mask *
>> x).sum(0) ?
>
>
> I am not sure which way your argument goes, but I don't think you would find
> the following natural:
>
>>>> x = array([True, True])
>>>> dot(x,x)
> True

this is weird but I would never do that.  maybe I would, but then i
would add 1 non boolean


>>>> (x*x).sum()
> 2
>>>> (x*x).sum(0)
> 2

That sounds right to me
>>> (mask**2 == mask).all()
True


>>>> (x*x).sum(False)
> 2

What is axis=False?


The way my argument goes:
I'm a heavy user of using * pretending the bool behaves like an int,
and of sum and accumulate.
It would be a pain to loose them.

>From where I come from (*) a bool is not a boolean it's just 0, 1,
given that numpy casting rules apply and it's sometimes cast back to
(0, 1)   Does this work as explanation for the pattern of + and -
also.

(*) places where the type system is more restricted.


What about max?

>>> np.maximum(mask, mask)
array([False, False, False,  True,  True], dtype=bool)
>>> np.maximum(mask, ~mask)
array([ True,  True,  True,  True,  True], dtype=bool)
>>> mask + mask
array([False, False, False,  True,  True], dtype=bool)
>>> mask + ~mask
array([ True,  True,  True,  True,  True], dtype=bool)

first mask is if the wife has a car, second mask is if the husband has a car.
The max is if there is a car in the family.

What's this as logical?

Josef


>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From efiring at hawaii.edu  Fri Dec  6 01:53:48 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Thu, 05 Dec 2013 20:53:48 -1000
Subject: [Numpy-discussion] surprising behavior of np.asarray on masked
 arrays
In-Reply-To: <CAJ0y5L_-b9U6gu2cZmMfwAzaYgLqWLg8UOP9iF5NzU90FWjqGw@mail.gmail.com>
References: <CAJ0y5L_-b9U6gu2cZmMfwAzaYgLqWLg8UOP9iF5NzU90FWjqGw@mail.gmail.com>
Message-ID: <52A1747C.1040200@hawaii.edu>

On 2013/12/05 5:14 PM, Faraz Mirzaei wrote:
> Hi,
>
> If I pass a masked array through np.asarray, I get original unmasked array.
>
> Example:
>
> test = np.array([[1, 0], [-1, 3]])
>
> testMasked = ma.masked_less_equal(test, 0)
>
>
> print testMasked
>
> [[1 --]
>
>   [-- 3]]
>
>
> print testMasked.fill_value
>
> 999999
>
>
> print np.asarray(testMasked)
>
> [[ 1 0]
>
>   [-1 3]]
>
>
> Is this behavior intentional? How does the np.asarray access the
> original masked values? Shouldn't the masked values be at least filled
> with fill_value?

It might be nice, but it's not the way it is.  If you want to preserve 
masked arrays, use np.asanyarray() instead of np.asarray().  If you want 
to end up with filled ndarrays, use np.ma.filled().

Eric

>
>
> Thanks,
>
>
> Faraz
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From njs at pobox.com  Fri Dec  6 03:24:30 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 6 Dec 2013 00:24:30 -0800
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
Message-ID: <CAPJVwBk+Q1ePeH3DqsoGXUxQRa91g=c8J9dPJqv6tWfeRiKkRw@mail.gmail.com>

On Thu, Dec 5, 2013 at 7:33 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> Hey,
>>
>> there was a discussion that for numpy booleans math operators +,-,* (and
>> the unary -), while defined, are not very helpful. I have set up a quick
>> PR with start (needs some fixes inside numpy still):
>>
>> https://github.com/numpy/numpy/pull/4105
>>
>> The idea is to deprecate these, since the binary operators |,^,| (and
>> the unary ~ even if it is weird) behave identical. This would not affect
>> sums of boolean arrays. For the moment I saw one "annoying" change in
>> numpy, and that is `abs(x - y)` being used for allclose and working
>> nicely currently.
>
> I like mask = mask1 * mask2
>
> That's what I learned working my way through scipy.stats.distributions
> a long time ago.

* is least problematic case, since there numpy and python bools
already almost agree. (They return the same values, but numpy returns
a bool array instead of an integer array.)

On Thu, Dec 5, 2013 at 8:05 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> For + and * (and thus `dot`), this will "fix" something that is not broken.
> It is in fact in conformance with a large literature on boolean arrays
> and boolean matrices.

Interesting point! I had assumed that dot() just upcast! But what do
you think about the inconsistency between sum() and dot() on bool
arrays?

-n


From sebastian at sipsolutions.net  Fri Dec  6 04:39:36 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 06 Dec 2013 10:39:36 +0100
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+DnJS=ne-VCpRwaq=pJh48oOpPP0G7YX-5i1Waqm2zoYg@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>
	<CAMMTP+DnJS=ne-VCpRwaq=pJh48oOpPP0G7YX-5i1Waqm2zoYg@mail.gmail.com>
Message-ID: <1386322776.3153.21.camel@sebastian-laptop>

On Thu, 2013-12-05 at 23:02 -0500, josef.pktd at gmail.com wrote:
> On Thu, Dec 5, 2013 at 10:56 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
> > On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg <sebastian at sipsolutions.net>
> > wrote:
> >> there was a discussion that for numpy booleans math operators +,-,* (and
> >> the unary -), while defined, are not very helpful.
> >
> > It has been suggested at the Github that there is an area where it is useful
> > to have linear algebra operations like matrix multiplication to be defined
> > over a semiring:
> >
> > http://en.wikipedia.org/wiki/Logical_matrix
> >
> > This still does not justify having unary or binary -, so I suggest that we
> > first discuss deprecation of those.
> 
> Does it make sense to only remove - and maybe / ?
> 
> would python sum still work?   (I almost never use it.)
> 
> >>> sum(mask)
> 2
> >>> sum(mask.tolist())
> 2
> 
> is accumulate the same as sum and would keep working?
> 
> >>> np.add.accumulate(mask)
> array([0, 0, 0, 1, 2])
> 
> 
> In operation with other dtypes, do they still dominate so these work?
> 

Hey,

of course the other types will always dominate interpreting bools as 0
and 1. This would only affect operations with only booleans. There is a
good point that * is well defined however you define it, though. (Btw. /
is not defined for bools, `np.bool_(True)/np.bool_(True)` will upcast to
int8 to do the operation)

However, while well defined, + is not defined like it is for python
bools (which are just ints) so that is the reason to consider
deprecation there (if we allow upcast to int8 -- or maybe the default
int -- in the future, in-place += and -= operations would not behave
differently, since they just cast back...).

I suppose python sum works because it first tries using the C-Api number
protocol, which also means it is not affected. If you were to write a
sum which just uses the `+` operator, it would be affected, but that
would seem good to me.

- Sebastian


> >>> x / mask
> array([0, 0, 0, 3, 4])
> >>> x * 1. / mask
> array([ nan,  inf,  inf,   3.,   4.])
> >>> x**mask
> array([1, 1, 1, 3, 4])
> >>> mask - 5
> array([-5, -5, -5, -4, -4])
> 
> Josef
> 
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Fri Dec  6 09:32:16 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 09:32:16 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386322776.3153.21.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>
	<CAMMTP+DnJS=ne-VCpRwaq=pJh48oOpPP0G7YX-5i1Waqm2zoYg@mail.gmail.com>
	<1386322776.3153.21.camel@sebastian-laptop>
Message-ID: <CAMMTP+Amt0adPEJKr83+48PaS-F_8jp=ULQsXNdKNOi5MowXYw@mail.gmail.com>

On Fri, Dec 6, 2013 at 4:39 AM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Thu, 2013-12-05 at 23:02 -0500, josef.pktd at gmail.com wrote:
>> On Thu, Dec 5, 2013 at 10:56 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
>> > On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg <sebastian at sipsolutions.net>
>> > wrote:
>> >> there was a discussion that for numpy booleans math operators +,-,* (and
>> >> the unary -), while defined, are not very helpful.
>> >
>> > It has been suggested at the Github that there is an area where it is useful
>> > to have linear algebra operations like matrix multiplication to be defined
>> > over a semiring:
>> >
>> > http://en.wikipedia.org/wiki/Logical_matrix
>> >
>> > This still does not justify having unary or binary -, so I suggest that we
>> > first discuss deprecation of those.
>>
>> Does it make sense to only remove - and maybe / ?
>>
>> would python sum still work?   (I almost never use it.)
>>
>> >>> sum(mask)
>> 2
>> >>> sum(mask.tolist())
>> 2
>>
>> is accumulate the same as sum and would keep working?
>>
>> >>> np.add.accumulate(mask)
>> array([0, 0, 0, 1, 2])
>>
>>
>> In operation with other dtypes, do they still dominate so these work?
>>
>
> Hey,


In statistics and econometrics (and economic theory) we just use an
indicator function 1_{x=5} which has largely the same properties as a
numpy bool array, at least in my code.

some of the common operations are *, dot and kron.

So far this has worked quite well as intuition, plus numpy casting rules.

dot is the main surprise, because I thought that it would upcast. (I
always think of dot as a np.linalg.)


>
> of course the other types will always dominate interpreting bools as 0
> and 1. This would only affect operations with only booleans.

My guess is that this would leave then 90% of our (statsmodels)
possible usage alone.

There is still the case that with * we can calculate the intersection.


There is a
> good point that * is well defined however you define it, though. (Btw. /
> is not defined for bools, `np.bool_(True)/np.bool_(True)` will upcast to
> int8 to do the operation)
>
> However, while well defined, + is not defined like it is for python
> bools (which are just ints) so that is the reason to consider
> deprecation there (if we allow upcast to int8 -- or maybe the default
> int -- in the future, in-place += and -= operations would not behave
> differently, since they just cast back...).

Actually, I used + once:

The calculation in terms of indicator functions is

1_{A} + 1_{B} - 1_{A & B}

The last part avoids double counting, which is not necessary if numpy
casts back to bool.
Nothing that couldn't be replaced by logical operators, but the
(linear) algebra is not "logical".

In this case I did care about memory because the arrays are (nobs,
nobs) (nobs is the number of observations shape[0]) which can be
large, and I have a sparse version also. In most other case we use
astype(int) already very often, because eventually we still have to
cast and memory won't be a big problem.

The mental model is set membership and set operations with indicator
functions, not "logical", and I don't remember running into problems
with this so far, and happily ignored logical_xxx when I do linear
algebra instead of just working with masks of booleans.

Nevertheless: If I'm forced to, then I will get used to logical_xxx. (*)
And the above bool addition hasn't made it into statsmodels yet. I
used a simpler version because I thought initially it's too cute. (And
I was using an older numpy that couldn't do broadcasted dot.)

(*) how do you search in the documentation of `&` or `|`, I cannot
find what the other symbols are, if there are any.

>
> I suppose python sum works because it first tries using the C-Api number
> protocol, which also means it is not affected. If you were to write a
> sum which just uses the `+` operator, it would be affected, but that
> would seem good to me.

based on the ticket example, I'm not sure whether `+` should upcast or not.

>>> mm.dtype
dtype('bool')
>>> mm.sum(0)
array([48, 45, 56, 47])

>>> mm.sum(0, bool)
array([ True,  True,  True,  True], dtype=bool)
I would just use any

but what happens with logical cumsum

>>> mm[:5].cumsum(0, bool)
array([[False,  True,  True,  True],
       [ True,  True,  True,  True],
       [ True,  True,  True,  True],
       [ True,  True,  True,  True],
       [ True,  True,  True,  True]], dtype=bool)

same as mm[:5].astype(int).cumsum(0, bool)  without casting

Josef


>
> - Sebastian
>
>
>> >>> x / mask
>> array([0, 0, 0, 3, 4])
>> >>> x * 1. / mask
>> array([ nan,  inf,  inf,   3.,   4.])
>> >>> x**mask
>> array([1, 1, 1, 3, 4])
>> >>> mask - 5
>> array([-5, -5, -5, -4, -4])
>>
>> Josef
>>
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From alan.isaac at gmail.com  Fri Dec  6 10:32:11 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 10:32:11 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAPJVwBk+Q1ePeH3DqsoGXUxQRa91g=c8J9dPJqv6tWfeRiKkRw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<CAPJVwBk+Q1ePeH3DqsoGXUxQRa91g=c8J9dPJqv6tWfeRiKkRw@mail.gmail.com>
Message-ID: <52A1EDFB.6090408@gmail.com>

> On Thu, Dec 5, 2013 at 8:05 PM, Alan G Isaac
> <alan.isaac at gmail.com> wrote:
>> For + and * (and thus `dot`), this will "fix" something that is not broken.
>> It is in fact in conformance with a large literature on boolean arrays
>> and boolean matrices.

On 12/6/2013 3:24 AM, Nathaniel Smith wrote:
> Interesting point! I had assumed that dot() just upcast! But what do
> you think about the inconsistency between sum() and dot() on bool
> arrays?


I don't like the behavior of sum on bool arrays.
(I.e., automatic upcasting.)
But I do not suggest changing it,
as much code is likely to depend on it.

Cheers,
Alan



From josef.pktd at gmail.com  Fri Dec  6 11:12:11 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 11:12:11 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+Amt0adPEJKr83+48PaS-F_8jp=ULQsXNdKNOi5MowXYw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAP7h-xbaow+YzwwcWRK1wR8aiYDHgzs-04iGw7pg9Jo+jGQCPw@mail.gmail.com>
	<CAMMTP+DnJS=ne-VCpRwaq=pJh48oOpPP0G7YX-5i1Waqm2zoYg@mail.gmail.com>
	<1386322776.3153.21.camel@sebastian-laptop>
	<CAMMTP+Amt0adPEJKr83+48PaS-F_8jp=ULQsXNdKNOi5MowXYw@mail.gmail.com>
Message-ID: <CAMMTP+Ag1nUfKyuvCULHe1+Ug40=6UMD9swfR=i5G_k8jJYM3A@mail.gmail.com>

On Fri, Dec 6, 2013 at 9:32 AM,  <josef.pktd at gmail.com> wrote:
> On Fri, Dec 6, 2013 at 4:39 AM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> On Thu, 2013-12-05 at 23:02 -0500, josef.pktd at gmail.com wrote:
>>> On Thu, Dec 5, 2013 at 10:56 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
>>> > On Thu, Dec 5, 2013 at 5:37 PM, Sebastian Berg <sebastian at sipsolutions.net>
>>> > wrote:
>>> >> there was a discussion that for numpy booleans math operators +,-,* (and
>>> >> the unary -), while defined, are not very helpful.
>>> >
>>> > It has been suggested at the Github that there is an area where it is useful
>>> > to have linear algebra operations like matrix multiplication to be defined
>>> > over a semiring:
>>> >
>>> > http://en.wikipedia.org/wiki/Logical_matrix
>>> >
>>> > This still does not justify having unary or binary -, so I suggest that we
>>> > first discuss deprecation of those.
>>>
>>> Does it make sense to only remove - and maybe / ?
>>>
>>> would python sum still work?   (I almost never use it.)
>>>
>>> >>> sum(mask)
>>> 2
>>> >>> sum(mask.tolist())
>>> 2
>>>
>>> is accumulate the same as sum and would keep working?
>>>
>>> >>> np.add.accumulate(mask)
>>> array([0, 0, 0, 1, 2])
>>>
>>>
>>> In operation with other dtypes, do they still dominate so these work?
>>>
>>
>> Hey,
>
>
> In statistics and econometrics (and economic theory) we just use an
> indicator function 1_{x=5} which has largely the same properties as a
> numpy bool array, at least in my code.
>
> some of the common operations are *, dot and kron.
>
> So far this has worked quite well as intuition, plus numpy casting rules.
>
> dot is the main surprise, because I thought that it would upcast. (I
> always think of dot as a np.linalg.)
>
>
>>
>> of course the other types will always dominate interpreting bools as 0
>> and 1. This would only affect operations with only booleans.
>
> My guess is that this would leave then 90% of our (statsmodels)
> possible usage alone.
>
> There is still the case that with * we can calculate the intersection.
>
>
> There is a
>> good point that * is well defined however you define it, though. (Btw. /
>> is not defined for bools, `np.bool_(True)/np.bool_(True)` will upcast to
>> int8 to do the operation)
>>
>> However, while well defined, + is not defined like it is for python
>> bools (which are just ints) so that is the reason to consider
>> deprecation there (if we allow upcast to int8 -- or maybe the default
>> int -- in the future, in-place += and -= operations would not behave
>> differently, since they just cast back...).
>
> Actually, I used + once:
>
> The calculation in terms of indicator functions is
>
> 1_{A} + 1_{B} - 1_{A & B}
>
> The last part avoids double counting, which is not necessary if numpy
> casts back to bool.
> Nothing that couldn't be replaced by logical operators, but the
> (linear) algebra is not "logical".
>
> In this case I did care about memory because the arrays are (nobs,
> nobs) (nobs is the number of observations shape[0]) which can be
> large, and I have a sparse version also. In most other case we use
> astype(int) already very often, because eventually we still have to
> cast and memory won't be a big problem.
>
> The mental model is set membership and set operations with indicator
> functions, not "logical", and I don't remember running into problems
> with this so far, and happily ignored logical_xxx when I do linear
> algebra instead of just working with masks of booleans.

http://en.wikipedia.org/wiki/Indicator_function
with the added advantage that we have also the version where +
constrains to (0, 1).
However `-` doesn't work properly because
>>> np.bool_(-5)
True
instead of False
except in the case `1 - mask`.

We really have two kinds of addition:

bool sum: for indicating set membership
counting sum: for counting number of elements.

from my viewpoint:

I would keep + and * since they work well     (bool + and count +)
minus - is partially broken and `/` looks useless

this casts anyway
>>> 1 - m1
array([1, 1, 0, 0, 0])

and I never thought of doing this
>>> True - m1
array([ True,  True, False, False, False], dtype=bool)

(python set defines minus but raises error on plus)

Josef

>
> Nevertheless: If I'm forced to, then I will get used to logical_xxx. (*)
> And the above bool addition hasn't made it into statsmodels yet. I
> used a simpler version because I thought initially it's too cute. (And
> I was using an older numpy that couldn't do broadcasted dot.)
>
> (*) how do you search in the documentation of `&` or `|`, I cannot
> find what the other symbols are, if there are any.
>
>>
>> I suppose python sum works because it first tries using the C-Api number
>> protocol, which also means it is not affected. If you were to write a
>> sum which just uses the `+` operator, it would be affected, but that
>> would seem good to me.
>
> based on the ticket example, I'm not sure whether `+` should upcast or not.
>
>>>> mm.dtype
> dtype('bool')
>>>> mm.sum(0)
> array([48, 45, 56, 47])
>
>>>> mm.sum(0, bool)
> array([ True,  True,  True,  True], dtype=bool)
> I would just use any
>
> but what happens with logical cumsum
>
>>>> mm[:5].cumsum(0, bool)
> array([[False,  True,  True,  True],
>        [ True,  True,  True,  True],
>        [ True,  True,  True,  True],
>        [ True,  True,  True,  True],
>        [ True,  True,  True,  True]], dtype=bool)
>
> same as mm[:5].astype(int).cumsum(0, bool)  without casting
>
> Josef
>
>
>>
>> - Sebastian
>>
>>
>>> >>> x / mask
>>> array([0, 0, 0, 3, 4])
>>> >>> x * 1. / mask
>>> array([ nan,  inf,  inf,   3.,   4.])
>>> >>> x**mask
>>> array([1, 1, 1, 3, 4])
>>> >>> mask - 5
>>> array([-5, -5, -5, -4, -4])
>>>
>>> Josef
>>>
>>> >
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From alan.isaac at gmail.com  Fri Dec  6 11:13:07 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 11:13:07 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
Message-ID: <52A1F793.3010506@gmail.com>

On 12/5/2013 11:14 PM, Alexander Belopolsky wrote:
> did you find minus to be as useful?


It is also a correct usage.

I think a good approach to this is to first realize that
there were good reasons for the current behavior.

Alan Isaac



From josef.pktd at gmail.com  Fri Dec  6 11:20:34 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 11:20:34 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A1F793.3010506@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
Message-ID: <CAMMTP+AZJdZYhaMpgSrS7cjQ1DNwpw2qiiU7FmXC7VnAEF45XQ@mail.gmail.com>

On Fri, Dec 6, 2013 at 11:13 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 12/5/2013 11:14 PM, Alexander Belopolsky wrote:
>> did you find minus to be as useful?
>
>
> It is also a correct usage.
>
> I think a good approach to this is to first realize that
> there were good reasons for the current behavior.

What's the meaning of minus?
I cannot make much sense out of it, or come up with any use case.

Josef

>
> Alan Isaac
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From ndarray at mac.com  Fri Dec  6 12:23:23 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Fri, 6 Dec 2013 12:23:23 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A1F793.3010506@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
Message-ID: <CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>

On Fri, Dec 6, 2013 at 11:13 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 12/5/2013 11:14 PM, Alexander Belopolsky wrote:
> > did you find minus to be as useful?
>
>
> It is also a correct usage.
>
>
Can you provide a reference?



> I think a good approach to this is to first realize that
> there were good reasons for the current behavior.
>
>
Maybe there were, in which case the current behavior should be documented
somewhere.

What is the rationale for this:

>>> -array(True) + array(True)
True

?

I am not aware of any algebraic system where unary minus denotes anything
other than additive inverse.

Having bools form a semiring under + and * is a fine (yet somewhat unusual)
choice, but once you've made that choice you loose subtraction because True
+ x = True no longer has a unique solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/44173195/attachment.html>

From josef.pktd at gmail.com  Fri Dec  6 12:47:06 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 12:47:06 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
Message-ID: <CAMMTP+DRD0KSU28T-OcTm0s3pvJJBKy4d-rVQmtPk-qjsXbj0g@mail.gmail.com>

On Fri, Dec 6, 2013 at 12:23 PM, Alexander Belopolsky <ndarray at mac.com> wrote:
> On Fri, Dec 6, 2013 at 11:13 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>
>> On 12/5/2013 11:14 PM, Alexander Belopolsky wrote:
>> > did you find minus to be as useful?
>>
>>
>> It is also a correct usage.
>>
>
> Can you provide a reference?
>
>
>>
>> I think a good approach to this is to first realize that
>> there were good reasons for the current behavior.
>>
>
> Maybe there were, in which case the current behavior should be documented
> somewhere.
>
> What is the rationale for this:
>
>>>> -array(True) + array(True)
> True
>
> ?
>
> I am not aware of any algebraic system where unary minus denotes anything
> other than additive inverse.

I would be perfectly happy if numpy would cast (negative) overflow to
the smallest value, instead of wrapping around.
The same is true for integers.

>>> np.array(0, np.int8) - np.array(-128, np.int8)
-128
>>> - np.array(-128, np.int8)
-128

Josef
<It's consistent. But does it make sense?>
>
> Having bools form a semiring under + and * is a fine (yet somewhat unusual)
> choice, but once you've made that choice you loose subtraction because True
> + x = True no longer has a unique solution.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ralf.gommers at gmail.com  Fri Dec  6 13:06:55 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 6 Dec 2013 19:06:55 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
Message-ID: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>

Hi all,

There are a few discussions on packaging for the scientific Python stack
ongoing, on the NumFOCUS and distutils lists:
https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY
http://thread.gmane.org/gmane.comp.python.distutils.devel/20202
http://thread.gmane.org/gmane.comp.python.distutils.devel/20296

One of the things that we should start doing for numpy is distribute
releases as wheels. On OS X at least this is quite simple, so I propose to
just experiment with it. I can create some to try out and put them on a
separate folder on SourceForge. If that works they can be put on PyPi.

For Windows things are less simple, because the wheel format doesn't handle
the multiple builds (no SSE, SSE2, SSE3) that are in the superpack
installers. A problem is that we don't really know how many users still
have old CPUs that don't support SSE3. The impact for those users is high,
numpy will install but crash (see https://github.com/scipy/scipy/issues/1697).
Questions:
1. does anyone have a good idea to obtain statistics?
2. in the absence of statistics, can we do an experiment by putting one
wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
propose, and seeing for how many (if any) users this goes wrong?

Ralf

P.S. related question: did anyone check whether the recently merged
NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/500e77e4/attachment.html>

From alan.isaac at gmail.com  Fri Dec  6 13:16:08 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 13:16:08 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
Message-ID: <52A21468.8010809@gmail.com>

On 12/6/2013 12:23 PM, Alexander Belopolsky wrote:
> What is the rationale for this:
>
>  >>> -array(True) + array(True)
> True


The minus is complementation.
So you are just writing
	False or True

Alan Isaac



From alan.isaac at gmail.com  Fri Dec  6 13:31:34 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 13:31:34 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
Message-ID: <52A21806.3080103@gmail.com>

>>     On 12/5/2013 11:14 PM, Alexander Belopolsky wrote:
>>> did you find minus to be as useful?

> On Fri, Dec 6, 2013 at 11:13 AM, Alan G Isaac
>> It is also a correct usage.

On 12/6/2013 12:23 PM, Alexander Belopolsky wrote:
> Can you provide a reference?


For use of the minus sign, I don't have one at hand, but
a quick Google seach comes up with:
http://www.csee.umbc.edu/~artola/fall02/BooleanAlgebra.ppt
It is more common to use a superscript `c`, but that's
just a notational issue.

For multiplication, addition, and dot,
you can see Ki Hang Kim's Boolean matrix
Theory and Applications.

Applications are endless and include graph theory
and (then naturally) circuit design.

Cheers,
Alan Isaac



From josef.pktd at gmail.com  Fri Dec  6 13:35:16 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 13:35:16 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A21468.8010809@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
Message-ID: <CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>

On Fri, Dec 6, 2013 at 1:16 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 12/6/2013 12:23 PM, Alexander Belopolsky wrote:
>> What is the rationale for this:
>>
>>  >>> -array(True) + array(True)
>> True
>
>
> The minus is complementation.
> So you are just writing
>         False or True

unary versus binary minus

>>> m1 + (-m2)
array([False, False,  True,  True,  True], dtype=bool)
>>> m1 - m2
array([ True,  True, False, False,  True], dtype=bool)

>>> -m2 + m1
array([False, False,  True,  True,  True], dtype=bool)

>>> m1 - (-m2)
array([False, False,  True,  True, False], dtype=bool)

I'd rather write ~ than unary - if that's what it is.

Josef


>
> Alan Isaac
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From alan.isaac at gmail.com  Fri Dec  6 13:46:20 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 13:46:20 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>	<52A1F793.3010506@gmail.com>	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
Message-ID: <52A21B7C.6000908@gmail.com>

On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
> unary versus binary minus

Oh right; I consider binary `-` broken for
Boolean arrays. (Sorry Alexander; I did not
see your entire issue.)


> I'd rather write ~ than unary - if that's what it is.

I agree.  So I have no objection to elimination
of the `-`.  I see it does the subtraction and then
a boolean conversion, which is not helpful.
Or rather, I do not see how it can be helpful.

Alan Isaac


From jtaylor.debian at googlemail.com  Fri Dec  6 13:58:59 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 06 Dec 2013 19:58:59 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
Message-ID: <52A21E73.5090200@googlemail.com>

On 06.12.2013 19:06, Ralf Gommers wrote:
> Hi all,
> 
> There are a few discussions on packaging for the scientific Python stack
> ongoing, on the NumFOCUS and distutils lists:
> https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
> https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY
> http://thread.gmane.org/gmane.comp.python.distutils.devel/20202
> http://thread.gmane.org/gmane.comp.python.distutils.devel/20296
> 
> One of the things that we should start doing for numpy is distribute
> releases as wheels. On OS X at least this is quite simple, so I propose
> to just experiment with it. I can create some to try out and put them on
> a separate folder on SourceForge. If that works they can be put on PyPi.
> 
> For Windows things are less simple, because the wheel format doesn't
> handle the multiple builds (no SSE, SSE2, SSE3) that are in the
> superpack installers. A problem is that we don't really know how many
> users still have old CPUs that don't support SSE3. The impact for those
> users is high, numpy will install but crash (see
> https://github.com/scipy/scipy/issues/1697). Questions:
> 1. does anyone have a good idea to obtain statistics?
> 2. in the absence of statistics, can we do an experiment by putting one
> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> propose, and seeing for how many (if any) users this goes wrong?

why SSE3 and not SSE2?
SSE2 is a requirement of the amd64 ABI, so it is present in all 64 bit
x86 cpus, so a even majority of windows machines running 32 bit will
have it.
SSE3 is not mandated by any ABI so it should more likely to find
machines without it.

to my knowledge SSE3 is not such big a difference to SSE2, only a little
better complex arithmetic and horizontal additions, I don't think its
worth it.
Are there performance comparisons for ATLAS with SSE2 and SSE3 available?


> 
> P.S. related question: did anyone check whether the recently merged
> NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
> 

according to https://github.com/numpy/numpy/issues/3760 SSE2 should be
off in the binaries created with mingw.

but there was also https://github.com/numpy/numpy/issues/3680, but that
might have been built with the VSC compiler (I think by Christoph Gohlke)
Assuming linux objdump works correctly on windows .pyd files there are
indeed sse2 instructions in the win32 build created with VSC.


From josef.pktd at gmail.com  Fri Dec  6 13:59:31 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 13:59:31 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A21B7C.6000908@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
Message-ID: <CAMMTP+BFcCs5ru3NjFJ4GT5KDhPpoCseO-uYK_k-pJKBDTLQ8w@mail.gmail.com>

On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>> unary versus binary minus
>
> Oh right; I consider binary `-` broken for
> Boolean arrays. (Sorry Alexander; I did not
> see your entire issue.)
>
>
>> I'd rather write ~ than unary - if that's what it is.
>
> I agree.  So I have no objection to elimination
> of the `-`.  I see it does the subtraction and then
> a boolean conversion, which is not helpful.
> Or rather, I do not see how it can be helpful.

What I would or might find useful is if binary `-` subtracts set
membership instead of doing xor

>>> m1 = np.array([0,0,1,1], bool)
>>> m2 = np.array([0,1,0,1], bool)
>>> m1 - m2
array([False,  True,  True, False], dtype=bool)
>>> np.logical_xor(m1, m2)
array([False,  True,  True, False], dtype=bool)

>>> np.clip(m1.astype(int) - m2.astype(int), 0, 1).astype(bool)
array([False, False,  True, False], dtype=bool)
>>> np.nonzero(_)[0]
array([2])

>>> s1 = set(np.arange(4)[m1])
>>> s2 = set(np.arange(4)[m2])
>>> s1 - s2
set([2])

Josef

>
> Alan Isaac
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From ndarray at mac.com  Fri Dec  6 14:55:41 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Fri, 6 Dec 2013 14:55:41 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <52A21B7C.6000908@gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
Message-ID: <CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>

On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
> > unary versus binary minus
>
> Oh right; I consider binary `-` broken for
> Boolean arrays. (Sorry Alexander; I did not
> see your entire issue.)
>
>
> > I'd rather write ~ than unary - if that's what it is.
>
> I agree.  So I have no objection to elimination
> of the `-`.


It looks like we are close to reaching a consensus on the following points:

1. * is well-defined on boolean arrays and may be used in preference of &
in code that is designed to handle 1s and 0s of any dtype in addition to
booleans.

2. + is defined consistently with * and the only issue is the absence of
additive inverse.  This is not a problem as long as presence of - does not
suggest otherwise.

3. binary and unary minus should be deprecated because its use in
expressions where variables can be either boolean or numeric would lead to
subtle bugs.  For example -x*y would produce different results from -(x*y)
depending on whether x is boolean or not.  In all situations, ^ is
preferable to binary - and ~ is preferable to unary -.

4. changing boolean arithmetics to auto-promotion to int is precluded by a
significant use-case of boolean matrices.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/2d440d6e/attachment.html>

From njs at pobox.com  Fri Dec  6 14:59:36 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 6 Dec 2013 11:59:36 -0800
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
Message-ID: <CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>

On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>
>
>
> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>
>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>> > unary versus binary minus
>>
>> Oh right; I consider binary `-` broken for
>> Boolean arrays. (Sorry Alexander; I did not
>> see your entire issue.)
>>
>>
>> > I'd rather write ~ than unary - if that's what it is.
>>
>> I agree.  So I have no objection to elimination
>> of the `-`.
>
>
> It looks like we are close to reaching a consensus on the following points:
>
> 1. * is well-defined on boolean arrays and may be used in preference of & in
> code that is designed to handle 1s and 0s of any dtype in addition to
> booleans.
>
> 2. + is defined consistently with * and the only issue is the absence of
> additive inverse.  This is not a problem as long as presence of - does not
> suggest otherwise.
>
> 3. binary and unary minus should be deprecated because its use in
> expressions where variables can be either boolean or numeric would lead to
> subtle bugs.  For example -x*y would produce different results from -(x*y)
> depending on whether x is boolean or not.  In all situations, ^ is
> preferable to binary - and ~ is preferable to unary -.
>
> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
> significant use-case of boolean matrices.

+1

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


From chris.barker at noaa.gov  Fri Dec  6 15:09:55 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 6 Dec 2013 12:09:55 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
Message-ID: <CALGmxEJYEbAzvTBJtsqyWjJLMCDDH2fuLC0-KQ8pYNEHgJxs1g@mail.gmail.com>

On Fri, Dec 6, 2013 at 10:06 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:


> One of the things that we should start doing for numpy is distribute
> releases as wheels. On OS X at least this is quite simple, so I propose to
> just experiment with it. I can create some to try out and put them on a
> separate folder on SourceForge. If that works they can be put on PyPi.
>
>
+1

For Windows things are less simple, because the wheel format doesn't handle
> the multiple builds (no SSE, SSE2, SSE3) that are in the superpack
> installers. A problem is that we don't really know how many users still
> have old CPUs that don't support SSE3. The impact for those users is high,
> numpy will install but crash (see
> https://github.com/scipy/scipy/issues/1697).
>

Could we have a run-time check, so at least folks would get a nice error
message?

2. in the absence of statistics, can we do an experiment by putting one
> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> propose, and seeing for how many (if any) users this goes wrong?
>

sounds good -- it looks like SSE3 has been around a good while:

http://en.wikipedia.org/wiki/SSE3

8+ years is a pretty long time in computer land!

anyone know how long SSE3 has been around?


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/92f41813/attachment.html>

From oscar.j.benjamin at gmail.com  Fri Dec  6 15:28:55 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 6 Dec 2013 20:28:55 +0000
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CALGmxEJYEbAzvTBJtsqyWjJLMCDDH2fuLC0-KQ8pYNEHgJxs1g@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxEJYEbAzvTBJtsqyWjJLMCDDH2fuLC0-KQ8pYNEHgJxs1g@mail.gmail.com>
Message-ID: <CAHVvXxSYsv9M3mz6V9aSawoaVi+jbpdD4CAxE821QOu29ydHJQ@mail.gmail.com>

On 6 December 2013 20:09, Chris Barker <chris.barker at noaa.gov> wrote:
>> 2. in the absence of statistics, can we do an experiment by putting one
>> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I propose,
>> and seeing for how many (if any) users this goes wrong?
>
>
> sounds good -- it looks like SSE3 has been around a good while:
>
> http://en.wikipedia.org/wiki/SSE3
>
> 8+ years is a pretty long time in computer land!
>
> anyone know how long SSE3 has been around?

I don't have statistics but I do have a couple of data points. Both of
the computers I regularly use (my work desktop and my girlfriend's
laptop) have SSE2 but not SSE3.

Really I'm not sure that releasing a potentially compatible binary -
with no install time checks - is such a good idea. What we really want
is a situation where you can confidently advise someone to just "pip
install numpy" without caveats i.e. a solution that "just works".


Oscar


From josef.pktd at gmail.com  Fri Dec  6 15:30:47 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 15:30:47 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
Message-ID: <CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>

On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>>
>>
>>
>> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>>
>>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>>> > unary versus binary minus
>>>
>>> Oh right; I consider binary `-` broken for
>>> Boolean arrays. (Sorry Alexander; I did not
>>> see your entire issue.)
>>>
>>>
>>> > I'd rather write ~ than unary - if that's what it is.
>>>
>>> I agree.  So I have no objection to elimination
>>> of the `-`.
>>
>>
>> It looks like we are close to reaching a consensus on the following points:
>>
>> 1. * is well-defined on boolean arrays and may be used in preference of & in
>> code that is designed to handle 1s and 0s of any dtype in addition to
>> booleans.
>>
>> 2. + is defined consistently with * and the only issue is the absence of
>> additive inverse.  This is not a problem as long as presence of - does not
>> suggest otherwise.
>>
>> 3. binary and unary minus should be deprecated because its use in
>> expressions where variables can be either boolean or numeric would lead to
>> subtle bugs.  For example -x*y would produce different results from -(x*y)
>> depending on whether x is boolean or not.  In all situations, ^ is
>> preferable to binary - and ~ is preferable to unary -.
>>
>> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
>> significant use-case of boolean matrices.
>
> +1

+0.5
(I would still prefer a different binary minus, but it would be
inconsistent with a logical unary minus that negates.)

5. `/` is useless
6 `**` follows from 1.

Josef


>
> --
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From cournape at gmail.com  Fri Dec  6 15:37:42 2013
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 6 Dec 2013 20:37:42 +0000
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CAHVvXxSYsv9M3mz6V9aSawoaVi+jbpdD4CAxE821QOu29ydHJQ@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxEJYEbAzvTBJtsqyWjJLMCDDH2fuLC0-KQ8pYNEHgJxs1g@mail.gmail.com>
	<CAHVvXxSYsv9M3mz6V9aSawoaVi+jbpdD4CAxE821QOu29ydHJQ@mail.gmail.com>
Message-ID: <CAGY4rcXLvGX4uRExpkpGAs_2v7bJLXpcOW_2jBgGqmbQCZjsSA@mail.gmail.com>

On Fri, Dec 6, 2013 at 8:28 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com>wrote:

> On 6 December 2013 20:09, Chris Barker <chris.barker at noaa.gov> wrote:
> >> 2. in the absence of statistics, can we do an experiment by putting one
> >> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> propose,
> >> and seeing for how many (if any) users this goes wrong?
> >
> >
> > sounds good -- it looks like SSE3 has been around a good while:
> >
> > http://en.wikipedia.org/wiki/SSE3
> >
> > 8+ years is a pretty long time in computer land!
> >
> > anyone know how long SSE3 has been around?
>
> I don't have statistics but I do have a couple of data points. Both of
> the computers I regularly use (my work desktop and my girlfriend's
> laptop) have SSE2 but not SSE3.
>
> Really I'm not sure that releasing a potentially compatible binary -
> with no install time checks - is such a good idea. What we really want
> is a situation where you can confidently advise someone to just "pip
> install numpy" without caveats i.e. a solution that "just works".
>

agreed.

Also, we should not lie to ourselves: our current ATLAS on windows are most
likely not very efficient anyway, SSE or not.

Ralf, you mentioned that openblas was problematic on windows ? I could not
find any recent discussion on that list.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/2afae39d/attachment.html>

From cgohlke at uci.edu  Fri Dec  6 15:38:32 2013
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 06 Dec 2013 12:38:32 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
Message-ID: <52A235C8.8030103@uci.edu>

On 12/6/2013 10:06 AM, Ralf Gommers wrote:
> Hi all,
>
> There are a few discussions on packaging for the scientific Python stack
> ongoing, on the NumFOCUS and distutils lists:
> https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
> <https://groups.google.com/forum/#%21topic/numfocus/mVNakFqfpZg>
> https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY
> <https://groups.google.com/forum/#%21topic/numfocus/HUcwXTM_jNY>
> http://thread.gmane.org/gmane.comp.python.distutils.devel/20202
> http://thread.gmane.org/gmane.comp.python.distutils.devel/20296
>
> One of the things that we should start doing for numpy is distribute
> releases as wheels. On OS X at least this is quite simple, so I propose
> to just experiment with it. I can create some to try out and put them on
> a separate folder on SourceForge. If that works they can be put on PyPi.
>
> For Windows things are less simple, because the wheel format doesn't
> handle the multiple builds (no SSE, SSE2, SSE3) that are in the
> superpack installers. A problem is that we don't really know how many
> users still have old CPUs that don't support SSE3. The impact for those
> users is high, numpy will install but crash (see
> https://github.com/scipy/scipy/issues/1697). Questions:
> 1. does anyone have a good idea to obtain statistics?
> 2. in the absence of statistics, can we do an experiment by putting one
> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> propose, and seeing for how many (if any) users this goes wrong?
>
> Ralf
>
> P.S. related question: did anyone check whether the recently merged
> NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
>
>

Has anyone succeeded building wheels for numpy, scipy, and matplotlib? 
On Windows `bdist_wheel` fails for me. It looks like numpy.distutils 
doesn't know about wheels and version 0.22.0 fails to package 
matplotlib. Pillow, pandas, scikit-image, scikits-learn work.

Christoph


Numpy 1.8.x
-----------
...
running build_scripts
creating build\scripts.win-amd64-3.3
Creating build\scripts.win-amd64-3.3\f2py.py
   adding 'build\scripts.win-amd64-3.3\f2py.py' to scripts
Running from numpy source directory.
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
    or: setup.py --help [cmd1 cmd2 ...]
    or: setup.py --help-commands
    or: setup.py cmd --help

error: invalid command 'bdist_wheel'


Scipy 0.13.x
------------
...
X:\Python27-x64\lib\distutils\dist.py:267: UserWarning: Unknown 
distribution option: 'test_suite'
   warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
    or: setup.py --help [cmd1 cmd2 ...]
    or: setup.py --help-commands
    or: setup.py cmd --help

error: invalid command 'bdist_wheel'


Matplotlib 1.3.x
----------------
...
installing to build\bdist.win32\wheel
running install
running install_lib
copying pylab.py -> build\bdist.win32\wheel\matplotlib-1.3.1.data\..
error: build\bdist.win32\wheel\matplotlib-1.3.1.data\..: Cannot create a 
file when that file already exists



From cournape at gmail.com  Fri Dec  6 15:40:59 2013
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 6 Dec 2013 20:40:59 +0000
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <52A235C8.8030103@uci.edu>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<52A235C8.8030103@uci.edu>
Message-ID: <CAGY4rcXCaKYfO7jJdjgCpe54HvBC=216KAbxhUy265Yd3UjrNQ@mail.gmail.com>

On Fri, Dec 6, 2013 at 8:38 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

> On 12/6/2013 10:06 AM, Ralf Gommers wrote:
> > Hi all,
> >
> > There are a few discussions on packaging for the scientific Python stack
> > ongoing, on the NumFOCUS and distutils lists:
> > https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
> > <https://groups.google.com/forum/#%21topic/numfocus/mVNakFqfpZg>
> > https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY
> > <https://groups.google.com/forum/#%21topic/numfocus/HUcwXTM_jNY>
> > http://thread.gmane.org/gmane.comp.python.distutils.devel/20202
> > http://thread.gmane.org/gmane.comp.python.distutils.devel/20296
> >
> > One of the things that we should start doing for numpy is distribute
> > releases as wheels. On OS X at least this is quite simple, so I propose
> > to just experiment with it. I can create some to try out and put them on
> > a separate folder on SourceForge. If that works they can be put on PyPi.
> >
> > For Windows things are less simple, because the wheel format doesn't
> > handle the multiple builds (no SSE, SSE2, SSE3) that are in the
> > superpack installers. A problem is that we don't really know how many
> > users still have old CPUs that don't support SSE3. The impact for those
> > users is high, numpy will install but crash (see
> > https://github.com/scipy/scipy/issues/1697). Questions:
> > 1. does anyone have a good idea to obtain statistics?
> > 2. in the absence of statistics, can we do an experiment by putting one
> > wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> > propose, and seeing for how many (if any) users this goes wrong?
> >
> > Ralf
> >
> > P.S. related question: did anyone check whether the recently merged
> > NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
> >
> >
>
> Has anyone succeeded building wheels for numpy, scipy, and matplotlib?
>

I did for numpy and scipy. You had to hack a bit numpy.distutils to make it
work for scipy,but nothing that would be too complicated to really fix.

In your case, the trick is to use the setupegg file: python setupegg.py
bdist_wheel

David

> On Windows `bdist_wheel` fails for me. It looks like numpy.distutils
> doesn't know about wheels and version 0.22.0 fails to package
> matplotlib. Pillow, pandas, scikit-image, scikits-learn work.
>
> Christoph
>
>
> Numpy 1.8.x
> -----------
> ...
> running build_scripts
> creating build\scripts.win-amd64-3.3
> Creating build\scripts.win-amd64-3.3\f2py.py
>    adding 'build\scripts.win-amd64-3.3\f2py.py' to scripts
> Running from numpy source directory.
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>     or: setup.py --help [cmd1 cmd2 ...]
>     or: setup.py --help-commands
>     or: setup.py cmd --help
>
> error: invalid command 'bdist_wheel'
>
>
> Scipy 0.13.x
> ------------
> ...
> X:\Python27-x64\lib\distutils\dist.py:267: UserWarning: Unknown
> distribution option: 'test_suite'
>    warnings.warn(msg)
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>     or: setup.py --help [cmd1 cmd2 ...]
>     or: setup.py --help-commands
>     or: setup.py cmd --help
>
> error: invalid command 'bdist_wheel'
>
>
> Matplotlib 1.3.x
> ----------------
> ...
> installing to build\bdist.win32\wheel
> running install
> running install_lib
> copying pylab.py -> build\bdist.win32\wheel\matplotlib-1.3.1.data\..
> error: build\bdist.win32\wheel\matplotlib-1.3.1.data\..: Cannot create a
> file when that file already exists
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/f6df0a28/attachment.html>

From sebastian at sipsolutions.net  Fri Dec  6 15:50:42 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 06 Dec 2013 21:50:42 +0100
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
Message-ID: <1386363042.29623.10.camel@sebastian-laptop>

On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
> >>
> >>
> >>
> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> >>>
> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
> >>> > unary versus binary minus
> >>>
> >>> Oh right; I consider binary `-` broken for
> >>> Boolean arrays. (Sorry Alexander; I did not
> >>> see your entire issue.)
> >>>
> >>>
> >>> > I'd rather write ~ than unary - if that's what it is.
> >>>
> >>> I agree.  So I have no objection to elimination
> >>> of the `-`.
> >>
> >>
> >> It looks like we are close to reaching a consensus on the following points:
> >>
> >> 1. * is well-defined on boolean arrays and may be used in preference of & in
> >> code that is designed to handle 1s and 0s of any dtype in addition to
> >> booleans.
> >>
> >> 2. + is defined consistently with * and the only issue is the absence of
> >> additive inverse.  This is not a problem as long as presence of - does not
> >> suggest otherwise.
> >>
> >> 3. binary and unary minus should be deprecated because its use in
> >> expressions where variables can be either boolean or numeric would lead to
> >> subtle bugs.  For example -x*y would produce different results from -(x*y)
> >> depending on whether x is boolean or not.  In all situations, ^ is
> >> preferable to binary - and ~ is preferable to unary -.
> >>
> >> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
> >> significant use-case of boolean matrices.
> >
> > +1
> 
> +0.5
> (I would still prefer a different binary minus, but it would be
> inconsistent with a logical unary minus that negates.)
> 

The question is if the current xor behaviour can make sense? It doesn't
seem to make much sense mathematically? Which only leaves that `abs(x -
y)` is actually what a (python) programmer might expect.
I think I would like to deprecate at least the unary one. The ~ kind of
behaviour just doesn't fit as far as I can see. 

> 5. `/` is useless
> 6 `**` follows from 1.

Both of these are currently not defined, they will just cause upcast to
int8. I suppose it would be possible to deprecate that upcast though
(same goes for most all other ufuncs/operators in principle).

> 
> Josef
> 
> 
> >
> > --
> > Nathaniel J. Smith
> > Postdoctoral researcher - Informatics - University of Edinburgh
> > http://vorpus.org
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From cgohlke at uci.edu  Fri Dec  6 16:01:32 2013
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 06 Dec 2013 13:01:32 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CAGY4rcXCaKYfO7jJdjgCpe54HvBC=216KAbxhUy265Yd3UjrNQ@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>	<52A235C8.8030103@uci.edu>
	<CAGY4rcXCaKYfO7jJdjgCpe54HvBC=216KAbxhUy265Yd3UjrNQ@mail.gmail.com>
Message-ID: <52A23B2C.9010807@uci.edu>

On 12/6/2013 12:40 PM, David Cournapeau wrote:
>
>
>
> On Fri, Dec 6, 2013 at 8:38 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>     On 12/6/2013 10:06 AM, Ralf Gommers wrote:
>     > Hi all,
>     >
>     > There are a few discussions on packaging for the scientific Python stack
>     > ongoing, on the NumFOCUS and distutils lists:
>     >https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
>     <https://groups.google.com/forum/#%21topic/numfocus/mVNakFqfpZg>
>     > <https://groups.google.com/forum/#%21topic/numfocus/mVNakFqfpZg>
>     >https://groups.google.com/forum/#!topic/numfocus/HUcwXTM_jNY
>     <https://groups.google.com/forum/#%21topic/numfocus/HUcwXTM_jNY>
>     > <https://groups.google.com/forum/#%21topic/numfocus/HUcwXTM_jNY>
>     >http://thread.gmane.org/gmane.comp.python.distutils.devel/20202
>     >http://thread.gmane.org/gmane.comp.python.distutils.devel/20296
>     >
>     > One of the things that we should start doing for numpy is distribute
>     > releases as wheels. On OS X at least this is quite simple, so I propose
>     > to just experiment with it. I can create some to try out and put them on
>     > a separate folder on SourceForge. If that works they can be put on PyPi.
>     >
>     > For Windows things are less simple, because the wheel format doesn't
>     > handle the multiple builds (no SSE, SSE2, SSE3) that are in the
>     > superpack installers. A problem is that we don't really know how many
>     > users still have old CPUs that don't support SSE3. The impact for those
>     > users is high, numpy will install but crash (see
>     >https://github.com/scipy/scipy/issues/1697). Questions:
>     > 1. does anyone have a good idea to obtain statistics?
>     > 2. in the absence of statistics, can we do an experiment by putting one
>     > wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
>     > propose, and seeing for how many (if any) users this goes wrong?
>     >
>     > Ralf
>     >
>     > P.S. related question: did anyone check whether the recently merged
>     > NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
>     >
>     >
>
>     Has anyone succeeded building wheels for numpy, scipy, and matplotlib?
>
>
> I did for numpy and scipy. You had to hack a bit numpy.distutils to make
> it work for scipy,but nothing that would be too complicated to really fix.
>
> In your case, the trick is to use the setupegg file: python setupegg.py
> bdist_wheel
>
> David
>

Thank you. The setupegg.py trick worked. Could the numpy.distutils hack 
be applied to the numpy 1.8.x and master branches? I'll try to fix the 
matplotlib issue.

Christoph


From josef.pktd at gmail.com  Fri Dec  6 16:14:13 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 16:14:13 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386363042.29623.10.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
Message-ID: <CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>

On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
>> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>> >>
>> >>
>> >>
>> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>> >>>
>> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>> >>> > unary versus binary minus
>> >>>
>> >>> Oh right; I consider binary `-` broken for
>> >>> Boolean arrays. (Sorry Alexander; I did not
>> >>> see your entire issue.)
>> >>>
>> >>>
>> >>> > I'd rather write ~ than unary - if that's what it is.
>> >>>
>> >>> I agree.  So I have no objection to elimination
>> >>> of the `-`.
>> >>
>> >>
>> >> It looks like we are close to reaching a consensus on the following points:
>> >>
>> >> 1. * is well-defined on boolean arrays and may be used in preference of & in
>> >> code that is designed to handle 1s and 0s of any dtype in addition to
>> >> booleans.
>> >>
>> >> 2. + is defined consistently with * and the only issue is the absence of
>> >> additive inverse.  This is not a problem as long as presence of - does not
>> >> suggest otherwise.
>> >>
>> >> 3. binary and unary minus should be deprecated because its use in
>> >> expressions where variables can be either boolean or numeric would lead to
>> >> subtle bugs.  For example -x*y would produce different results from -(x*y)
>> >> depending on whether x is boolean or not.  In all situations, ^ is
>> >> preferable to binary - and ~ is preferable to unary -.
>> >>
>> >> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
>> >> significant use-case of boolean matrices.
>> >
>> > +1
>>
>> +0.5
>> (I would still prefer a different binary minus, but it would be
>> inconsistent with a logical unary minus that negates.)
>>
>
> The question is if the current xor behaviour can make sense? It doesn't
> seem to make much sense mathematically? Which only leaves that `abs(x -
> y)` is actually what a (python) programmer might expect.
> I think I would like to deprecate at least the unary one. The ~ kind of
> behaviour just doesn't fit as far as I can see.

I haven't seen any real use cases for xor yet.
My impression is that both plus and minus are just overflow accidents
and not intentional. plus works in a useful way, minus as xor might be
used once per century.

I would deprecate both unary and binary minus.

(And when nobody is looking in two versions from now, I would add a
binary minus that overflows to the clipped version, so I get a set
subtraction. :)

>
>> 5. `/` is useless
>> 6 `**` follows from 1.

>>> m1 ** m2
array([1, 0, 1, 1], dtype=int8)
>>> m1 ** 2
array([False, False,  True,  True], dtype=bool)
>>> m1 ** 3
array([0, 0, 1, 1])

but I'm using python with an old numpy right now
>>> np.__version__
'1.6.1'

>
> Both of these are currently not defined, they will just cause upcast to
> int8. I suppose it would be possible to deprecate that upcast though
> (same goes for most all other ufuncs/operators in principle).

We would have to start the discussion again for all other
operators/ufuncs to see if they are useful in some cases.
For most treating as int will make sense, I guess.

Josef

>
>>
>> Josef
>>
>>
>> >
>> > --
>> > Nathaniel J. Smith
>> > Postdoctoral researcher - Informatics - University of Edinburgh
>> > http://vorpus.org
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From ralf.gommers at gmail.com  Fri Dec  6 16:26:15 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 6 Dec 2013 22:26:15 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <52A23B2C.9010807@uci.edu>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<52A235C8.8030103@uci.edu>
	<CAGY4rcXCaKYfO7jJdjgCpe54HvBC=216KAbxhUy265Yd3UjrNQ@mail.gmail.com>
	<52A23B2C.9010807@uci.edu>
Message-ID: <CABL7CQiJL22b6Fk=7cEu2o7UJ7Soewv6MV2rAMAhPYOn1yxjHA@mail.gmail.com>

On Fri, Dec 6, 2013 at 10:01 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

> On 12/6/2013 12:40 PM, David Cournapeau wrote:
> >
> >
> > On Fri, Dec 6, 2013 at 8:38 PM, Christoph Gohlke <cgohlke at uci.edu
> > <mailto:cgohlke at uci.edu>> wrote:
> >
> >     Has anyone succeeded building wheels for numpy, scipy, and
> matplotlib?
> >
> > I did for numpy and scipy. You had to hack a bit numpy.distutils to make
> > it work for scipy,but nothing that would be too complicated to really
> fix.
> >
> > In your case, the trick is to use the setupegg file: python setupegg.py
> > bdist_wheel
> >
> > David
> >
>
> Thank you. The setupegg.py trick worked. Could the numpy.distutils hack
> be applied to the numpy 1.8.x and master branches? I'll try to fix the
> matplotlib issue.
>

This should make ``python setup.py bdist_wheel`` work:
https://github.com/numpy/numpy/pull/4110

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/ddbc5995/attachment.html>

From alan.isaac at gmail.com  Fri Dec  6 17:09:59 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 17:09:59 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>	<52A1F793.3010506@gmail.com>	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>	<52A21468.8010809@gmail.com>	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>	<52A21B7C.6000908@gmail.com>	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
Message-ID: <52A24B37.7090200@gmail.com>

On 12/6/2013 3:30 PM, josef.pktd at gmail.com wrote:
> 6 `**` follows from 1.


Yes, but what really matters is that
linalg.matrix_power
give the correct (boolean) result.

Alan



From alan.isaac at gmail.com  Fri Dec  6 17:14:23 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 06 Dec 2013 17:14:23 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <1386363042.29623.10.camel@sebastian-laptop>
References: <1386283039.23728.6.camel@sebastian-laptop>	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>	<52A14D0C.1060209@gmail.com>	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>	<52A1F793.3010506@gmail.com>	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>	<52A21468.8010809@gmail.com>	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>	<52A21B7C.6000908@gmail.com>	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
Message-ID: <52A24C3F.4060308@gmail.com>

On 12/6/2013 3:50 PM, Sebastian Berg wrote:
> Both of these are currently not defined, they will just cause upcast to
> int8.


What does currently mean?
`**` works fine for boolean arrays in 1.7.1.
(It's useless, but it works.)

Alan Isaac



From ralf.gommers at gmail.com  Fri Dec  6 17:16:37 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 6 Dec 2013 23:16:37 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CAGY4rcXLvGX4uRExpkpGAs_2v7bJLXpcOW_2jBgGqmbQCZjsSA@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxEJYEbAzvTBJtsqyWjJLMCDDH2fuLC0-KQ8pYNEHgJxs1g@mail.gmail.com>
	<CAHVvXxSYsv9M3mz6V9aSawoaVi+jbpdD4CAxE821QOu29ydHJQ@mail.gmail.com>
	<CAGY4rcXLvGX4uRExpkpGAs_2v7bJLXpcOW_2jBgGqmbQCZjsSA@mail.gmail.com>
Message-ID: <CABL7CQhYauGTjBkgnEz47+Mtjupsn8L5WEC+f7=L=cedqkPbbg@mail.gmail.com>

On Fri, Dec 6, 2013 at 9:37 PM, David Cournapeau <cournape at gmail.com> wrote:

>
>
>
> On Fri, Dec 6, 2013 at 8:28 PM, Oscar Benjamin <oscar.j.benjamin at gmail.com
> > wrote:
>
>> On 6 December 2013 20:09, Chris Barker <chris.barker at noaa.gov> wrote:
>> >> 2. in the absence of statistics, can we do an experiment by putting one
>> >> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
>> propose,
>> >> and seeing for how many (if any) users this goes wrong?
>> >
>> >
>> > sounds good -- it looks like SSE3 has been around a good while:
>> >
>> > http://en.wikipedia.org/wiki/SSE3
>> >
>> > 8+ years is a pretty long time in computer land!
>> >
>> > anyone know how long SSE3 has been around?
>>
>> I don't have statistics but I do have a couple of data points. Both of
>> the computers I regularly use (my work desktop and my girlfriend's
>> laptop) have SSE2 but not SSE3.
>>
>> Really I'm not sure that releasing a potentially compatible binary -
>> with no install time checks - is such a good idea. What we really want
>> is a situation where you can confidently advise someone to just "pip
>> install numpy" without caveats i.e. a solution that "just works".
>>
>
> agreed.
>
> Also, we should not lie to ourselves: our current ATLAS on windows are
> most likely not very efficient anyway, SSE or not.
>
> Ralf, you mentioned that openblas was problematic on windows ? I could not
> find any recent discussion on that list.
>

I didn't mean specifically on Windows. I based that on comments like:
https://github.com/numpy/numpy/issues/4007#issuecomment-27688947
http://article.gmane.org/gmane.comp.python.scientific.devel/18098
https://github.com/numpy/numpy/issues/3545

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/011946ed/attachment.html>

From josef.pktd at gmail.com  Fri Dec  6 17:29:34 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 17:29:34 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
	<CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
Message-ID: <CAMMTP+CyaC_1Cf3qRtxaswSiavxor+gttgKRvezePM_FKSVpvw@mail.gmail.com>

On Fri, Dec 6, 2013 at 4:14 PM,  <josef.pktd at gmail.com> wrote:
> On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
>>> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>> >>>
>>> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>>> >>> > unary versus binary minus
>>> >>>
>>> >>> Oh right; I consider binary `-` broken for
>>> >>> Boolean arrays. (Sorry Alexander; I did not
>>> >>> see your entire issue.)
>>> >>>
>>> >>>
>>> >>> > I'd rather write ~ than unary - if that's what it is.
>>> >>>
>>> >>> I agree.  So I have no objection to elimination
>>> >>> of the `-`.
>>> >>
>>> >>
>>> >> It looks like we are close to reaching a consensus on the following points:
>>> >>
>>> >> 1. * is well-defined on boolean arrays and may be used in preference of & in
>>> >> code that is designed to handle 1s and 0s of any dtype in addition to
>>> >> booleans.
>>> >>
>>> >> 2. + is defined consistently with * and the only issue is the absence of
>>> >> additive inverse.  This is not a problem as long as presence of - does not
>>> >> suggest otherwise.
>>> >>
>>> >> 3. binary and unary minus should be deprecated because its use in
>>> >> expressions where variables can be either boolean or numeric would lead to
>>> >> subtle bugs.  For example -x*y would produce different results from -(x*y)
>>> >> depending on whether x is boolean or not.  In all situations, ^ is
>>> >> preferable to binary - and ~ is preferable to unary -.
>>> >>
>>> >> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
>>> >> significant use-case of boolean matrices.
>>> >
>>> > +1
>>>
>>> +0.5
>>> (I would still prefer a different binary minus, but it would be
>>> inconsistent with a logical unary minus that negates.)
>>>
>>
>> The question is if the current xor behaviour can make sense? It doesn't
>> seem to make much sense mathematically? Which only leaves that `abs(x -
>> y)` is actually what a (python) programmer might expect.
>> I think I would like to deprecate at least the unary one. The ~ kind of
>> behaviour just doesn't fit as far as I can see.
>
> I haven't seen any real use cases for xor yet.
> My impression is that both plus and minus are just overflow accidents
> and not intentional. plus works in a useful way, minus as xor might be
> used once per century.
>
> I would deprecate both unary and binary minus.
>
> (And when nobody is looking in two versions from now, I would add a
> binary minus that overflows to the clipped version, so I get a set
> subtraction. :)

Actually minus works as expected if we avoid negative overflow:

>>> m1 - m1*m2
array([False, False,  True, False], dtype=bool)
>>> m1 * ~m2
array([False, False,  True, False], dtype=bool)
>>> m1 & ~m2
array([False, False,  True, False], dtype=bool)

I find the first easy to read, but m1 - m2 would be one operation
less, and chain more easily m1 - m2 - m3
m1 are mailing list subscribers, take away
m2 owners of apples, take away
m3 users of Linux
= exotic developers

Josef





>
>>
>>> 5. `/` is useless
>>> 6 `**` follows from 1.
>
>>>> m1 ** m2
> array([1, 0, 1, 1], dtype=int8)
>>>> m1 ** 2
> array([False, False,  True,  True], dtype=bool)
>>>> m1 ** 3
> array([0, 0, 1, 1])
>
> but I'm using python with an old numpy right now
>>>> np.__version__
> '1.6.1'
>
>>
>> Both of these are currently not defined, they will just cause upcast to
>> int8. I suppose it would be possible to deprecate that upcast though
>> (same goes for most all other ufuncs/operators in principle).
>
> We would have to start the discussion again for all other
> operators/ufuncs to see if they are useful in some cases.
> For most treating as int will make sense, I guess.
>
> Josef
>
>>
>>>
>>> Josef
>>>
>>>
>>> >
>>> > --
>>> > Nathaniel J. Smith
>>> > Postdoctoral researcher - Informatics - University of Edinburgh
>>> > http://vorpus.org
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From njs at pobox.com  Fri Dec  6 17:45:28 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 6 Dec 2013 14:45:28 -0800
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+CyaC_1Cf3qRtxaswSiavxor+gttgKRvezePM_FKSVpvw@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
	<CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
	<CAMMTP+CyaC_1Cf3qRtxaswSiavxor+gttgKRvezePM_FKSVpvw@mail.gmail.com>
Message-ID: <CAPJVwB=77LGqGLmu-6KMdiNQyhXmOy2Nue+O5csgy+S52PLX6Q@mail.gmail.com>

Not sure how much time it's worth spending on coming up with new
definitions for boolean subtraction, since even if we deprecate the
current behavior now we won't be able to implement any of them for a
year+, and then we'll end up having to go through these debates again
then anyway.

-n

On Fri, Dec 6, 2013 at 2:29 PM,  <josef.pktd at gmail.com> wrote:
> On Fri, Dec 6, 2013 at 4:14 PM,  <josef.pktd at gmail.com> wrote:
>> On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
>> <sebastian at sipsolutions.net> wrote:
>>> On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
>>>> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>>> >>>
>>>> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>>>> >>> > unary versus binary minus
>>>> >>>
>>>> >>> Oh right; I consider binary `-` broken for
>>>> >>> Boolean arrays. (Sorry Alexander; I did not
>>>> >>> see your entire issue.)
>>>> >>>
>>>> >>>
>>>> >>> > I'd rather write ~ than unary - if that's what it is.
>>>> >>>
>>>> >>> I agree.  So I have no objection to elimination
>>>> >>> of the `-`.
>>>> >>
>>>> >>
>>>> >> It looks like we are close to reaching a consensus on the following points:
>>>> >>
>>>> >> 1. * is well-defined on boolean arrays and may be used in preference of & in
>>>> >> code that is designed to handle 1s and 0s of any dtype in addition to
>>>> >> booleans.
>>>> >>
>>>> >> 2. + is defined consistently with * and the only issue is the absence of
>>>> >> additive inverse.  This is not a problem as long as presence of - does not
>>>> >> suggest otherwise.
>>>> >>
>>>> >> 3. binary and unary minus should be deprecated because its use in
>>>> >> expressions where variables can be either boolean or numeric would lead to
>>>> >> subtle bugs.  For example -x*y would produce different results from -(x*y)
>>>> >> depending on whether x is boolean or not.  In all situations, ^ is
>>>> >> preferable to binary - and ~ is preferable to unary -.
>>>> >>
>>>> >> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
>>>> >> significant use-case of boolean matrices.
>>>> >
>>>> > +1
>>>>
>>>> +0.5
>>>> (I would still prefer a different binary minus, but it would be
>>>> inconsistent with a logical unary minus that negates.)
>>>>
>>>
>>> The question is if the current xor behaviour can make sense? It doesn't
>>> seem to make much sense mathematically? Which only leaves that `abs(x -
>>> y)` is actually what a (python) programmer might expect.
>>> I think I would like to deprecate at least the unary one. The ~ kind of
>>> behaviour just doesn't fit as far as I can see.
>>
>> I haven't seen any real use cases for xor yet.
>> My impression is that both plus and minus are just overflow accidents
>> and not intentional. plus works in a useful way, minus as xor might be
>> used once per century.
>>
>> I would deprecate both unary and binary minus.
>>
>> (And when nobody is looking in two versions from now, I would add a
>> binary minus that overflows to the clipped version, so I get a set
>> subtraction. :)
>
> Actually minus works as expected if we avoid negative overflow:
>
>>>> m1 - m1*m2
> array([False, False,  True, False], dtype=bool)
>>>> m1 * ~m2
> array([False, False,  True, False], dtype=bool)
>>>> m1 & ~m2
> array([False, False,  True, False], dtype=bool)
>
> I find the first easy to read, but m1 - m2 would be one operation
> less, and chain more easily m1 - m2 - m3
> m1 are mailing list subscribers, take away
> m2 owners of apples, take away
> m3 users of Linux
> = exotic developers
>
> Josef
>
>
>
>
>
>>
>>>
>>>> 5. `/` is useless
>>>> 6 `**` follows from 1.
>>
>>>>> m1 ** m2
>> array([1, 0, 1, 1], dtype=int8)
>>>>> m1 ** 2
>> array([False, False,  True,  True], dtype=bool)
>>>>> m1 ** 3
>> array([0, 0, 1, 1])
>>
>> but I'm using python with an old numpy right now
>>>>> np.__version__
>> '1.6.1'
>>
>>>
>>> Both of these are currently not defined, they will just cause upcast to
>>> int8. I suppose it would be possible to deprecate that upcast though
>>> (same goes for most all other ufuncs/operators in principle).
>>
>> We would have to start the discussion again for all other
>> operators/ufuncs to see if they are useful in some cases.
>> For most treating as int will make sense, I guess.
>>
>> Josef
>>
>>>
>>>>
>>>> Josef
>>>>
>>>>
>>>> >
>>>> > --
>>>> > Nathaniel J. Smith
>>>> > Postdoctoral researcher - Informatics - University of Edinburgh
>>>> > http://vorpus.org
>>>> > _______________________________________________
>>>> > NumPy-Discussion mailing list
>>>> > NumPy-Discussion at scipy.org
>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


From josef.pktd at gmail.com  Fri Dec  6 23:19:21 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 6 Dec 2013 23:19:21 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAPJVwB=77LGqGLmu-6KMdiNQyhXmOy2Nue+O5csgy+S52PLX6Q@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
	<CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
	<CAMMTP+CyaC_1Cf3qRtxaswSiavxor+gttgKRvezePM_FKSVpvw@mail.gmail.com>
	<CAPJVwB=77LGqGLmu-6KMdiNQyhXmOy2Nue+O5csgy+S52PLX6Q@mail.gmail.com>
Message-ID: <CAMMTP+By0NOfT_5XTjq0w7_6Wk2cF=vMeRx8f7-03akojrWmKQ@mail.gmail.com>

On Fri, Dec 6, 2013 at 5:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
> Not sure how much time it's worth spending on coming up with new
> definitions for boolean subtraction, since even if we deprecate the
> current behavior now we won't be able to implement any of them for a
> year+, and then we'll end up having to go through these debates again
> then anyway.

I didn't argue against deprecation of the boolean minuses. I'm fine with that.

Just some early lobbying, and so I can save my examples where I can
google them in case I'm still around if or when we can revisit the
issue.
Once I turn of the python interpreter that I used for the examples, I
will forget everything about weird boolean operations.

One advantage of this thread is that I had to look up the math for
indicator functions, and that I have a better idea where I could use
logical operators instead of (linear) algebra.

Josef

>
> -n
>
> On Fri, Dec 6, 2013 at 2:29 PM,  <josef.pktd at gmail.com> wrote:
>> On Fri, Dec 6, 2013 at 4:14 PM,  <josef.pktd at gmail.com> wrote:
>>> On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
>>> <sebastian at sipsolutions.net> wrote:
>>>> On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
>>>>> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>>> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <ndarray at mac.com> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>>>> >>>
>>>>> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>>>>> >>> > unary versus binary minus
>>>>> >>>
>>>>> >>> Oh right; I consider binary `-` broken for
>>>>> >>> Boolean arrays. (Sorry Alexander; I did not
>>>>> >>> see your entire issue.)
>>>>> >>>
>>>>> >>>
>>>>> >>> > I'd rather write ~ than unary - if that's what it is.
>>>>> >>>
>>>>> >>> I agree.  So I have no objection to elimination
>>>>> >>> of the `-`.
>>>>> >>
>>>>> >>
>>>>> >> It looks like we are close to reaching a consensus on the following points:
>>>>> >>
>>>>> >> 1. * is well-defined on boolean arrays and may be used in preference of & in
>>>>> >> code that is designed to handle 1s and 0s of any dtype in addition to
>>>>> >> booleans.
>>>>> >>
>>>>> >> 2. + is defined consistently with * and the only issue is the absence of
>>>>> >> additive inverse.  This is not a problem as long as presence of - does not
>>>>> >> suggest otherwise.
>>>>> >>
>>>>> >> 3. binary and unary minus should be deprecated because its use in
>>>>> >> expressions where variables can be either boolean or numeric would lead to
>>>>> >> subtle bugs.  For example -x*y would produce different results from -(x*y)
>>>>> >> depending on whether x is boolean or not.  In all situations, ^ is
>>>>> >> preferable to binary - and ~ is preferable to unary -.
>>>>> >>
>>>>> >> 4. changing boolean arithmetics to auto-promotion to int is precluded by a
>>>>> >> significant use-case of boolean matrices.
>>>>> >
>>>>> > +1
>>>>>
>>>>> +0.5
>>>>> (I would still prefer a different binary minus, but it would be
>>>>> inconsistent with a logical unary minus that negates.)
>>>>>
>>>>
>>>> The question is if the current xor behaviour can make sense? It doesn't
>>>> seem to make much sense mathematically? Which only leaves that `abs(x -
>>>> y)` is actually what a (python) programmer might expect.
>>>> I think I would like to deprecate at least the unary one. The ~ kind of
>>>> behaviour just doesn't fit as far as I can see.
>>>
>>> I haven't seen any real use cases for xor yet.
>>> My impression is that both plus and minus are just overflow accidents
>>> and not intentional. plus works in a useful way, minus as xor might be
>>> used once per century.
>>>
>>> I would deprecate both unary and binary minus.
>>>
>>> (And when nobody is looking in two versions from now, I would add a
>>> binary minus that overflows to the clipped version, so I get a set
>>> subtraction. :)
>>
>> Actually minus works as expected if we avoid negative overflow:
>>
>>>>> m1 - m1*m2
>> array([False, False,  True, False], dtype=bool)
>>>>> m1 * ~m2
>> array([False, False,  True, False], dtype=bool)
>>>>> m1 & ~m2
>> array([False, False,  True, False], dtype=bool)
>>
>> I find the first easy to read, but m1 - m2 would be one operation
>> less, and chain more easily m1 - m2 - m3
>> m1 are mailing list subscribers, take away
>> m2 owners of apples, take away
>> m3 users of Linux
>> = exotic developers
>>
>> Josef
>>
>>
>>
>>
>>
>>>
>>>>
>>>>> 5. `/` is useless
>>>>> 6 `**` follows from 1.
>>>
>>>>>> m1 ** m2
>>> array([1, 0, 1, 1], dtype=int8)
>>>>>> m1 ** 2
>>> array([False, False,  True,  True], dtype=bool)
>>>>>> m1 ** 3
>>> array([0, 0, 1, 1])
>>>
>>> but I'm using python with an old numpy right now
>>>>>> np.__version__
>>> '1.6.1'
>>>
>>>>
>>>> Both of these are currently not defined, they will just cause upcast to
>>>> int8. I suppose it would be possible to deprecate that upcast though
>>>> (same goes for most all other ufuncs/operators in principle).
>>>
>>> We would have to start the discussion again for all other
>>> operators/ufuncs to see if they are useful in some cases.
>>> For most treating as int will make sense, I guess.
>>>
>>> Josef
>>>
>>>>
>>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>> >
>>>>> > --
>>>>> > Nathaniel J. Smith
>>>>> > Postdoctoral researcher - Informatics - University of Edinburgh
>>>>> > http://vorpus.org
>>>>> > _______________________________________________
>>>>> > NumPy-Discussion mailing list
>>>>> > NumPy-Discussion at scipy.org
>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From charlesr.harris at gmail.com  Fri Dec  6 23:25:00 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 6 Dec 2013 21:25:00 -0700
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
	<CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
Message-ID: <CAB6mnx+hbNFo+A6ZH_TeT3p8UPRRWP==z6PrMWKbMi4R+99f7A@mail.gmail.com>

On Fri, Dec 6, 2013 at 2:14 PM, <josef.pktd at gmail.com> wrote:

> On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
> > On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
> >> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky <
> ndarray at mac.com> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com>
> wrote:
> >> >>>
> >> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
> >> >>> > unary versus binary minus
> >> >>>
> >> >>> Oh right; I consider binary `-` broken for
> >> >>> Boolean arrays. (Sorry Alexander; I did not
> >> >>> see your entire issue.)
> >> >>>
> >> >>>
> >> >>> > I'd rather write ~ than unary - if that's what it is.
> >> >>>
> >> >>> I agree.  So I have no objection to elimination
> >> >>> of the `-`.
> >> >>
> >> >>
> >> >> It looks like we are close to reaching a consensus on the following
> points:
> >> >>
> >> >> 1. * is well-defined on boolean arrays and may be used in preference
> of & in
> >> >> code that is designed to handle 1s and 0s of any dtype in addition to
> >> >> booleans.
> >> >>
> >> >> 2. + is defined consistently with * and the only issue is the
> absence of
> >> >> additive inverse.  This is not a problem as long as presence of -
> does not
> >> >> suggest otherwise.
> >> >>
> >> >> 3. binary and unary minus should be deprecated because its use in
> >> >> expressions where variables can be either boolean or numeric would
> lead to
> >> >> subtle bugs.  For example -x*y would produce different results from
> -(x*y)
> >> >> depending on whether x is boolean or not.  In all situations, ^ is
> >> >> preferable to binary - and ~ is preferable to unary -.
> >> >>
> >> >> 4. changing boolean arithmetics to auto-promotion to int is
> precluded by a
> >> >> significant use-case of boolean matrices.
> >> >
> >> > +1
> >>
> >> +0.5
> >> (I would still prefer a different binary minus, but it would be
> >> inconsistent with a logical unary minus that negates.)
> >>
> >
> > The question is if the current xor behaviour can make sense? It doesn't
> > seem to make much sense mathematically? Which only leaves that `abs(x -
> > y)` is actually what a (python) programmer might expect.
> > I think I would like to deprecate at least the unary one. The ~ kind of
> > behaviour just doesn't fit as far as I can see.
>
> I haven't seen any real use cases for xor yet.
>

Using it instead of '+' yields a boolean ring instead of semi-ring. Papers
from the first quarter of the last century used it pretty often on that
account, hence 'sigma-rings', etc. Eventually the simplicity of the
inclusive or overcame that tendency.

My impression is that both plus and minus are just overflow accidents
> and not intentional. plus works in a useful way, minus as xor might be
> used once per century.
>

It's certainly weird given that '+' means the inclusive or. I think '^' is
much preferable.
Although it makes some sense if one can keep the semantics straight.
Complicated, though.


> I would deprecate both unary and binary minus.
>
> (And when nobody is looking in two versions from now, I would add a
> binary minus that overflows to the clipped version, so I get a set
> subtraction. :)
>

Where is '\' when you need it?

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/e69c1049/attachment.html>

From chris.barker at noaa.gov  Sat Dec  7 01:44:32 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 6 Dec 2013 22:44:32 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
Message-ID: <CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>

On Fri, Dec 6, 2013 at 10:06 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

> One of the things that we should start doing for numpy is distribute
> releases as wheels. On OS X at least this is quite simple, so I propose to
> just experiment with it.
>

OK -- maybe on the wrong list, but an itch of mine is OSX binaries of
IPython (and the dependencies required for the notebook, too. There is
right no way for an OS_X user without the compiler setup to get iPython
without going to Anaconda or Canopy,

Yet it's a really great tool for newbies....

So I just sat down and did a simple:

pip wheel --wheel-dir=wheelhouse2 ipython[all]

Wow -- took a little while, but presto!  A pile of wheels, ready to go:

$ ls wheelhouse/
Jinja2-2.7.1-py27-none-any.whl
pyzmq-14.0.1-cp27-none-macosx_10_6_intel.whl
MarkupSafe-0.18-cp27-none-macosx_10_6_intel.whl
readline-6.2.4.1-cp27-none-macosx_10_6_intel.whl
Pygments-1.6-py27-none-any.whl
tornado-3.1.1-py27-none-any.whl
ipython-1.1.0-py27-none-any.whl


Now, do they work? They do on my machine. Is there somewhere I could put
them up so folks could test?

-Chris






I can create some to try out and put them on a separate folder on
> SourceForge. If that works they can be put on PyPi.
>
> For Windows things are less simple, because the wheel format doesn't
> handle the multiple builds (no SSE, SSE2, SSE3) that are in the superpack
> installers. A problem is that we don't really know how many users still
> have old CPUs that don't support SSE3. The impact for those users is high,
> numpy will install but crash (see
> https://github.com/scipy/scipy/issues/1697). Questions:
> 1. does anyone have a good idea to obtain statistics?
> 2. in the absence of statistics, can we do an experiment by putting one
> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
> propose, and seeing for how many (if any) users this goes wrong?
>
> Ralf
>
> P.S. related question: did anyone check whether the recently merged
> NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131206/63a4bea1/attachment.html>

From ralf.gommers at gmail.com  Sat Dec  7 07:09:17 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 7 Dec 2013 13:09:17 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
Message-ID: <CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>

On Sat, Dec 7, 2013 at 7:44 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Dec 6, 2013 at 10:06 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>
>> One of the things that we should start doing for numpy is distribute
>> releases as wheels. On OS X at least this is quite simple, so I propose to
>> just experiment with it.
>>
>
> OK -- maybe on the wrong list, but an itch of mine is OSX binaries of
> IPython (and the dependencies required for the notebook, too. There is
> right no way for an OS_X user without the compiler setup to get iPython
> without going to Anaconda or Canopy,
>
> Yet it's a really great tool for newbies....
>
> So I just sat down and did a simple:
>
> pip wheel --wheel-dir=wheelhouse2 ipython[all]
>
> Wow -- took a little while, but presto!  A pile of wheels, ready to go:
>
> $ ls wheelhouse/
> Jinja2-2.7.1-py27-none-any.whl
> pyzmq-14.0.1-cp27-none-macosx_10_6_intel.whl
> MarkupSafe-0.18-cp27-none-macosx_10_6_intel.whl
> readline-6.2.4.1-cp27-none-macosx_10_6_intel.whl
> Pygments-1.6-py27-none-any.whl
> tornado-3.1.1-py27-none-any.whl
> ipython-1.1.0-py27-none-any.whl
>
>
> Now, do they work? They do on my machine. Is there somewhere I could put
> them up so folks could test?
>

You can't upload that whole stack anywhere pip finds it automatically.
Temporarily you can put them on SourceForge or on any public download site.
Then people can download and install with wheel. If you send me a link to
those files, then I'll put them up together with the numpy wheels on SF.

Ralf


>
>  -Chris
>
>
>
>
>
>
> I can create some to try out and put them on a separate folder on
>> SourceForge. If that works they can be put on PyPi.
>>
>> For Windows things are less simple, because the wheel format doesn't
>> handle the multiple builds (no SSE, SSE2, SSE3) that are in the superpack
>> installers. A problem is that we don't really know how many users still
>> have old CPUs that don't support SSE3. The impact for those users is high,
>> numpy will install but crash (see
>> https://github.com/scipy/scipy/issues/1697). Questions:
>> 1. does anyone have a good idea to obtain statistics?
>> 2. in the absence of statistics, can we do an experiment by putting one
>> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
>> propose, and seeing for how many (if any) users this goes wrong?
>>
>> Ralf
>>
>> P.S. related question: did anyone check whether the recently merged
>> NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131207/9f6c68ef/attachment.html>

From chris.barker at noaa.gov  Sat Dec  7 14:01:38 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 7 Dec 2013 11:01:38 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
	<CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>
Message-ID: <CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>

On Sat, Dec 7, 2013 at 4:09 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Wow -- took a little while, but presto!  A pile of wheels, ready to go:
>
>>
>> $ ls wheelhouse/
>> Jinja2-2.7.1-py27-none-any.whl
>> pyzmq-14.0.1-cp27-none-macosx_10_6_intel.whl
>> MarkupSafe-0.18-cp27-none-macosx_10_6_intel.whl
>> readline-6.2.4.1-cp27-none-macosx_10_6_intel.whl
>> Pygments-1.6-py27-none-any.whl
>> tornado-3.1.1-py27-none-any.whl
>> ipython-1.1.0-py27-none-any.whl
>>
>>
>> Now, do they work? They do on my machine. Is there somewhere I could put
>> them up so folks could test?
>>
>
> You can't upload that whole stack anywhere pip finds it automatically.
>

yeah, that's where I'm still a little confused about pip and a "wheelhouse"
-- other than PyPi, is there a way to put a pile of wheels somewhere and
point pip to them -- i.e. simple http or ftp server or something? Or are
folks going to need to download the whole pile first, then point pip at a
local dir?

Temporarily you can put them on SourceForge or on any public download site.
> Then people can download and install with wheel. If you send me a link to
> those files, then I'll put them up together with the numpy wheels on SF.
>

Thanks -- I'll try to do that later today.

-Chris





>
> Ralf
>
>
>>
>>  -Chris
>>
>>
>>
>>
>>
>>
>> I can create some to try out and put them on a separate folder on
>>> SourceForge. If that works they can be put on PyPi.
>>>
>>> For Windows things are less simple, because the wheel format doesn't
>>> handle the multiple builds (no SSE, SSE2, SSE3) that are in the superpack
>>> installers. A problem is that we don't really know how many users still
>>> have old CPUs that don't support SSE3. The impact for those users is high,
>>> numpy will install but crash (see
>>> https://github.com/scipy/scipy/issues/1697). Questions:
>>> 1. does anyone have a good idea to obtain statistics?
>>> 2. in the absence of statistics, can we do an experiment by putting one
>>> wheel up on PyPi which contains SSE3 instructions, for python 3.3 I
>>> propose, and seeing for how many (if any) users this goes wrong?
>>>
>>> Ralf
>>>
>>> P.S. related question: did anyone check whether the recently merged
>>> NPY_HAVE_SSE2_INTRINSIC puts SSE2 instructions into the no-SSE binary?
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131207/f9ed25eb/attachment.html>

From chris.barker at noaa.gov  Sat Dec  7 21:24:19 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 7 Dec 2013 18:24:19 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
	<CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>
	<CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>
Message-ID: <CALGmxEJF7R_-bJof+H5FNRUPuyiH6uaMZjc3bojGC=9BqOpHjg@mail.gmail.com>

>
>  Temporarily you can put them on SourceForge or on any public download
>> site. Then people can download and install with wheel. If you send me a
>> link to those files, then I'll put them up together with the numpy wheels
>> on SF.
>>
>
> Thanks -- I'll try to do that later today.--
>

Done:

https://www.dropbox.com/sh/pqn6fag18rgewlr/QQdNUwT7Fw/OSX_Wheels




Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131207/a819e170/attachment.html>

From ralf.gommers at gmail.com  Sun Dec  8 05:06:55 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 8 Dec 2013 11:06:55 +0100
Subject: [Numpy-discussion] ANN: Scipy 0.13.2 release
Message-ID: <CABL7CQh82_O2R=0DdCKty8z--wCkKHOWVUm69wMbe-ZScYAwUw@mail.gmail.com>

Hi,

I'm happy to announce the availability of the scipy 0.13.2 release. This is
a bugfix only release; it contains fixes for ndimage and optimize, and most
importantly was compiled with Cython 0.19.2 to fix memory leaks in code
using Cython fused types.

Source tarballs, binaries and release notes can be found at
http://sourceforge.net/projects/scipy/files/scipy/0.13.2/

Cheers,
Ralf


==========================
SciPy 0.13.2 Release Notes
==========================

SciPy 0.13.2 is a bug-fix release with no new features compared to 0.13.1.


Issues fixed
------------

- 3096: require Cython 0.19, earlier versions have memory leaks in fused
types
- 3079: ``ndimage.label`` fix swapped 64-bitness test
- 3108: ``optimize.fmin_slsqp`` constraint violation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/6e1e9862/attachment.html>

From ralf.gommers at gmail.com  Sun Dec  8 05:59:18 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 8 Dec 2013 11:59:18 +0100
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
	<CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>
	<CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>
Message-ID: <CABL7CQi0SUF=dpHwgHLzy2CmT=NzvWKEfikzHuTTWTum-a23kw@mail.gmail.com>

On Sat, Dec 7, 2013 at 8:01 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Sat, Dec 7, 2013 at 4:09 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>
> Wow -- took a little while, but presto!  A pile of wheels, ready to go:
>>
>>>
>>> $ ls wheelhouse/
>>> Jinja2-2.7.1-py27-none-any.whl
>>> pyzmq-14.0.1-cp27-none-macosx_10_6_intel.whl
>>> MarkupSafe-0.18-cp27-none-macosx_10_6_intel.whl
>>> readline-6.2.4.1-cp27-none-macosx_10_6_intel.whl
>>> Pygments-1.6-py27-none-any.whl
>>> tornado-3.1.1-py27-none-any.whl
>>> ipython-1.1.0-py27-none-any.whl
>>>
>>>
>>> Now, do they work? They do on my machine. Is there somewhere I could put
>>> them up so folks could test?
>>>
>>
>> You can't upload that whole stack anywhere pip finds it automatically.
>>
>
> yeah, that's where I'm still a little confused about pip and a
> "wheelhouse" -- other than PyPi, is there a way to put a pile of wheels
> somewhere and point pip to them -- i.e. simple http or ftp server or
> something? Or are folks going to need to download the whole pile first,
> then point pip at a local dir?
>

I'm under the impression that
     $ pip install --use-wheel --no-index --find-links=/local_download_dir
ipython
and
    $ pip install --use-wheel --no-index --find-links=hosting_url ipython
should both work. But I've been running into multiple issues so far - from
having to upgrade pip itself and having to manually remove setuptools to
having no wheel-2.7 command (when "wheel" is the 2.6 version).

I've uploaded numpy and scipy wheels plus your set at
http://sourceforge.net/projects/numpy/files/wheels_to_test/. I'll start a
new thread with a request for testing.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/122b95be/attachment.html>

From ralf.gommers at gmail.com  Sun Dec  8 06:12:07 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 8 Dec 2013 12:12:07 +0100
Subject: [Numpy-discussion] wheels for OS X: request for testing
Message-ID: <CABL7CQjQ88UGr6ANYv_F63vMfdkgurWE1vc+yz8Fqt=67aDC-A@mail.gmail.com>

Hi all,

At http://sourceforge.net/projects/numpy/files/wheels_to_test/ you can find
a set of wheels, for numpy, scipy and ipython plus its dependencies. These
are for the following configuration only:
  - OSX >= 10.6
  - Python 2.7 32/64-bit from python.org

It would be great if OS X users could test these. We're interested in
whether these wheels work, and also if you run into any issues installing
wheels. Since (a) I'm not really sure myself what the recommended and most
robust way is to install wheels, and (b) it's interesting to see if the
docs and usability of pip/wheel are ready for prime time, I'm not going to
give the commands to execute but instead link to the relevant docs:

http://www.pip-installer.org/en/latest/index.html
http://python-packaging-user-guide.readthedocs.org/en/latest/
http://wheel.readthedocs.org/en/latest/index.html

Please try this out and share your experience.

Thanks,
Ralf

P.S.if you're wondering what on earth wheels are, they're the new binary
install format for the standard python packaging and distribution tools
(pip, setuptools).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/91fd95ec/attachment.html>

From josef.pktd at gmail.com  Sun Dec  8 09:40:39 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 8 Dec 2013 09:40:39 -0500
Subject: [Numpy-discussion] Deprecate boolean math operators?
In-Reply-To: <CAB6mnx+hbNFo+A6ZH_TeT3p8UPRRWP==z6PrMWKbMi4R+99f7A@mail.gmail.com>
References: <1386283039.23728.6.camel@sebastian-laptop>
	<CAMMTP+DYiN8ve0hOtV0BwWKJcoWZ6jd3oYV=6Gh5LOHLrjXexw@mail.gmail.com>
	<52A14D0C.1060209@gmail.com>
	<CAP7h-xZb_PVMy+Vh1DrmYBdAktT+Mj=6eYQBBuWkEoYmrs6_Uw@mail.gmail.com>
	<52A1F793.3010506@gmail.com>
	<CAP7h-xaHXnVf+p6YqEdfcY9sDQy9BPkNroPuAu70i40DoTKriA@mail.gmail.com>
	<52A21468.8010809@gmail.com>
	<CAMMTP+ANkd4-3UJPiKr2cHOxPZY__deGYR48t0QZPDsia4mAJw@mail.gmail.com>
	<52A21B7C.6000908@gmail.com>
	<CAP7h-xb=p4P083h2FL=kvXutM_16BHr9F3V6YWN7WrUPCknZUQ@mail.gmail.com>
	<CAPJVwBndEg7_YA0iKRNhdE5aMCmJhcTViVt+A2OoJObh=V7U5Q@mail.gmail.com>
	<CAMMTP+A=G3PGFt1hkcCkhdGAMLh7quCU-Ap8s=Nm7TcBkp4bAA@mail.gmail.com>
	<1386363042.29623.10.camel@sebastian-laptop>
	<CAMMTP+BDQ7u+gAkvc9swcD_c_J+mO=zPEZpHhULy6vg+9RiDHA@mail.gmail.com>
	<CAB6mnx+hbNFo+A6ZH_TeT3p8UPRRWP==z6PrMWKbMi4R+99f7A@mail.gmail.com>
Message-ID: <CAMMTP+DcdEzxkMtaN_8pu6z2Ry6fYu7V_Yavcj5hqpA1oiYCVQ@mail.gmail.com>

On Fri, Dec 6, 2013 at 11:25 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
>
> On Fri, Dec 6, 2013 at 2:14 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Fri, Dec 6, 2013 at 3:50 PM, Sebastian Berg
>> <sebastian at sipsolutions.net> wrote:
>> > On Fri, 2013-12-06 at 15:30 -0500, josef.pktd at gmail.com wrote:
>> >> On Fri, Dec 6, 2013 at 2:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> >> > On Fri, Dec 6, 2013 at 11:55 AM, Alexander Belopolsky
>> >> > <ndarray at mac.com> wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Fri, Dec 6, 2013 at 1:46 PM, Alan G Isaac <alan.isaac at gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> On 12/6/2013 1:35 PM, josef.pktd at gmail.com wrote:
>> >> >>> > unary versus binary minus
>> >> >>>
>> >> >>> Oh right; I consider binary `-` broken for
>> >> >>> Boolean arrays. (Sorry Alexander; I did not
>> >> >>> see your entire issue.)
>> >> >>>
>> >> >>>
>> >> >>> > I'd rather write ~ than unary - if that's what it is.
>> >> >>>
>> >> >>> I agree.  So I have no objection to elimination
>> >> >>> of the `-`.
>> >> >>
>> >> >>
>> >> >> It looks like we are close to reaching a consensus on the following
>> >> >> points:
>> >> >>
>> >> >> 1. * is well-defined on boolean arrays and may be used in preference
>> >> >> of & in
>> >> >> code that is designed to handle 1s and 0s of any dtype in addition
>> >> >> to
>> >> >> booleans.
>> >> >>
>> >> >> 2. + is defined consistently with * and the only issue is the
>> >> >> absence of
>> >> >> additive inverse.  This is not a problem as long as presence of -
>> >> >> does not
>> >> >> suggest otherwise.
>> >> >>
>> >> >> 3. binary and unary minus should be deprecated because its use in
>> >> >> expressions where variables can be either boolean or numeric would
>> >> >> lead to
>> >> >> subtle bugs.  For example -x*y would produce different results from
>> >> >> -(x*y)
>> >> >> depending on whether x is boolean or not.  In all situations, ^ is
>> >> >> preferable to binary - and ~ is preferable to unary -.
>> >> >>
>> >> >> 4. changing boolean arithmetics to auto-promotion to int is
>> >> >> precluded by a
>> >> >> significant use-case of boolean matrices.
>> >> >
>> >> > +1
>> >>
>> >> +0.5
>> >> (I would still prefer a different binary minus, but it would be
>> >> inconsistent with a logical unary minus that negates.)
>> >>
>> >
>> > The question is if the current xor behaviour can make sense? It doesn't
>> > seem to make much sense mathematically? Which only leaves that `abs(x -
>> > y)` is actually what a (python) programmer might expect.
>> > I think I would like to deprecate at least the unary one. The ~ kind of
>> > behaviour just doesn't fit as far as I can see.
>>
>> I haven't seen any real use cases for xor yet.
>
>
> Using it instead of '+' yields a boolean ring instead of semi-ring. Papers
> from the first quarter of the last century used it pretty often on that
> account, hence 'sigma-rings', etc. Eventually the simplicity of the
> inclusive or overcame that tendency.
>
>> My impression is that both plus and minus are just overflow accidents
>> and not intentional. plus works in a useful way, minus as xor might be
>> used once per century.
>
>
> It's certainly weird given that '+' means the inclusive or. I think '^' is
> much preferable.
> Although it makes some sense if one can keep the semantics straight.
> Complicated, though.

I'm looking at the test failure with allclose

Looks like - as xor still makes sense in some cases, because it
doesn't need special cases for equality checks for example.
x - y == 0 iff x == y

What happens to np.diff?

>>> np.diff(m1)
array([False,  True, False], dtype=bool)

I'm using code like that to get the length of runs in a runstest.

But in my current code, I actually have astype(int) and also use
floats later on.
If I read my (incompletely documented) code correctly, I needed also
the sign not just the run length.

Just another argument what minus could be.

Josef

>
>>
>> I would deprecate both unary and binary minus.
>>
>> (And when nobody is looking in two versions from now, I would add a
>> binary minus that overflows to the clipped version, so I get a set
>> subtraction. :)
>
>
> Where is '\' when you need it?
>
> <snip>
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From chris.barker at noaa.gov  Sun Dec  8 12:23:09 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 8 Dec 2013 09:23:09 -0800
Subject: [Numpy-discussion] distributing wheels & SSE/superpack options
In-Reply-To: <CABL7CQi0SUF=dpHwgHLzy2CmT=NzvWKEfikzHuTTWTum-a23kw@mail.gmail.com>
References: <CABL7CQijJ8MPLpbkmMsHK7DE22FP9+LyuGbzqFBecWm_GerHUw@mail.gmail.com>
	<CALGmxE+rc7YUwEcRsXRR=v8s49EP43VgsKQ_FqWK1Qd7XYzDXw@mail.gmail.com>
	<CABL7CQi2NM459ZgBj57voboP=CSFLK7bW2e+78fvbmu72__DpA@mail.gmail.com>
	<CALGmxE+3pihvJjU=yc6-QcZC+fjYHTD_La5f4UwCVM+yOuQYsg@mail.gmail.com>
	<CABL7CQi0SUF=dpHwgHLzy2CmT=NzvWKEfikzHuTTWTum-a23kw@mail.gmail.com>
Message-ID: <CALGmxEKjC0prOt64c0av71rT1Gk4NFPja8TjE=3xHCKcfmj17g@mail.gmail.com>

On Sun, Dec 8, 2013 at 2:59 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> I'm under the impression that
>      $ pip install --use-wheel --no-index --find-links=/local_download_dir
> ipython
> and
>     $ pip install --use-wheel --no-index --find-links=hosting_url ipython
> should both work.
>

Cool that _should_ be easy and useful

 But I've been running into multiple issues so far - from having to upgrade
> pip itself and having to manually remove setuptools to having no wheel-2.7
> command (when "wheel" is the 2.6 version).
>

oh well -- this just shows how little that has been tested. Whici is why
it's good we're doing this.

I've uploaded numpy and scipy wheels plus your set at
> http://sourceforge.net/projects/numpy/files/wheels_to_test/. I'll start a
> new thread with a request for testing.
>
> great, thanks!

-Chris





> Ralf
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/509b378b/attachment.html>

From chris.barker at noaa.gov  Sun Dec  8 12:34:55 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 8 Dec 2013 09:34:55 -0800
Subject: [Numpy-discussion] wheels for OS X: request for testing
In-Reply-To: <CABL7CQjQ88UGr6ANYv_F63vMfdkgurWE1vc+yz8Fqt=67aDC-A@mail.gmail.com>
References: <CABL7CQjQ88UGr6ANYv_F63vMfdkgurWE1vc+yz8Fqt=67aDC-A@mail.gmail.com>
Message-ID: <CALGmxEKW++0ikjNwkCSN2saLox7Kcinzrt5N4i29rNW2+2nyPQ@mail.gmail.com>

Thanks Ralf!

Since (a) I'm not really sure myself what the recommended and most robust
way is to install wheels, and (b) it's interesting to see if the docs and
usability of pip/wheel are ready for prime time, I'm not going to give the
commands to execute but instead link to the relevant docs:

Good idea. But I will add one tip:

you should be able to install ipython with one command, and have it
auto-mgically suck in its dependencies (and you probably want to use:

 ipython[all]

So that you get all the deps you need for the notebook -- that's where the
challenge lies...

-Chris




> http://www.pip-installer.org/en/latest/index.html
> http://python-packaging-user-guide.readthedocs.org/en/latest/
> http://wheel.readthedocs.org/en/latest/index.html
>
> Please try this out and share your experience.
>
> Thanks,
> Ralf
>
> P.S.if you're wondering what on earth wheels are, they're the new binary
> install format for the standard python packaging and distribution tools
> (pip, setuptools).
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/b3770a57/attachment.html>

From chris.barker at noaa.gov  Sun Dec  8 17:04:10 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 8 Dec 2013 14:04:10 -0800
Subject: [Numpy-discussion] wheels for OS X: request for testing
In-Reply-To: <CALGmxEKW++0ikjNwkCSN2saLox7Kcinzrt5N4i29rNW2+2nyPQ@mail.gmail.com>
References: <CABL7CQjQ88UGr6ANYv_F63vMfdkgurWE1vc+yz8Fqt=67aDC-A@mail.gmail.com>
	<CALGmxEKW++0ikjNwkCSN2saLox7Kcinzrt5N4i29rNW2+2nyPQ@mail.gmail.com>
Message-ID: <CALGmxE+VdE4__9t0MJkyB1RfiN9XU1iCDFYRiSkRboYqPXjTRQ@mail.gmail.com>

One more note:

These were built on a 10.7 machine, so I'm particularly interested to hear
how they work on 10.6, if anyone has an old machine around...

-Chris



On Sun, Dec 8, 2013 at 9:34 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> Thanks Ralf!
>
> Since (a) I'm not really sure myself what the recommended and most robust
> way is to install wheels, and (b) it's interesting to see if the docs and
> usability of pip/wheel are ready for prime time, I'm not going to give the
> commands to execute but instead link to the relevant docs:
>
> Good idea. But I will add one tip:
>
> you should be able to install ipython with one command, and have it
> auto-mgically suck in its dependencies (and you probably want to use:
>
>  ipython[all]
>
> So that you get all the deps you need for the notebook -- that's where the
> challenge lies...
>
> -Chris
>
>
>
>
>> http://www.pip-installer.org/en/latest/index.html
>> http://python-packaging-user-guide.readthedocs.org/en/latest/
>> http://wheel.readthedocs.org/en/latest/index.html
>>
>> Please try this out and share your experience.
>>
>> Thanks,
>> Ralf
>>
>> P.S.if you're wondering what on earth wheels are, they're the new binary
>> install format for the standard python packaging and distribution tools
>> (pip, setuptools).
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131208/2310263a/attachment.html>

From cmkleffner at gmail.com  Mon Dec  9 18:34:23 2013
From: cmkleffner at gmail.com (Carl Kleffner)
Date: Tue, 10 Dec 2013 00:34:23 +0100
Subject: [Numpy-discussion] OpenBLAS based numpy scipy test builds for WIN64
Message-ID: <CAGGsPMxcsL5sNr8NVLvjnOjthqk=ubGfzDAU9AgLACndhNXFoA@mail.gmail.com>

Hi list,

I've uploaded on https://code.google.com/p/mingw-w64-static/  numpy/scipy
binaries as wheel builds for testing. The binaries have been build with the
help of a customized mingw-w64 toolchain and a recent (git) openBLAS.

Regards

Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131210/8d536785/attachment.html>

From andrew.collette at gmail.com  Mon Dec  9 19:28:46 2013
From: andrew.collette at gmail.com (Andrew Collette)
Date: Mon, 9 Dec 2013 17:28:46 -0700
Subject: [Numpy-discussion] ANN: HDF5 for Python 2.2.1
Message-ID: <CALmrCV2vn71ot4V0kbDODUnnBOaEuk2CabZyXEodYtEsVb+0yw@mail.gmail.com>

Announcing HDF5 for Python (h5py) 2.2.1
=======================================

The h5py team is happy, in a sense, to announce the availability of h5py 2.2.1.
This release fixes a critical bug reported by Jim Parker on December 7th, which
affects code using HDF5 compound types.

We recommend that all users of h5py 2.2.0 upgrade to avoid crashes or possible
data corruption.

About h5py, downloads, documentation: http://www.h5py.org


Scope of bug
------------

The issue affects a feature introduced in h5py 2.2.0, in which HDF5 compound
datasets may be updated in-place, by specifying a field name or names when
writing to the dataset:

>>> dataset['field_name'] = value

Under certain conditions, h5py can supply uninitialized memory to the HDF5
conversion machinery, leading (in the case reported) to a segmentation fault.
It is also possible for other fields of the type to be corrupted.

This issue affects only code which updates a subset of the fields in the
compound type.  Programs reading from a compound type, writing all fields, or
using other datatypes, are not affected; nor are versions of h5py
prior to 2.2.0.


More information
----------------

Github issue:       https://github.com/h5py/h5py/issues/372
Original thread:    https://groups.google.com/forum/#!topic/h5py/AbUOZ1MXf3U

Thanks also to Christoph Gohlke for making Windows installers available on very
short notice, after a glitch in the h5py build system.


From pierre.haessig at crans.org  Thu Dec 12 09:20:05 2013
From: pierre.haessig at crans.org (Pierre Haessig)
Date: Thu, 12 Dec 2013 15:20:05 +0100
Subject: [Numpy-discussion] repeat array in a fake dim without stride_tricks
	?
Message-ID: <52A9C615.1010909@crans.org>

Hello,

In order to repeat rows or columns of an array as
http://stackoverflow.com/questions/1550130/cloning-row-or-column-vectors
I can use np.repeat as suggested by pv. However, looking at the flags of
the resulting array, data seems to be copied and actually repeated in
memory. This is not applicable if want a 1000x repetition.

What are the other options for such a repeat ?

On scipy lectures, there is a suggestion to use as_strided :
http://scipy-lectures.github.io/advanced/advanced_numpy/#example-fake-dimensions-with-strides

Otherwise, I see broadcast_arrays :

> N = 3
> data = np.arange(N)
> np.broadcast_arrays(data[:,None], np.zeros((1,2)))[0]
array([[0, 0],
       [1, 1],
       [2, 2]])

This works but it feels like invoking a magic formula. Did I miss a
simpler function ?

best,
Pierre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pierre_haessig.vcf
Type: text/x-vcard
Size: 329 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131212/4927c845/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 897 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131212/4927c845/attachment.sig>

From david.jones74 at gmail.com  Thu Dec 12 13:58:39 2013
From: david.jones74 at gmail.com (David Jones)
Date: Thu, 12 Dec 2013 13:58:39 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
Message-ID: <52AA075F.8060708@gmail.com>

I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:

"Broken toolchain: cannot link a simple C program"

It get's the compile flags right, but not the linker:

C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
_configtest.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status

I'm bulding it using a 32bit python build that I compiled on the same system.

I tried:

   OPT="-m32" FOPT="-m32" python setup.py build

and

   setarch x86_64 -B python setup.py build

But with the same results.


Someone worked around this by altering ccompiler.py, but I'm trying to find a cleaner solution.

See:http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-version-of-numpy

Is there a standard way of doing this?

Regards,
David J.



From jtaylor.debian at googlemail.com  Thu Dec 12 15:54:14 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Thu, 12 Dec 2013 21:54:14 +0100
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA075F.8060708@gmail.com>
References: <52AA075F.8060708@gmail.com>
Message-ID: <52AA2276.8020007@googlemail.com>

On 12.12.2013 19:58, David Jones wrote:
> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
> 
> "Broken toolchain: cannot link a simple C program"
> 
> It get's the compile flags right, but not the linker:
> 
> C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
> compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c'
> gcc: _configtest.c
> gcc -pthread _configtest.o -o _configtest
> _configtest.o: could not read symbols: File in wrong format
> collect2: ld returned 1 exit status
> 
> I'm bulding it using a 32bit python build that I compiled on the same system.
> 
> I tried:
> 
>    OPT="-m32" FOPT="-m32" python setup.py build
> 
> and
> 
>    setarch x86_64 -B python setup.py build
> 
> But with the same results.
> 
> 
> Someone worked around this by altering ccompiler.py, but I'm trying to find a cleaner solution.
> 
> See:http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-version-of-numpy
> 
> Is there a standard way of doing this?
> 

this might work:

CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
setup.py build



From david.jones74 at gmail.com  Thu Dec 12 14:40:06 2013
From: david.jones74 at gmail.com (David Jones)
Date: Thu, 12 Dec 2013 14:40:06 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA2276.8020007@googlemail.com>
References: <52AA075F.8060708@gmail.com> <52AA2276.8020007@googlemail.com>
Message-ID: <52AA1116.6080809@gmail.com>

On 12/12/13 15:54, Julian Taylor wrote:
> On 12.12.2013 19:58, David Jones wrote:
>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>
>> "Broken toolchain: cannot link a simple C program"
>>
>> It get's the compile flags right, but not the linker:
>>
>> C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -m32 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
>> compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/python/ia32/include/python2.7 -c'
>> gcc: _configtest.c
>> gcc -pthread _configtest.o -o _configtest
>> _configtest.o: could not read symbols: File in wrong format
>> collect2: ld returned 1 exit status
>>
>> I'm bulding it using a 32bit python build that I compiled on the same system.
>>
>> I tried:
>>
>>     OPT="-m32" FOPT="-m32" python setup.py build
>>
>> and
>>
>>     setarch x86_64 -B python setup.py build
>>
>> But with the same results.
>>
>>
>> Someone worked around this by altering ccompiler.py, but I'm trying to find a cleaner solution.
>>
>> See:http://stackoverflow.com/questions/11265057/how-do-i-install-a-32-bit-version-of-numpy
>>
>> Is there a standard way of doing this?
>>
> this might work:
>
> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
> setup.py build
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

That didn't work. It says it can't find executable "gcc -m32". I tried 
changing ccompiler.py, per the stackoverflow post. I got further that 
time, but it fails with the message:

build/temp.linux-i686-2.7/numpy/linalg/lapack_litemodule.o: could not 
read symbols: File in wrong format
collect2: ld returned 1 exit status
build/temp.linux-i686-2.7/numpy/linalg/lapack_litemodule.o: could not 
read symbols: File in wrong format
collect2: ld returned 1 exit status
error: Command "/usr/bin/gfortran -Wall -Wall -shared 
build/temp.linux-i686-2.7/numpy/linalg/lapack_litemodule.o 
build/temp.linux-i686-2.7/numpy/linalg/lapack_lite/python_xerbla.o 
-L/usr/lib/atlas -L/opt/python/ia32/lib -Lbuild/temp.linux-i686-2.7 
-llapack -lptf77blas -lptcblas -latlas -lpython2.7 -lgfortran -o 
build/lib.linux-i686-2.7/numpy/linalg/lapack_lite.so" failed with exit 
status 1

What build options does numpy recognize? The docs ony mention CC, OPT, 
and FOPT, but don't give any details on how to use them. Can you pass 
something to setup.py?



From jtaylor.debian at googlemail.com  Thu Dec 12 16:35:10 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Thu, 12 Dec 2013 22:35:10 +0100
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA1116.6080809@gmail.com>
References: <52AA075F.8060708@gmail.com> <52AA2276.8020007@googlemail.com>
	<52AA1116.6080809@gmail.com>
Message-ID: <52AA2C0E.2090600@googlemail.com>

On 12.12.2013 20:40, David Jones wrote:
> On 12/12/13 15:54, Julian Taylor wrote:
>> On 12.12.2013 19:58, David Jones wrote:
>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>
>>> "Broken toolchain: cannot link a simple C program"
>>>
...
>>>
>> this might work:
>>
>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>> setup.py build
>>

> 
> That didn't work. It says it can't find executable "gcc -m32". I tried 
> changing ccompiler.py, per the stackoverflow post. I got further that 
> time, but it fails with the message:
> 


weird it works for me with libpython2.7-dev:i386 installed on amd64
Ubuntu 13.10 (multiarch is friggin great :D)
But I had to add an additional LDFLAGS="-m32 -shared"

CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
python setup.py build

you should be able to work around this error message by writing small
wrapper scripts for the compiler:

cat << EOF > gcc-32
#!/bin/sh
gcc -m32 "$@"
EOF
...
CC=gcc-32 ....



From jaime.frio at gmail.com  Fri Dec 13 02:07:28 2013
From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=)
Date: Thu, 12 Dec 2013 23:07:28 -0800
Subject: [Numpy-discussion] Using np.partition to extract n largest/smallest
	items from an array
Message-ID: <CAPOWHW=eby8KrYao+5qkJP9BBL3j431i2x-YCHATjM622jxCcA@mail.gmail.com>

With the new np.partition functionality, there is a more efficient, but
also less obvious, way of extracting the n largest (or smallest) elements
from an array, i.e.:

def smallest_n(a, n):
    return np.sort(np.partition(a, n)[:n])

def argsmallest_n(a, n):
    ret = np.argpartition(a, n)[:n]
    b = np.take(a, ret)
    return np.take(ret, np.argsort(b))

instead of the usual:

np.sort(a)[:n]
np.argsort(a)[:n]

Are those 4 functions (smallest, argsmallest, largest, arglargest), with
adequate axis support, worthy of including in numpy, or is the name space
already too cluttered?

Jaime

-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131212/96984829/attachment.html>

From cournape at gmail.com  Fri Dec 13 05:02:05 2013
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Dec 2013 10:02:05 +0000
Subject: [Numpy-discussion] [SciPy-Dev] ANN: Scipy 0.13.2 release
In-Reply-To: <CABL7CQh82_O2R=0DdCKty8z--wCkKHOWVUm69wMbe-ZScYAwUw@mail.gmail.com>
References: <CABL7CQh82_O2R=0DdCKty8z--wCkKHOWVUm69wMbe-ZScYAwUw@mail.gmail.com>
Message-ID: <CAGY4rcV758VAWUrvBqj9kLvmtpWKHLDutAP9WM0dA+ZCGo48iQ@mail.gmail.com>

Hi Ralf,

Thanks a lot for the quick fix release.

I can confirm it builds and tests correctly on windows, rhel5 and osx (both
32 and 64 bits).

cheers,
David


On Sun, Dec 8, 2013 at 10:06 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

> Hi,
>
> I'm happy to announce the availability of the scipy 0.13.2 release. This
> is a bugfix only release; it contains fixes for ndimage and optimize, and
> most importantly was compiled with Cython 0.19.2 to fix memory leaks in
> code using Cython fused types.
>
> Source tarballs, binaries and release notes can be found at
> http://sourceforge.net/projects/scipy/files/scipy/0.13.2/
>
> Cheers,
> Ralf
>
>
> ==========================
> SciPy 0.13.2 Release Notes
> ==========================
>
> SciPy 0.13.2 is a bug-fix release with no new features compared to 0.13.1.
>
>
> Issues fixed
> ------------
>
> - 3096: require Cython 0.19, earlier versions have memory leaks in fused
> types
> - 3079: ``ndimage.label`` fix swapped 64-bitness test
> - 3108: ``optimize.fmin_slsqp`` constraint violation
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131213/7bf99d15/attachment.html>

From sebastian at sipsolutions.net  Fri Dec 13 07:45:37 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 13 Dec 2013 13:45:37 +0100
Subject: [Numpy-discussion] repeat array in a fake dim without
 stride_tricks ?
In-Reply-To: <52A9C615.1010909@crans.org>
References: <52A9C615.1010909@crans.org>
Message-ID: <1386938737.2127.1.camel@sebastian-laptop>

Hey,

On Thu, 2013-12-12 at 15:20 +0100, Pierre Haessig wrote:
> Hello,
> 
> In order to repeat rows or columns of an array as
> http://stackoverflow.com/questions/1550130/cloning-row-or-column-vectors
> I can use np.repeat as suggested by pv. However, looking at the flags of
> the resulting array, data seems to be copied and actually repeated in
> memory. This is not applicable if want a 1000x repetition.
> 
> What are the other options for such a repeat ?
> 

No, I don't think there are any other options. stride tricks are a bit
hidden, since in many cases it is more dangerous than helping.
Though with some care you can easily implement such functions using
stride_tricks.

Regards,

Sebastian

> On scipy lectures, there is a suggestion to use as_strided :
> http://scipy-lectures.github.io/advanced/advanced_numpy/#example-fake-dimensions-with-strides
> 
> Otherwise, I see broadcast_arrays :
> 
> > N = 3
> > data = np.arange(N)
> > np.broadcast_arrays(data[:,None], np.zeros((1,2)))[0]
> array([[0, 0],
>        [1, 1],
>        [2, 2]])
> 
> This works but it feels like invoking a magic formula. Did I miss a
> simpler function ?
> 
> best,
> Pierre
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From david.jones74 at gmail.com  Fri Dec 13 07:50:31 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 07:50:31 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA2C0E.2090600@googlemail.com>
References: <52AA075F.8060708@gmail.com>
	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>
	<52AA2C0E.2090600@googlemail.com>
Message-ID: <52AB0297.1030707@gmail.com>


On 12/12/13 16:35, Julian Taylor wrote:
> On 12.12.2013 20:40, David Jones wrote:
>> On 12/12/13 15:54, Julian Taylor wrote:
>>> On 12.12.2013 19:58, David Jones wrote:
>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>
>>>> "Broken toolchain: cannot link a simple C program"
>>>>
> ...
>>> this might work:
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>> setup.py build
>>>
>> That didn't work. It says it can't find executable "gcc -m32". I tried
>> changing ccompiler.py, per the stackoverflow post. I got further that
>> time, but it fails with the message:
>>
>
> weird it works for me with libpython2.7-dev:i386 installed on amd64
> Ubuntu 13.10 (multiarch is friggin great :D)
> But I had to add an additional LDFLAGS="-m32 -shared"
>
> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
> python setup.py build
>
> you should be able to work around this error message by writing small
> wrapper scripts for the compiler:
>
> cat << EOF > gcc-32
> #!/bin/sh
> gcc -m32 "$@"
> EOF
> ...
> CC=gcc-32 ....

I actually didn't expect it to work. It just seemed too simple. So, I'm 
surprised it worked for you. First off, are we using the same version of 
everything?

I'm using Python 2.7.5, built with the following options:

CFLAGS="-g -O2 -m32" LDFLAGS="-m32" ./configure --enable-unicode=ucs4 
--enable-shared --prefix=/opt/python/ia32 --with-valgrind

I'm building Numpy 1.8.0.

I'll try the wrapper method. It seems that distutils isn't always 
passing the build options to the compiler and linker, so spoofing gcc 
and gfortran may work.

It appears that numpy.distutils doesn't actually apply the linker flags. 
It never calls the link_executable function, except indirectly through 
it's reimplementation of the distutils config module. Moreover, the 
config module doesn't expose the necessary arguments of link_executable 
to set the build flags.

I want to look more into the code to see exactly what's happening. It 
looks like the distutils config module and command package could use a 
lot of work. It makes sense to pass in all the build options as 
arguments to setup.py, rather than using environment variable. The 
compiler modules support this, but they're never getting that 
information from the config module. Of course, the may have been fixed 
in Python 3. I don't know.

 From what I can tell, distutils uses 3 methods to determine the build 
configuration:
1 - What version is of Python is being used? 32 bit or 64 bit?
2 - What's the architecture of the system? i.e. via uname -r.
3 - Environment variables.

Unfortunately the behavior is inconsistent and poorly documented.


>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From david.jones74 at gmail.com  Fri Dec 13 08:38:47 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 08:38:47 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA2C0E.2090600@googlemail.com>
References: <52AA075F.8060708@gmail.com>
	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>
	<52AA2C0E.2090600@googlemail.com>
Message-ID: <52AB0DE7.1020503@gmail.com>

On 12/12/13 16:35, Julian Taylor wrote:
> On 12.12.2013 20:40, David Jones wrote:
>> On 12/12/13 15:54, Julian Taylor wrote:
>>> On 12.12.2013 19:58, David Jones wrote:
>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>
>>>> "Broken toolchain: cannot link a simple C program"
>>>>
> ...
>>> this might work:
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>> setup.py build
>>>
>> That didn't work. It says it can't find executable "gcc -m32". I tried
>> changing ccompiler.py, per the stackoverflow post. I got further that
>> time, but it fails with the message:
>>
>
> weird it works for me with libpython2.7-dev:i386 installed on amd64
> Ubuntu 13.10 (multiarch is friggin great :D)
> But I had to add an additional LDFLAGS="-m32 -shared"
>
> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
> python setup.py build
>
> you should be able to work around this error message by writing small
> wrapper scripts for the compiler:
>
> cat << EOF > gcc-32
> #!/bin/sh
> gcc -m32 "$@"
> EOF
> ...
> CC=gcc-32 ....

No luck. Here are my results:

bash~$ CC=gcc-32 LDSHARED=ld-32 LDFLAGS="-m32 -shared" linux32 python2.7 
setup.py build

/home/build/bin/ld-32: line 1: !/bin/sh: No such file or directory
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
/home/build/bin/ld-32: line 1: !/bin/sh: No such file or directory
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
error: Command "ld-32 -m32 -shared 
build/temp.linux-i686-2.7/numpy/core/src/dummymodule.o -L. 
-Lbuild/temp.linux-i686-2.7 -lm -lpython2.7 -o 
build/lib.linux-i686-2.7/numpy/core/_dummy.so" failed with exit status 1

This makes no sense to me. The linker is set to 32 bit. My python 
library path is 32 bit. I know it can find the library, or the build 
wouldn't have even started.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From david.jones74 at gmail.com  Fri Dec 13 09:02:35 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 09:02:35 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AA2C0E.2090600@googlemail.com>
References: <52AA075F.8060708@gmail.com>
	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>
	<52AA2C0E.2090600@googlemail.com>
Message-ID: <52AB137B.3000708@gmail.com>


On 12/12/13 16:35, Julian Taylor wrote:
> On 12.12.2013 20:40, David Jones wrote:
>> On 12/12/13 15:54, Julian Taylor wrote:
>>> On 12.12.2013 19:58, David Jones wrote:
>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>
>>>> "Broken toolchain: cannot link a simple C program"
>>>>
> ...
>>> this might work:
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>> setup.py build
>>>
>> That didn't work. It says it can't find executable "gcc -m32". I tried
>> changing ccompiler.py, per the stackoverflow post. I got further that
>> time, but it fails with the message:
>>
So this is what ended of working:

CC=gcc-32 LDSHARED=ld-32 LDFLAGS="-m32 -shared -L$LD_LIBRARY_PATH" 
python2.7 setup.py build

Somehow the value of LD_LIBRARY_PATH gets lost along the way. Perhaps 
distutils re-exports it with a different value, prior to calling the linker.

I was getting a gfortran error at one point, but I guess it inherits 
-m32 from gcc-m32, so it doesn't have to be set explicitly.

I still want to look into the code, to see what's going on in there. I 
may just write a custom config module, that allows me to pass in the 
build flags more cleanly.
>
> weird it works for me with libpython2.7-dev:i386 installed on amd64
> Ubuntu 13.10 (multiarch is friggin great :D)
> But I had to add an additional LDFLAGS="-m32 -shared"
>
> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
> python setup.py build
>
> you should be able to work around this error message by writing small
> wrapper scripts for the compiler:
>
> cat << EOF > gcc-32
> #!/bin/sh
> gcc -m32 "$@"
> EOF
> ...
> CC=gcc-32 ....
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From charles at crunch.io  Fri Dec 13 11:24:49 2013
From: charles at crunch.io (Charles G. Waldman)
Date: Fri, 13 Dec 2013 10:24:49 -0600
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AB137B.3000708@gmail.com>
References: <52AA075F.8060708@gmail.com> <52AA2276.8020007@googlemail.com>
	<52AA1116.6080809@gmail.com> <52AA2C0E.2090600@googlemail.com>
	<52AB137B.3000708@gmail.com>
Message-ID: <CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>

>>> this might work:
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>> setup.py build

Compiler flags can't be added to CC, they should be in CFLAGS.  (and
ld flags go into LDFLAGS).  Otherwise you are saying to use a program
called "gcc -m32" (with a space in the name).



On Fri, Dec 13, 2013 at 8:02 AM, David Jones <david.jones74 at gmail.com> wrote:
>
> On 12/12/13 16:35, Julian Taylor wrote:
>> On 12.12.2013 20:40, David Jones wrote:
>>> On 12/12/13 15:54, Julian Taylor wrote:
>>>> On 12.12.2013 19:58, David Jones wrote:
>>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>>
>>>>> "Broken toolchain: cannot link a simple C program"
>>>>>
>> ...
>>>> this might work:
>>>>
>>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>>> setup.py build
>>>>
>>> That didn't work. It says it can't find executable "gcc -m32". I tried
>>> changing ccompiler.py, per the stackoverflow post. I got further that
>>> time, but it fails with the message:
>>>
> So this is what ended of working:
>
> CC=gcc-32 LDSHARED=ld-32 LDFLAGS="-m32 -shared -L$LD_LIBRARY_PATH"
> python2.7 setup.py build
>
> Somehow the value of LD_LIBRARY_PATH gets lost along the way. Perhaps
> distutils re-exports it with a different value, prior to calling the linker.
>
> I was getting a gfortran error at one point, but I guess it inherits
> -m32 from gcc-m32, so it doesn't have to be set explicitly.
>
> I still want to look into the code, to see what's going on in there. I
> may just write a custom config module, that allows me to pass in the
> build flags more cleanly.
>>
>> weird it works for me with libpython2.7-dev:i386 installed on amd64
>> Ubuntu 13.10 (multiarch is friggin great :D)
>> But I had to add an additional LDFLAGS="-m32 -shared"
>>
>> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
>> python setup.py build
>>
>> you should be able to work around this error message by writing small
>> wrapper scripts for the compiler:
>>
>> cat << EOF > gcc-32
>> #!/bin/sh
>> gcc -m32 "$@"
>> EOF
>> ...
>> CC=gcc-32 ....
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From david.jones74 at gmail.com  Fri Dec 13 10:34:35 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 10:34:35 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>
References: <52AA075F.8060708@gmail.com>
	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>
	<52AA2C0E.2090600@googlemail.com>	<52AB137B.3000708@gmail.com>
	<CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>
Message-ID: <52AB290B.3020407@gmail.com>


On 12/13/13 11:24, Charles G. Waldman wrote:
>>>> this might work:
>>>>
>>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>>> setup.py build
> Compiler flags can't be added to CC, they should be in CFLAGS.  (and
> ld flags go into LDFLAGS).  Otherwise you are saying to use a program
> called "gcc -m32" (with a space in the name).
I had tried that. But it seems that distutils, or numpy.distutils, 
doesn't honor those flags consistently. Instead, I used the gcc 
wrappers, as suggested below.

As I said, a cleaner, universally applicable method is definitely needed.
>
>
>
> On Fri, Dec 13, 2013 at 8:02 AM, David Jones <david.jones74 at gmail.com> wrote:
>> On 12/12/13 16:35, Julian Taylor wrote:
>>> On 12.12.2013 20:40, David Jones wrote:
>>>> On 12/12/13 15:54, Julian Taylor wrote:
>>>>> On 12.12.2013 19:58, David Jones wrote:
>>>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>>>
>>>>>> "Broken toolchain: cannot link a simple C program"
>>>>>>
>>> ...
>>>>> this might work:
>>>>>
>>>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>>>> setup.py build
>>>>>
>>>> That didn't work. It says it can't find executable "gcc -m32". I tried
>>>> changing ccompiler.py, per the stackoverflow post. I got further that
>>>> time, but it fails with the message:
>>>>
>> So this is what ended of working:
>>
>> CC=gcc-32 LDSHARED=ld-32 LDFLAGS="-m32 -shared -L$LD_LIBRARY_PATH"
>> python2.7 setup.py build
>>
>> Somehow the value of LD_LIBRARY_PATH gets lost along the way. Perhaps
>> distutils re-exports it with a different value, prior to calling the linker.
>>
>> I was getting a gfortran error at one point, but I guess it inherits
>> -m32 from gcc-m32, so it doesn't have to be set explicitly.
>>
>> I still want to look into the code, to see what's going on in there. I
>> may just write a custom config module, that allows me to pass in the
>> build flags more cleanly.
>>> weird it works for me with libpython2.7-dev:i386 installed on amd64
>>> Ubuntu 13.10 (multiarch is friggin great :D)
>>> But I had to add an additional LDFLAGS="-m32 -shared"
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
>>> python setup.py build
>>>
>>> you should be able to work around this error message by writing small
>>> wrapper scripts for the compiler:
>>>
>>> cat << EOF > gcc-32
>>> #!/bin/sh
>>> gcc -m32 "$@"
>>> EOF
>>> ...
>>> CC=gcc-32 ....
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From david.jones74 at gmail.com  Fri Dec 13 12:46:25 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 12:46:25 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>
References: <52AA075F.8060708@gmail.com>
	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>
	<52AA2C0E.2090600@googlemail.com>	<52AB137B.3000708@gmail.com>
	<CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>
Message-ID: <52AB47F1.2040802@gmail.com>


On 12/13/13 11:24, Charles G. Waldman wrote:
>>>> this might work:
>>>>
>>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>>> setup.py build
> Compiler flags can't be added to CC, they should be in CFLAGS.  (and
> ld flags go into LDFLAGS).  Otherwise you are saying to use a program
> called "gcc -m32" (with a space in the name).
>
>
>
> On Fri, Dec 13, 2013 at 8:02 AM, David Jones <david.jones74 at gmail.com> wrote:
>> On 12/12/13 16:35, Julian Taylor wrote:
>>> On 12.12.2013 20:40, David Jones wrote:
>>>> On 12/12/13 15:54, Julian Taylor wrote:
>>>>> On 12.12.2013 19:58, David Jones wrote:
>>>>>> I'm trying to compile 32-bit numpy on a 64 bit Centos 6 system, but fails with the message:
>>>>>>
>>>>>> "Broken toolchain: cannot link a simple C program"
>>>>>>
>>> ...
>>>>> this might work:
>>>>>
>>>>> CC="gcc -m32" LDSHARED="gcc -m32" FF="gfortran -m32" linux32 python
>>>>> setup.py build
>>>>>
>>>> That didn't work. It says it can't find executable "gcc -m32". I tried
>>>> changing ccompiler.py, per the stackoverflow post. I got further that
>>>> time, but it fails with the message:
>>>>
>> So this is what ended of working:
>>
>> CC=gcc-32 LDSHARED=ld-32 LDFLAGS="-m32 -shared -L$LD_LIBRARY_PATH"
>> python2.7 setup.py build
>>
>> Somehow the value of LD_LIBRARY_PATH gets lost along the way. Perhaps
>> distutils re-exports it with a different value, prior to calling the linker.
>>
>> I was getting a gfortran error at one point, but I guess it inherits
>> -m32 from gcc-m32, so it doesn't have to be set explicitly.
>>
>> I still want to look into the code, to see what's going on in there. I
>> may just write a custom config module, that allows me to pass in the
>> build flags more cleanly.
>>> weird it works for me with libpython2.7-dev:i386 installed on amd64
>>> Ubuntu 13.10 (multiarch is friggin great :D)
>>> But I had to add an additional LDFLAGS="-m32 -shared"
>>>
>>> CC="gcc -m32" LDSHARED="gcc -m32 -shared" LDFLAGS="-m32 -shared" linux32
>>> python setup.py build
>>>
>>> you should be able to work around this error message by writing small
>>> wrapper scripts for the compiler:
>>>
>>> cat << EOF > gcc-32
>>> #!/bin/sh
>>> gcc -m32 "$@"
>>> EOF
>>> ...
>>> CC=gcc-32 ....
>>>

Correction. Of course LD_LIBRARY_PATH isn't seen by the compiler. It 
only applies at run time. How embarrasing:) This isn't the first time 
I've been bitten by that.

I don't mind doing that with manual builds, but what about with pip? Is 
there a way to avoid explicitly setting the library path every time you 
call pip, when using a custom python install?



From jtaylor.debian at googlemail.com  Fri Dec 13 13:48:36 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 13 Dec 2013 19:48:36 +0100
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AB47F1.2040802@gmail.com>
References: <52AA075F.8060708@gmail.com>	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>	<52AA2C0E.2090600@googlemail.com>	<52AB137B.3000708@gmail.com>	<CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>
	<52AB47F1.2040802@gmail.com>
Message-ID: <52AB5684.5050907@googlemail.com>

On 13.12.2013 18:46, David Jones wrote:
> ...
> 
> Correction. Of course LD_LIBRARY_PATH isn't seen by the compiler. It 
> only applies at run time. How embarrasing:) This isn't the first time 
> I've been bitten by that.
> 
> I don't mind doing that with manual builds, but what about with pip? Is 
> there a way to avoid explicitly setting the library path every time you 
> call pip, when using a custom python install?
> 

why are you actually doing this?

the easiest way to get 32 bit binaries for any program is simply using a
32 bit chroot to build them.
This is trivial to do on a debian based systems:

pbuilder-dist i386 unstable create
pbuilder-dist i386 unstable login
# install dependencies and build as usual


From david.jones74 at gmail.com  Fri Dec 13 13:07:15 2013
From: david.jones74 at gmail.com (David Jones)
Date: Fri, 13 Dec 2013 13:07:15 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AB5684.5050907@googlemail.com>
References: <52AA075F.8060708@gmail.com>	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>	<52AA2C0E.2090600@googlemail.com>	<52AB137B.3000708@gmail.com>	<CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>	<52AB47F1.2040802@gmail.com>
	<52AB5684.5050907@googlemail.com>
Message-ID: <52AB4CD3.5050103@gmail.com>


On 12/13/13 13:48, Julian Taylor wrote:
> On 13.12.2013 18:46, David Jones wrote:
>> ...
>>
>> Correction. Of course LD_LIBRARY_PATH isn't seen by the compiler. It
>> only applies at run time. How embarrasing:) This isn't the first time
>> I've been bitten by that.
>>
>> I don't mind doing that with manual builds, but what about with pip? Is
>> there a way to avoid explicitly setting the library path every time you
>> call pip, when using a custom python install?
>>
> why are you actually doing this?
>
> the easiest way to get 32 bit binaries for any program is simply using a
> 32 bit chroot to build them.
> This is trivial to do on a debian based systems:
>
> pbuilder-dist i386 unstable create
> pbuilder-dist i386 unstable login
> # install dependencies and build as usual
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

I'm on Centos 6, but I imagine it's not that hard there either. I am 
doing all this in a chroot, but it's 64 bit.

The purpose is to use python (via cython) for testing libraries built on 
a 64 bit CentOS system. It's much faster than running all the tests 
directly in C, because I can do everything interactively in ipython. 
However, the libraries are built 32 bit. WIth the simpler components I 
can just rebuild them as 64-bit, but with others that gets pretty 
complicated.

  To use a more current version of Python on CentOS I have to build it 
myself, and the simplest way to keep it up to date is using pip. 
However, pip installs everything from source. I ran into trouble 
installing numpy, but I imagine I'd have similar problems with other 
packages.

I figure I can set the necessary environment variables in the 
virtualenv, and then pip should work.

Anyways, thanks for your help.


From ndarray at mac.com  Sat Dec 14 14:39:50 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Sat, 14 Dec 2013 14:39:50 -0500
Subject: [Numpy-discussion] Does NumPy support indirect memory views?
Message-ID: <CAP7h-xZji_miVE=2x-LT=QLeKH+eZnyFTFYWaq+ETE86Em2O7w@mail.gmail.com>

PEP 3118 [1] allows exposing multi-dimensional data that is organized as
array of pointers.  It appears, however that NumPy cannot consume such
memory views.

Looking at _array_from_buffer_3118() function [2], I don't see any attempt
to process suboffsets.   The documentation [3] is also silent on this issue.

What is the status of indirect memory views/buffers support in NumPy?


[1] http://www.python.org/dev/peps/pep-3118/
[2]
https://github.com/numpy/numpy/blob/4050ac73af79ae8cc513648ff02e9a22041501c4/numpy/core/src/multiarray/ctors.c#L1253
[3] http://docs.scipy.org/doc/numpy/reference/arrays.interface.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131214/8a5dc933/attachment.html>

From cournape at gmail.com  Sat Dec 14 14:59:10 2013
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 14 Dec 2013 19:59:10 +0000
Subject: [Numpy-discussion] Does NumPy support indirect memory views?
In-Reply-To: <CAP7h-xZji_miVE=2x-LT=QLeKH+eZnyFTFYWaq+ETE86Em2O7w@mail.gmail.com>
References: <CAP7h-xZji_miVE=2x-LT=QLeKH+eZnyFTFYWaq+ETE86Em2O7w@mail.gmail.com>
Message-ID: <CAGY4rcXWoCMgaWg6fyW12q9UCWPdCnfS8TkkUFc4raRzD2mf-g@mail.gmail.com>

On Sat, Dec 14, 2013 at 7:39 PM, Alexander Belopolsky <ndarray at mac.com>wrote:

> PEP 3118 [1] allows exposing multi-dimensional data that is organized as
> array of pointers.  It appears, however that NumPy cannot consume such
> memory views.
>
> Looking at _array_from_buffer_3118() function [2], I don't see any attempt
> to process suboffsets.   The documentation [3] is also silent on this issue.
>
> What is the status of indirect memory views/buffers support in NumPy?
>

There is indeed no support in NumPy for this. Unfortunately, fixing this
would be a significant amount of work, as buffer management is not really
abstracted in NumPy ATM.

David

>
>
> [1] http://www.python.org/dev/peps/pep-3118/
> [2]
> https://github.com/numpy/numpy/blob/4050ac73af79ae8cc513648ff02e9a22041501c4/numpy/core/src/multiarray/ctors.c#L1253
> [3] http://docs.scipy.org/doc/numpy/reference/arrays.interface.html
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131214/0904a553/attachment.html>

From ndarray at mac.com  Sat Dec 14 17:22:31 2013
From: ndarray at mac.com (Alexander Belopolsky)
Date: Sat, 14 Dec 2013 17:22:31 -0500
Subject: [Numpy-discussion] Does NumPy support indirect memory views?
In-Reply-To: <CAGY4rcXWoCMgaWg6fyW12q9UCWPdCnfS8TkkUFc4raRzD2mf-g@mail.gmail.com>
References: <CAP7h-xZji_miVE=2x-LT=QLeKH+eZnyFTFYWaq+ETE86Em2O7w@mail.gmail.com>
	<CAGY4rcXWoCMgaWg6fyW12q9UCWPdCnfS8TkkUFc4raRzD2mf-g@mail.gmail.com>
Message-ID: <CAP7h-xZXWRvdD+m6M2_99fusN-SQ2PdOGZNwhRekCeeC9z2TrQ@mail.gmail.com>

On Sat, Dec 14, 2013 at 2:59 PM, David Cournapeau <cournape at gmail.com>wrote:

> There is indeed no support in NumPy for this. Unfortunately, fixing this
> would be a significant amount of work, as buffer management is not really
> abstracted in NumPy ATM.


While providing a full support for indirect buffers as a storage for NumPy
ndarrays does look like a daunting task, I think some partial support can
be implemented rather easily.

When an ndarray from object constructor encounters an object that can only
expose its memory as an indirect buffer, the constructor can gather the
data into a contiguous buffer.

At the very least, _array_from_buffer_3118() should detect non-null
suboffsets and bail out with a meaningful message rather than expose
pointers as data.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131214/f94a6ed8/attachment.html>

From tmp50 at ukr.net  Sun Dec 15 06:51:01 2013
From: tmp50 at ukr.net (Dmitrey)
Date: Sun, 15 Dec 2013 13:51:01 +0200
Subject: [Numpy-discussion] [ANN] OpenOpt suite v 0.52
Message-ID: <1387108021.728479007.a89e227c@frv46.ukr.net>

Hi all, 
I'm glad to inform you about new? OpenOpt ? Suite release 0.52 (2013-Dec-15): 
? ? Minor? interalg ? speedup 
? ? oofun expression 
? ? MATLAB solvers fmincon and fsolve have been connected 
? ? Several MATLAB ODE solvers have been connected 
? ? New ODE solvers, parameters abstol and reltol 
? ? New GLP solver: direct 
? ? Some minor bugfixes and improvements 

Regards, D. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131215/b105e861/attachment.html>

From aspire.janani at gmail.com  Sun Dec 15 21:03:32 2013
From: aspire.janani at gmail.com (janani padmanabhan)
Date: Mon, 16 Dec 2013 07:33:32 +0530
Subject: [Numpy-discussion] request for help
Message-ID: <CAPB4OAV2SP_qNDugfSZ-MzNfV83GLooBveDd3CkFH5hTotPppA@mail.gmail.com>

Hello
Hearty greetings
I am a novice who is extremely interested in contributing to NumPy
opensource community. How and where do I start?
Thanking you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131216/3eaf59dd/attachment.html>

From david.jones74 at gmail.com  Mon Dec 16 17:25:32 2013
From: david.jones74 at gmail.com (David Jones)
Date: Mon, 16 Dec 2013 17:25:32 -0500
Subject: [Numpy-discussion] building 32 bit numpy on 64 bit linux
In-Reply-To: <52AB5684.5050907@googlemail.com>
References: <52AA075F.8060708@gmail.com>	<52AA2276.8020007@googlemail.com>	<52AA1116.6080809@gmail.com>	<52AA2C0E.2090600@googlemail.com>	<52AB137B.3000708@gmail.com>	<CADU3R7gpPbhTuqty5Fy-y5CsKOc88+_YOHhNP2rJ5zSbQVgq8A@mail.gmail.com>	<52AB47F1.2040802@gmail.com>
	<52AB5684.5050907@googlemail.com>
Message-ID: <52AF7DDC.4020706@gmail.com>


On 12/13/13 13:48, Julian Taylor wrote:
> On 13.12.2013 18:46, David Jones wrote:
>> ...
>>
>> Correction. Of course LD_LIBRARY_PATH isn't seen by the compiler. It
>> only applies at run time. How embarrasing:) This isn't the first time
>> I've been bitten by that.
>>
>> I don't mind doing that with manual builds, but what about with pip? Is
>> there a way to avoid explicitly setting the library path every time you
>> call pip, when using a custom python install?
>>
> why are you actually doing this?
>
> the easiest way to get 32 bit binaries for any program is simply using a
> 32 bit chroot to build them.
> This is trivial to do on a debian based systems:
>
> pbuilder-dist i386 unstable create
> pbuilder-dist i386 unstable login
> # install dependencies and build as usual
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Just a follow-up. After building python with the -rpath linker flag set, 
I can install numpy using pip without having to specify the library path 
in LDFLAGS.

CC=gcc-32 LDSHARED=ld-32 pip install numpy

However, this doesn't work for building numpy directly. So it seems that 
pip is doing some clever magic here.

In addition, pip perpetuates the -rpath value from python when building 
packages, so they have the correct run-time path to libpython.


From pierre.haessig at crans.org  Tue Dec 17 10:41:37 2013
From: pierre.haessig at crans.org (Pierre Haessig)
Date: Tue, 17 Dec 2013 16:41:37 +0100
Subject: [Numpy-discussion] repeat array in a fake dim without
 stride_tricks ?
In-Reply-To: <1386938737.2127.1.camel@sebastian-laptop>
References: <52A9C615.1010909@crans.org>
	<1386938737.2127.1.camel@sebastian-laptop>
Message-ID: <52B070B1.60209@crans.org>

Le 13/12/2013 13:45, Sebastian Berg a ?crit :
>> What are the other options for such a repeat ?
> No, I don't think there are any other options. stride tricks are a bit
> hidden, since in many cases it is more dangerous than helping.
> Though with some care you can easily implement such functions using
> stride_tricks.
Thanks !
Since it's in a buried helper function, I could use stride tricks. But
for now I'll stick with broadcast_arrays because I have it working.

Maybe the np.repeat function could be changed to avoid copying (when
possible) ?

best,
Pierre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pierre_haessig.vcf
Type: text/x-vcard
Size: 329 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131217/4fe8a50f/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 897 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131217/4fe8a50f/attachment.sig>

From sebastian at sipsolutions.net  Wed Dec 18 07:20:26 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 18 Dec 2013 13:20:26 +0100
Subject: [Numpy-discussion] repeat array in a fake dim without
 stride_tricks ?
In-Reply-To: <52B070B1.60209@crans.org>
References: <52A9C615.1010909@crans.org>
	<1386938737.2127.1.camel@sebastian-laptop> <52B070B1.60209@crans.org>
Message-ID: <1387369226.10610.6.camel@sebastian-laptop>

On Tue, 2013-12-17 at 16:41 +0100, Pierre Haessig wrote:
> Le 13/12/2013 13:45, Sebastian Berg a ?crit :
> >> What are the other options for such a repeat ?
> > No, I don't think there are any other options. stride tricks are a bit
> > hidden, since in many cases it is more dangerous than helping.
> > Though with some care you can easily implement such functions using
> > stride_tricks.
> Thanks !
> Since it's in a buried helper function, I could use stride tricks. But
> for now I'll stick with broadcast_arrays because I have it working.
> 
> Maybe the np.repeat function could be changed to avoid copying (when
> possible) ?
> 

I doubt it fits repeat well, since repeat also allows for repetition of
single elements. The repeats that could be done like this are a few of
the many ways repeat can be used. What may make sense is to add one or
two thought out functions for common `as_strided` use cases, which
always return valid arrays...

- Sebastian

> best,
> Pierre
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From alan.isaac at gmail.com  Wed Dec 18 17:23:01 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Wed, 18 Dec 2013 17:23:01 -0500
Subject: [Numpy-discussion] polyfit
Message-ID: <52B22045.4080901@gmail.com>

For teaching it is certainly nice to have numpy.polynomial.polynomial.polyfit
providing modern (vs. traditional) parameter order, but

- it is rather buried
- np.polyfit uses traditional order and has the same name

I recall there was some controversy (?) over all of this,
but might it not be appropriate to have a keyword argument to
both specifying whether the parameter order is to be modern
or traditional (in both polyfits and polyvals)?

fwiw,
Alan Isaac


From charlesr.harris at gmail.com  Wed Dec 18 17:38:13 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Dec 2013 15:38:13 -0700
Subject: [Numpy-discussion] polyfit
In-Reply-To: <52B22045.4080901@gmail.com>
References: <52B22045.4080901@gmail.com>
Message-ID: <CAB6mnxKedExWMAt+ePXmrEM_qSz25QUk-V=Aq=Bywu41MmmG7Q@mail.gmail.com>

On Wed, Dec 18, 2013 at 3:23 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> For teaching it is certainly nice to have
> numpy.polynomial.polynomial.polyfit
> providing modern (vs. traditional) parameter order, but
>
> - it is rather buried
> - np.polyfit uses traditional order and has the same name
>
> I recall there was some controversy (?) over all of this,
> but might it not be appropriate to have a keyword argument to
> both specifying whether the parameter order is to be modern
> or traditional (in both polyfits and polyvals)?
>
> It would be messy, as there are a lot of functions in
polynomial.polynomial that depend on the coefficient order. For the higher
level Polynomial class, there is a routine in ipython that will do a pretty
display of Polynomial instances which might be useful. Let's see... look
here
http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/Custom%20Display%20Logic.ipynband
search the page for Polynomial.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131218/fa6a059d/attachment.html>

From charlesr.harris at gmail.com  Wed Dec 18 17:51:40 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Dec 2013 15:51:40 -0700
Subject: [Numpy-discussion] polyfit
In-Reply-To: <CAB6mnxKedExWMAt+ePXmrEM_qSz25QUk-V=Aq=Bywu41MmmG7Q@mail.gmail.com>
References: <52B22045.4080901@gmail.com>
	<CAB6mnxKedExWMAt+ePXmrEM_qSz25QUk-V=Aq=Bywu41MmmG7Q@mail.gmail.com>
Message-ID: <CAB6mnxKEQC+Zbox+WGmY=X-2dJ9cAfdQhnN3Td_tbs6qHnQ5vw@mail.gmail.com>

On Wed, Dec 18, 2013 at 3:38 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
>
> On Wed, Dec 18, 2013 at 3:23 PM, Alan G Isaac <alan.isaac at gmail.com>wrote:
>
>> For teaching it is certainly nice to have
>> numpy.polynomial.polynomial.polyfit
>> providing modern (vs. traditional) parameter order, but
>>
>> - it is rather buried
>> - np.polyfit uses traditional order and has the same name
>>
>> I recall there was some controversy (?) over all of this,
>> but might it not be appropriate to have a keyword argument to
>> both specifying whether the parameter order is to be modern
>> or traditional (in both polyfits and polyvals)?
>>
>> It would be messy, as there are a lot of functions in
> polynomial.polynomial that depend on the coefficient order. For the higher
> level Polynomial class, there is a routine in ipython that will do a pretty
> display of Polynomial instances which might be useful. Let's see... look
> here
> http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/Custom%20Display%20Logic.ipynband search the page for Polynomial.
>
>
Although beginning students might find the domain/window concept used in
Polynomial confusing... Strictly speaking, the polynomial should be
rendered as powers of ((x - off)/scale) if the domain differs from the
window.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131218/a3071702/attachment.html>

From ethanbuchman at hotmail.com  Wed Dec 18 21:17:11 2013
From: ethanbuchman at hotmail.com (ebuchman)
Date: Wed, 18 Dec 2013 18:17:11 -0800 (PST)
Subject: [Numpy-discussion] numpy.dot causes segfault after ctypes call to
	cudaMalloc
Message-ID: <1387419431447-35910.post@n7.nabble.com>

I'm learning cuda and decided to use python with ctypes to call all the cuda
functions but I'm finding some memory issues.  I've boiled it down to the
simplest scenario.  I use ctypes to call a cuda function which allocates
memory on the device and then frees it.  This works fine, but if I then try
to use np.dot on a totally other array declared in python, I get a
segmentation fault.  Note this only happens if the numpy array is
sufficiently large.  If I change the cuda mallocs to simple c mallocs, all
the problems go away, but thats not really helpful.  Any ideas what's going
on here?

CUDA CODE (debug.cu): 

#include <stdio.h>
#include <stdlib.h>

extern "C" {
void all_together( size_t N)
{
    void*d;
    int size = N *sizeof(float);
    int err;

    err = cudaMalloc(&d, size);
    if (err != 0) printf("cuda malloc error: %d\n", err);

    err = cudaFree(d);
    if (err != 0) printf("cuda free error: %d\n", err);
}}

PYTHON CODE (master.py):

import numpy as np
import ctypes
from ctypes import *

dll = ctypes.CDLL('./cuda_lib.so', mode=ctypes.RTLD_GLOBAL)

def build_all_together_f(dll):
    func = dll.all_together
    func.argtypes = [c_size_t]
    return func

__pycu_all_together = build_all_together_f(dll)


if __name__ == '__main__':
    N = 5001 # if this is less, the error doesn't show up

    a = np.random.randn(N).astype('float32')

    da = __pycu_all_together(N)

    # toggle this line on/off to get error
    #np.dot(a, a)

    print 'end of python'

COMPILE: nvcc -Xcompiler -fPIC -shared -o cuda_lib.so debug.cu

RUN: python master.py



--
View this message in context: http://numpy-discussion.10968.n7.nabble.com/numpy-dot-causes-segfault-after-ctypes-call-to-cudaMalloc-tp35910.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.


From andrei at ruivo.org  Thu Dec 19 07:51:05 2013
From: andrei at ruivo.org (rootspin)
Date: Thu, 19 Dec 2013 04:51:05 -0800 (PST)
Subject: [Numpy-discussion] Array search considering order
Message-ID: <1387457465139-35911.post@n7.nabble.com>

Hello,

Need some help in searching arrays (Im new to numpy)
Is it possible to search a array, using another array considering
order/sequence? 

x = np.array([1,2,3,4,5,6], np.int32)

y = np.array([1,4,3,2,6,5], np.int32)

query= np.array([1,2,3],np.int32)

x versus query True
y versus query False


Tried with: 

np.searchsorted(x,query)  -------> array([0, 1, 2])
np.searchsorted(y,query)  -------> array([0, 1, 4])

Thanks




--
View this message in context: http://numpy-discussion.10968.n7.nabble.com/Array-search-considering-order-tp35911.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.


From oscar.j.benjamin at gmail.com  Thu Dec 19 08:51:30 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Thu, 19 Dec 2013 13:51:30 +0000
Subject: [Numpy-discussion] Array search considering order
In-Reply-To: <1387457465139-35911.post@n7.nabble.com>
References: <1387457465139-35911.post@n7.nabble.com>
Message-ID: <CAHVvXxTi8r9BLp-ZitKNSPcotf6pb6bU72tWaXaVbqXgM5K5WQ@mail.gmail.com>

On 19 December 2013 12:51, rootspin <andrei at ruivo.org> wrote:
> Hello,
>
> Need some help in searching arrays (Im new to numpy)
> Is it possible to search a array, using another array considering
> order/sequence?
>
> x = np.array([1,2,3,4,5,6], np.int32)
>
> y = np.array([1,4,3,2,6,5], np.int32)
>
> query= np.array([1,2,3],np.int32)
>
> x versus query True
> y versus query False
>
>
> Tried with:
>
> np.searchsorted(x,query)  -------> array([0, 1, 2])
> np.searchsorted(y,query)  -------> array([0, 1, 4])

I'm not sure if I understand your problem do you mean that you want to
find subarrays of an array the same way that e.g. "in" for strings
tests for substrings:

>>> 'asd' in 'qweasdzxc'
True

If so then perhaps this SO question has the answer you want:
http://stackoverflow.com/questions/7100242/python-numpy-first-occurrence-of-subarray


Oscar


From andrei at ruivo.org  Thu Dec 19 11:49:38 2013
From: andrei at ruivo.org (Andrei Rozanski)
Date: Thu, 19 Dec 2013 11:49:38 -0500
Subject: [Numpy-discussion] Array search considering order
In-Reply-To: <CAHVvXxTi8r9BLp-ZitKNSPcotf6pb6bU72tWaXaVbqXgM5K5WQ@mail.gmail.com>
References: <1387457465139-35911.post@n7.nabble.com>
	<CAHVvXxTi8r9BLp-ZitKNSPcotf6pb6bU72tWaXaVbqXgM5K5WQ@mail.gmail.com>
Message-ID: <20131219164937.GH25832@jake.ruivo.org>

On Thu, Dec 19, 2013 at 01:51:30PM +0000, Oscar Benjamin wrote:
> Date: Thu, 19 Dec 2013 13:51:30 +0000
> From: Oscar Benjamin <oscar.j.benjamin at gmail.com>
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Subject: Re: [Numpy-discussion] Array search considering order
> 
> On 19 December 2013 12:51, rootspin <andrei at ruivo.org> wrote:
> > Hello,
> >
> > Need some help in searching arrays (Im new to numpy)
> > Is it possible to search a array, using another array considering
> > order/sequence?
> >
> > x = np.array([1,2,3,4,5,6], np.int32)
> >
> > y = np.array([1,4,3,2,6,5], np.int32)
> >
> > query= np.array([1,2,3],np.int32)
> >
> > x versus query True
> > y versus query False
> >
> >
> > Tried with:
> >
> > np.searchsorted(x,query)  -------> array([0, 1, 2])
> > np.searchsorted(y,query)  -------> array([0, 1, 4])
> 
> I'm not sure if I understand your problem do you mean that you want to
> find subarrays of an array the same way that e.g. "in" for strings
> tests for substrings:
> 
> >>> 'asd' in 'qweasdzxc'
> True
> 
> If so then perhaps this SO question has the answer you want:
> http://stackoverflow.com/questions/7100242/python-numpy-first-occurrence-of-subarray
> 
> 
> Oscar
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Thank you Oscar for your answer.

Sorry if I was not clear enough.
The SO question is alike what I want. However, in my problem, Im not sure if there will be only one occurence.
What I do expect is: 
Given one array (big one), to retrieve indexes for "query array" occurences.
Something like that:

array1(big one) - 1,2,3,4,5,6,7,1,2,3,4,5,7,8,9,1,2,3,4,5,6,7,8
array2(query) - 1,2,3
array3(another query) - 8,7,6

result (array1 versus array2) - 0,7,15 (position for match)
result (array1 versus array3) - no matches

Thanks for the help.







From sebastian at sipsolutions.net  Thu Dec 19 12:26:02 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 19 Dec 2013 18:26:02 +0100
Subject: [Numpy-discussion] Scalar result for field access -- Fix in minor
	release?
Message-ID: <1387473962.3386.10.camel@sebastian-laptop>

Hey,

fixing a corner case indexing regression in 1.8, I noticed/fixed
accidentally this behavior of returning a scalar when indexing a 0-d
array with fields (see also [1]):

arr = np.array((1,), dtype=[('a', 'f8')])

arr['a']   # Returns an array
arr[['a']] # Currently returns a *scalar*

I think no field access should try to convert 0-d arrays to scalars, and
it was probably just an oversight. However, if anyone thinks there is
even the slightest chance that this creates bugs in production code, we
certainly should not change it in a bug-fix only release.

Or am I missing something and the old behavior was actually intended?

- Sebastian


[1] https://github.com/numpy/numpy/issues/4109



From warren.weckesser at gmail.com  Fri Dec 20 00:16:14 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Fri, 20 Dec 2013 00:16:14 -0500
Subject: [Numpy-discussion] git tag for version 1.8?
Message-ID: <CAGzF1ue9CbuKhaSsVcxpD7xY0Q0CkV8FVHF0Pken5HhK+Xr4ww@mail.gmail.com>

Is version 1.8.0 tagged in git?  I see tags up to 1.7.1.  I suspect
the tagging convention has changed in the git repo.  How do I checkout
v1.8.0?

Warren


From siegfried.gonzi at ed.ac.uk  Fri Dec 20 03:26:22 2013
From: siegfried.gonzi at ed.ac.uk (Siegfried Gonzi)
Date: Fri, 20 Dec 2013 08:26:22 +0000
Subject: [Numpy-discussion] Need help with np.ma.median and
	np.apply_along_axis
Message-ID: <20131220082622.13573qinn1ii0484@www.staffmail.ed.ac.uk>

Please have a look at version1 and version2. What are my other options  
here? Do I need to go the cython route here? Thanks, Siegfried

==
My array is as follows (shown here for dummy values; and yes this kind  
of arrays do exist: 150 observations x 8 years x 366 days x 24 hours x  
7 model levels):

data = np.random.random((150,8,366,24,7))


My function "np.apply_along_axis(my_moment,4,data)" takes 15 minutes.

I thought making use of masked arrays "my_moment_fast(data,axis=4)"  
will speed up things but

1. It blows up the memory consumption to 6 GB at times

and

2. It also takes... I do not know as I killed it after 20 minutes (it  
hangs at the median print statement).

The calculation of the median is the bottleneck here.

==
import numpy as np


def my_moment(data_in,nan_val=-999.0):


     tmp = data_in[np.where(data_in<>nan_val)]

     erg = np.array([np.min(tmp),np.mean(tmp),np.median(tmp),\
                     np.max(tmp),np.std(tmp),np.size(tmp)])

     return erg


def my_moment_fast(data_in,nan_val=-999.0,axis=4):

     import numpy as np

     print 'min max',np.min(data_in),np.max(data_in)

     mx = np.ma.masked_where((data_in<=0.0)&(data_in<=nan_val), data_in)

     print 'shape mx',np.shape(mx),np.min(mx),np.max(mx)

     print 'min'
     tmp_min = np.ma.min(mx,axis=axis)
     print 'max'
     tmp_max = np.ma.max(mx,axis=axis)
     print 'mean'
     tmp_mean = np.ma.mean(mx,axis=axis)
     print 'median'
     #tmp_median = np.ma.sort(mx,axis=axis)
     tmp_median = np.ma.median(mx,axis=axis)
     print 'std'
     tmp_std = np.ma.std(mx,axis=axis)
     print 'N'
     tmp_N = np.ones(np.shape(mx))
     tmp_N[mx.mask] = 0.0e0
     tmp_N = np.ma.sum(tmp_N,axis=axis)

     print 'shape min',np.shape(tmp_min),np.min(tmp_min),np.max(tmp_min)
     print 'shape max',np.shape(tmp_max),np.min(tmp_max),np.max(tmp_min)
     print 'shape mean',np.shape(tmp_mean),np.min(tmp_mean),np.max(tmp_mean)
     print 'shape median', np.shape(tmp_median), np.min(tmp_median),  
np.max(tmp_median)
     print 'shape std',np.shape(tmp_std),np.min(tmp_std),np.max(tmp_std)
     print 'shape N', np.shape(tmp_N), np.min(tmp_N), np.max(tmp_N),  
np.shape(mx.mask)

     return np.array([tmp_min,tmp_mean,tmp_median,tmp_max,tmp_std,tmp_N])



data = np.random.random((150,8,366,24,7))
data[134,5,300,:,2] = -999.0
data[14,3,300,:,0] = -999.0

version1 = my_moment_fast(data,axis=4)
exit()
version2 = np.apply_along_axis(my_moment,4,data)
==

What  am I doing wrong here? I haven't tested it againts Fortran and  
have got no idea if sorting for fetching the median would be faster.

Thanks,
Siegfried
==

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




From oscar.j.benjamin at gmail.com  Fri Dec 20 06:19:49 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 20 Dec 2013 11:19:49 +0000
Subject: [Numpy-discussion] Array search considering order
In-Reply-To: <20131219164937.GH25832@jake.ruivo.org>
References: <1387457465139-35911.post@n7.nabble.com>
	<CAHVvXxTi8r9BLp-ZitKNSPcotf6pb6bU72tWaXaVbqXgM5K5WQ@mail.gmail.com>
	<20131219164937.GH25832@jake.ruivo.org>
Message-ID: <CAHVvXxTE-VkdAzQesbegtN6KNyAycz_xq8SHyynevJAmO9e7MA@mail.gmail.com>

On 19 December 2013 16:49, Andrei Rozanski <andrei at ruivo.org> wrote:
>
> Sorry if I was not clear enough.
> The SO question is alike what I want. However, in my problem, Im not sure if there will be only one occurence.
> What I do expect is:
> Given one array (big one), to retrieve indexes for "query array" occurences.
> Something like that:
>
> array1(big one) - 1,2,3,4,5,6,7,1,2,3,4,5,7,8,9,1,2,3,4,5,6,7,8
> array2(query) - 1,2,3
> array3(another query) - 8,7,6
>
> result (array1 versus array2) - 0,7,15 (position for match)
> result (array1 versus array3) - no matches

If you look more closely at the SO question you'll see that it does
answer your problem. The top-rated answer shows how to get an boolean
array indicating where the matches are. In your case that would be

[True, False, False, False, False, False, False, True, False, False,
False, False, False, False, False, True, False, False, False, False,
False]

Now just use the numpy.where function to get the indices of the True
value from this array.


Oscar


From andrei at ruivo.org  Fri Dec 20 06:41:04 2013
From: andrei at ruivo.org (Andrei Rozanski)
Date: Fri, 20 Dec 2013 06:41:04 -0500
Subject: [Numpy-discussion] Array search considering order
In-Reply-To: <CAHVvXxTE-VkdAzQesbegtN6KNyAycz_xq8SHyynevJAmO9e7MA@mail.gmail.com>
References: <1387457465139-35911.post@n7.nabble.com>
	<CAHVvXxTi8r9BLp-ZitKNSPcotf6pb6bU72tWaXaVbqXgM5K5WQ@mail.gmail.com>
	<20131219164937.GH25832@jake.ruivo.org>
	<CAHVvXxTE-VkdAzQesbegtN6KNyAycz_xq8SHyynevJAmO9e7MA@mail.gmail.com>
Message-ID: <20131220114104.GI25832@jake.ruivo.org>

On Fri, Dec 20, 2013 at 11:19:49AM +0000, Oscar Benjamin wrote:
> Date: Fri, 20 Dec 2013 11:19:49 +0000
> From: Oscar Benjamin <oscar.j.benjamin at gmail.com>
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Subject: Re: [Numpy-discussion] Array search considering order
> 
> On 19 December 2013 16:49, Andrei Rozanski <andrei at ruivo.org> wrote:
> >
> > Sorry if I was not clear enough.
> > The SO question is alike what I want. However, in my problem, Im not sure if there will be only one occurence.
> > What I do expect is:
> > Given one array (big one), to retrieve indexes for "query array" occurences.
> > Something like that:
> >
> > array1(big one) - 1,2,3,4,5,6,7,1,2,3,4,5,7,8,9,1,2,3,4,5,6,7,8
> > array2(query) - 1,2,3
> > array3(another query) - 8,7,6
> >
> > result (array1 versus array2) - 0,7,15 (position for match)
> > result (array1 versus array3) - no matches
> 
> If you look more closely at the SO question you'll see that it does
> answer your problem. The top-rated answer shows how to get an boolean
> array indicating where the matches are. In your case that would be
> 
> [True, False, False, False, False, False, False, True, False, False,
> False, False, False, False, False, True, False, False, False, False,
> False]
> 
> Now just use the numpy.where function to get the indices of the True
> value from this array.
> 
> 
> Oscar
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi Oscar,

Thank you for the help.
Indeed. My lack of knowledge on numpy functions didnt aloud my to get that.

That works for me.

best regards.





From charlesr.harris at gmail.com  Fri Dec 20 07:36:29 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 Dec 2013 05:36:29 -0700
Subject: [Numpy-discussion] git tag for version 1.8?
In-Reply-To: <CAGzF1ue9CbuKhaSsVcxpD7xY0Q0CkV8FVHF0Pken5HhK+Xr4ww@mail.gmail.com>
References: <CAGzF1ue9CbuKhaSsVcxpD7xY0Q0CkV8FVHF0Pken5HhK+Xr4ww@mail.gmail.com>
Message-ID: <CAB6mnxJ8S-Vn2+qmszYwuw+f372aZ03NDRXQ7prX0X++AeQKzQ@mail.gmail.com>

On Thu, Dec 19, 2013 at 10:16 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

> Is version 1.8.0 tagged in git?  I see tags up to 1.7.1.  I suspect
> the tagging convention has changed in the git repo.  How do I checkout
> v1.8.0?
>
>
It's tagged. You can see it on github under branches/tags if you hit the
tags tab. To get tags from upstream do `git fetch upstream --tags`

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131220/6174e0bf/attachment.html>

From warren.weckesser at gmail.com  Fri Dec 20 09:29:53 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Fri, 20 Dec 2013 09:29:53 -0500
Subject: [Numpy-discussion] git tag for version 1.8?
In-Reply-To: <CAB6mnxJ8S-Vn2+qmszYwuw+f372aZ03NDRXQ7prX0X++AeQKzQ@mail.gmail.com>
References: <CAGzF1ue9CbuKhaSsVcxpD7xY0Q0CkV8FVHF0Pken5HhK+Xr4ww@mail.gmail.com>
	<CAB6mnxJ8S-Vn2+qmszYwuw+f372aZ03NDRXQ7prX0X++AeQKzQ@mail.gmail.com>
Message-ID: <CAGzF1udVFRY6DAQ0xCmawPjU-Pk9tvWD0YRSJDb-WFmh6a2Abg@mail.gmail.com>

On 12/20/13, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On Thu, Dec 19, 2013 at 10:16 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> Is version 1.8.0 tagged in git?  I see tags up to 1.7.1.  I suspect
>> the tagging convention has changed in the git repo.  How do I checkout
>> v1.8.0?
>>
>>
> It's tagged. You can see it on github under branches/tags if you hit the
> tags tab. To get tags from upstream do `git fetch upstream --tags`
>


Great, thanks.

Warren

> Chuck
>


From ralf.gommers at gmail.com  Fri Dec 20 14:54:12 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 20 Dec 2013 20:54:12 +0100
Subject: [Numpy-discussion] request for help
In-Reply-To: <CAPB4OAV2SP_qNDugfSZ-MzNfV83GLooBveDd3CkFH5hTotPppA@mail.gmail.com>
References: <CAPB4OAV2SP_qNDugfSZ-MzNfV83GLooBveDd3CkFH5hTotPppA@mail.gmail.com>
Message-ID: <CABL7CQiSn__QtyebWAAjTG3_4axY7jbafY_i8haWbaNuO8FEsQ@mail.gmail.com>

On Mon, Dec 16, 2013 at 3:03 AM, janani padmanabhan <aspire.janani at gmail.com
> wrote:

> Hello
> Hearty greetings
> I am a novice who is extremely interested in contributing to NumPy
> opensource community. How and where do I start?
> Thanking you
>

Hi Janani, welcome!

Here is a description on the mechanics on contributing (setting up a dev
environment, using git/Github, etc.):
http://docs.scipy.org/doc/numpy-dev/dev/. And this text gives some more
background on what kind of things you can contribute and what's expected in
terms of code quality, testing etc. (for scipy, but applies also to numpy):
https://github.com/scipy/scipy/blob/master/HACKING.rst.txt.

Here are some issues labeled easy-fix, if you'd like to try out the
workflow on something relatively straightforward to start with:
https://github.com/numpy/numpy/issues?labels=Easy+Fix&page=1&state=open

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131220/940beb26/attachment.html>

From opensource.numpy at user.fastmail.fm  Sat Dec 21 16:16:11 2013
From: opensource.numpy at user.fastmail.fm (Hugo Gagnon)
Date: Sat, 21 Dec 2013 16:16:11 -0500
Subject: [Numpy-discussion] Segmentation fault with pickled numpy float64
	arrays since 1.8
Message-ID: <28825B47-2CC7-41DB-A6D8-614098F5D803@user.fastmail.fm>

Hi,

Since I've updated numpy from 1.7 to 1.8 with EPD I get segmentation faults whenever I load back pickled float64 arrays.  Here's a minimal example:

"""

import numpy
import cPickle

a = numpy.arange(5, dtype='float64')

with open('test.p', 'wb') as fh:
    cPickle.dump(a, fh)

with open('test.p') as fh:
    a2 = cPickle.load(fh)

print a2

"""

However the above works fine with int32 arrays, i.e. with a = numpy.arange(5).

Does anyone else experience this problem?

Thanks,
--
  Hugo Gagnon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131221/4a6beb30/attachment.html>

From charlesr.harris at gmail.com  Sat Dec 21 17:20:06 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 21 Dec 2013 15:20:06 -0700
Subject: [Numpy-discussion] Segmentation fault with pickled numpy
 float64 arrays since 1.8
In-Reply-To: <28825B47-2CC7-41DB-A6D8-614098F5D803@user.fastmail.fm>
References: <28825B47-2CC7-41DB-A6D8-614098F5D803@user.fastmail.fm>
Message-ID: <CAB6mnxKJ=KRPJzE=XZ=bVZjtEvqm-SY65YWweYXz04GGdXf4nA@mail.gmail.com>

On Sat, Dec 21, 2013 at 2:16 PM, Hugo Gagnon <
opensource.numpy at user.fastmail.fm> wrote:

> Hi,
>
> Since I've updated numpy from 1.7 to 1.8 with EPD I get segmentation
> faults whenever I load back pickled float64 arrays.  Here's a minimal
> example:
>
> """
>
> import numpy
> import cPickle
>
> a = numpy.arange(5, dtype='float64')
>
> with open('test.p', 'wb') as fh:
>     cPickle.dump(a, fh)
>
> with open('test.p') as fh:
>     a2 = cPickle.load(fh)
>
> print a2
>
> """
>
> However the above works fine with int32 arrays, i.e. with a =
> numpy.arange(5).
>
> Does anyone else experience this problem?
>
>
I don't see that here on 64 bit fedora 20, numpy-devel. What OS are you
running?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131221/2117a086/attachment.html>

From opensource.numpy at user.fastmail.fm  Sat Dec 21 17:38:39 2013
From: opensource.numpy at user.fastmail.fm (Hugo Gagnon)
Date: Sat, 21 Dec 2013 17:38:39 -0500
Subject: [Numpy-discussion] Segmentation fault with pickled numpy
	float64 arrays since 1.8
In-Reply-To: <CAB6mnxKJ=KRPJzE=XZ=bVZjtEvqm-SY65YWweYXz04GGdXf4nA@mail.gmail.com>
References: <28825B47-2CC7-41DB-A6D8-614098F5D803@user.fastmail.fm>
	<CAB6mnxKJ=KRPJzE=XZ=bVZjtEvqm-SY65YWweYXz04GGdXf4nA@mail.gmail.com>
Message-ID: <6133D4BE-65FF-49A0-9BB1-CDAEB1F2E8BD@user.fastmail.fm>

Mac OS 10.8.5

--
  Hugo Gagnon

On 2013-12-21, at 5:20 PM, Charles R Harris <charlesr.harris at gmail.com> wrote:

> 
> 
> 
> On Sat, Dec 21, 2013 at 2:16 PM, Hugo Gagnon <opensource.numpy at user.fastmail.fm> wrote:
> Hi,
> 
> Since I've updated numpy from 1.7 to 1.8 with EPD I get segmentation faults whenever I load back pickled float64 arrays.  Here's a minimal example:
> 
> """
> 
> import numpy
> import cPickle
> 
> a = numpy.arange(5, dtype='float64')
> 
> with open('test.p', 'wb') as fh:
>     cPickle.dump(a, fh)
> 
> with open('test.p') as fh:
>     a2 = cPickle.load(fh)
> 
> print a2
> 
> """
> 
> However the above works fine with int32 arrays, i.e. with a = numpy.arange(5).
> 
> Does anyone else experience this problem?
> 
> 
> I don't see that here on 64 bit fedora 20, numpy-devel. What OS are you running?
> 
> Chuck 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131221/4a78d0ab/attachment.html>

From jrocher at enthought.com  Sun Dec 22 15:34:29 2013
From: jrocher at enthought.com (Jonathan Rocher)
Date: Sun, 22 Dec 2013 14:34:29 -0600
Subject: [Numpy-discussion] Segmentation fault with pickled numpy
 float64 arrays since 1.8
In-Reply-To: <6133D4BE-65FF-49A0-9BB1-CDAEB1F2E8BD@user.fastmail.fm>
References: <28825B47-2CC7-41DB-A6D8-614098F5D803@user.fastmail.fm>
	<CAB6mnxKJ=KRPJzE=XZ=bVZjtEvqm-SY65YWweYXz04GGdXf4nA@mail.gmail.com>
	<6133D4BE-65FF-49A0-9BB1-CDAEB1F2E8BD@user.fastmail.fm>
Message-ID: <CAOzk5QekU3JHLQ6tA1UjrX_UBR2or+d01ooX6sveLwpcGGiKgg@mail.gmail.com>

I am not observing this with Canopy on MacOS 10.8.5 (64bit). I have numpy
1.8.0 and cPickle 1.71.

My guess is that it is a problem with your setup. To figure out what is
going on, I would have more questions for you so please email me privately
a good email address to reach you at.

Best,
Jonathan


On Sat, Dec 21, 2013 at 2:38 PM, Hugo Gagnon <
opensource.numpy at user.fastmail.fm> wrote:

> Mac OS 10.8.5
>
>  --
>   Hugo Gagnon
>
> On 2013-12-21, at 5:20 PM, Charles R Harris <charlesr.harris at gmail.com>
> wrote:
>
>
>
>
> On Sat, Dec 21, 2013 at 2:16 PM, Hugo Gagnon <
> opensource.numpy at user.fastmail.fm> wrote:
>
>> Hi,
>>
>> Since I've updated numpy from 1.7 to 1.8 with EPD I get segmentation
>> faults whenever I load back pickled float64 arrays.  Here's a minimal
>> example:
>>
>> """
>>
>> import numpy
>> import cPickle
>>
>> a = numpy.arange(5, dtype='float64')
>>
>> with open('test.p', 'wb') as fh:
>>     cPickle.dump(a, fh)
>>
>> with open('test.p') as fh:
>>     a2 = cPickle.load(fh)
>>
>> print a2
>>
>> """
>>
>> However the above works fine with int32 arrays, i.e. with a =
>> numpy.arange(5).
>>
>> Does anyone else experience this problem?
>>
>>
> I don't see that here on 64 bit fedora 20, numpy-devel. What OS are you
> running?
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Jonathan Rocher, PhD
Scientific software developer
Enthought, Inc.
jrocher at enthought.com
1-512-536-1057
http://www.enthought.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131222/a03f3431/attachment.html>

From matti.picus at gmail.com  Sun Dec 22 18:14:03 2013
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 23 Dec 2013 01:14:03 +0200
Subject: [Numpy-discussion] C99 compatible complex number tests fail
Message-ID: <52B7723B.7080900@gmail.com>

Hi. I started to port the stdlib cmath C99 compatible complex number 
tests to numpy, after noticing that numpy seems to have different 
complex number routines than cmath. The work is available on a 
"retest_complex" branch of numpy
https://github.com/mattip/numpy/tree/retest_complex
The tests can be run by pulling the branch (no need to rebuild numpy) 
and running

python <path-to-branch>/numpy/core/tests/test_umath_complex.py > 
test.log 2>&1

So far it is just a couple of  commits that run the tests on numpy, I 
did not dive into modifying the math routines. If I did the work 
correctly, failures point to some differences, most due to edge cases 
with inf and nan, but there are a number of failures due to different 
finite values (for some small definition of different).
I guess my first question is "did I do the tests properly".
Assuming I did, the next question is "are the inconsistencies 
intentional" i.e. are they that way in order to be compatible with 
Matlab  or some other non-C99 conformant library?

For instance, a comparison between the implementation of cmath's sqrt 
and numpy's sqrt shows that numpy does not check for subnormals. And I 
am probably mistaken since I am new to the generator methods of numpy, 
but could it be that trigonometric functions like acos and acosh are 
generated in umath/funcs.inc.src, using a very different algorithm than 
cmathmodule.c?

Would there be interest in a pull request that changed the routines to 
be more compatible with results from cmath?
Matti



From aspire.janani at gmail.com  Tue Dec 24 01:35:49 2013
From: aspire.janani at gmail.com (janani padmanabhan)
Date: Tue, 24 Dec 2013 12:05:49 +0530
Subject: [Numpy-discussion] Regression test
Message-ID: <CAPB4OAWO9sot4SLwo=tUEP3r1FDbqbmnBgAACWrkpWj0Gww92w@mail.gmail.com>

Hello folks!
I am a newbie, and I want to know how to add a regression test.
Intuitively, I simply executed the corresponding test.py and got errors,
even before having made any changes in the source code. How do I proceed?
Thanking you in anticipation
Janani
(jennystone)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131224/c65eb02b/attachment.html>

From ralf.gommers at gmail.com  Tue Dec 24 05:02:27 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 24 Dec 2013 11:02:27 +0100
Subject: [Numpy-discussion] Regression test
In-Reply-To: <CAPB4OAWO9sot4SLwo=tUEP3r1FDbqbmnBgAACWrkpWj0Gww92w@mail.gmail.com>
References: <CAPB4OAWO9sot4SLwo=tUEP3r1FDbqbmnBgAACWrkpWj0Gww92w@mail.gmail.com>
Message-ID: <CABL7CQhQhfdZyCBk9JC+qWBdaY+xnuRPuh+gUxnHAs+anEm+Dw@mail.gmail.com>

On Tue, Dec 24, 2013 at 7:35 AM, janani padmanabhan <aspire.janani at gmail.com
> wrote:

> Hello folks!
> I am a newbie, and I want to know how to add a regression test.
> Intuitively, I simply executed the corresponding test.py and got errors,
> even before having made any changes in the source code. How do I proceed?
>

You run the numpy test suite by:

    import numpy as np
    np.test()  # has some options, like 'full' and verbose=2

If you're developing it may also be helpful to use:

    python runtests.py

This builds numpy and runs the tests at once. If you change something in
the git repo and re-run this command, it's very fast due to only rebuilding
the part that changed.

After you can run the test, you can add your new test by following the
examples that are already there. For any source file filename.py there's a
corresponding file tests/test_filename.py

Cheers,
Ralf


Thanking you in anticipation
> Janani
> (jennystone)
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131224/8e0360fe/attachment.html>

From argriffi at ncsu.edu  Fri Dec 27 04:54:47 2013
From: argriffi at ncsu.edu (alex)
Date: Fri, 27 Dec 2013 04:54:47 -0500
Subject: [Numpy-discussion] Need help with np.ma.median and
	np.apply_along_axis
Message-ID: <CAE5GFcKiQ32NZx+eDigjUtz_9vY7-+wrvS0kJXMNSd3NqnC7-g@mail.gmail.com>

median is faster in version 1.8


From jtaylor.debian at googlemail.com  Fri Dec 27 05:30:14 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 27 Dec 2013 11:30:14 +0100
Subject: [Numpy-discussion] Need help with np.ma.median and
	np.apply_along_axis
In-Reply-To: <CAE5GFcKiQ32NZx+eDigjUtz_9vY7-+wrvS0kJXMNSd3NqnC7-g@mail.gmail.com>
References: <CAE5GFcKiQ32NZx+eDigjUtz_9vY7-+wrvS0kJXMNSd3NqnC7-g@mail.gmail.com>
Message-ID: <52BD56B6.7020406@googlemail.com>

On 27.12.2013 10:54, alex wrote:
> median is faster in version 1.8
> _______________________________________________

unfortunately that won't help here because masked median uses
apply_along_axis again which is very slow, especially if one wants to
calculate thousands of medians of 7 elements as in this case.

Also masked median has not been changed to use partition yet.


From ndbecker2 at gmail.com  Mon Dec 30 11:29:08 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 30 Dec 2013 11:29:08 -0500
Subject: [Numpy-discussion] proposal: min, max of complex should give warning
Message-ID: <l9s708$fof$1@ger.gmane.org>

I propose the following change:  min, max applied to complex should
give a warning.

The rationale is, when the user applies min or max to complex, it's probably
a mistake.



From chekir.amira at gmail.com  Tue Dec 31 08:13:57 2013
From: chekir.amira at gmail.com (Amira Chekir)
Date: Tue, 31 Dec 2013 14:13:57 +0100
Subject: [Numpy-discussion] Loading large NIfTI file -> MemoryError
Message-ID: <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg=EQ29Zw@mail.gmail.com>

Hello together,

I try to load a (large)  NIfTI file (DMRI from Human Connectome Project,
about 1 GB) with NiBabel.

import nibabel as nib
img = nib.load("dmri.nii.gz")
data = img.get_data()

The program crashes during "img.get_data()" with an "MemoryError" (having 4
GB of RAM in my machine).

Any suggestions?

Best regards,
AMIRA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131231/b13969b3/attachment.html>

From jtaylor.debian at googlemail.com  Tue Dec 31 08:29:42 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Tue, 31 Dec 2013 14:29:42 +0100
Subject: [Numpy-discussion] Loading large NIfTI file -> MemoryError
In-Reply-To: <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg=EQ29Zw@mail.gmail.com>
References: <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg=EQ29Zw@mail.gmail.com>
Message-ID: <52C2C6C6.6070002@googlemail.com>

On 31.12.2013 14:13, Amira Chekir wrote:
> Hello together,
> 
> I try to load a (large)  NIfTI file (DMRI from Human Connectome Project,
> about 1 GB) with NiBabel.
> 
> import nibabel as nib
> img = nib.load("dmri.nii.gz")
> data = img.get_data()
> 
> The program crashes during "img.get_data()" with an "MemoryError"
> (having 4 GB of RAM in my machine).
> 
> Any suggestions?

are you using a 64 bit operating system?
which version of numpy?

assuming nibabel uses np.load under the hood you could try it with numpy
1.8 which reduces excess memory usage when loading compressed files.


From tim at cerazone.net  Tue Dec 31 08:51:52 2013
From: tim at cerazone.net (Cera, Tim)
Date: Tue, 31 Dec 2013 08:51:52 -0500
Subject: [Numpy-discussion] proposal: min,
	max of complex should give warning
In-Reply-To: <l9s708$fof$1@ger.gmane.org>
References: <l9s708$fof$1@ger.gmane.org>
Message-ID: <CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg@mail.gmail.com>

I don't work with complex numbers, but just sampling what others do:


Python: no ordering, results in TypeError

Matlab: sorts by magnitude
http://www.mathworks.com/help/matlab/ref/sort.html

R: sorts first by real, then by imaginary
http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html

Numpy: sorts first by real, then by imaginary (the documentation link
below calls this sort 'lexicographical' which I don't think is
correct)
http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html


I would think that the Matlab sort might be more useful, but easy
enough by using the absolute value.

I think what Numpy does is normal enough to not justify a warning, but
leave this to others because as I pointed out in the beginning I don't
work with complex numbers.

Kindest regards,
Tim


From ndbecker2 at gmail.com  Tue Dec 31 10:52:47 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 31 Dec 2013 10:52:47 -0500
Subject: [Numpy-discussion] proposal: min,
	max of complex should give warning
References: <l9s708$fof$1@ger.gmane.org>
	<CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg@mail.gmail.com>
Message-ID: <l9up83$549$1@ger.gmane.org>

Cera, Tim wrote:

> I don't work with complex numbers, but just sampling what others do:
> 
> 
> Python: no ordering, results in TypeError
> 
> Matlab: sorts by magnitude
> http://www.mathworks.com/help/matlab/ref/sort.html
> 
> R: sorts first by real, then by imaginary
> http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> 
> Numpy: sorts first by real, then by imaginary (the documentation link
> below calls this sort 'lexicographical' which I don't think is
> correct)
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> 
> 
> I would think that the Matlab sort might be more useful, but easy
> enough by using the absolute value.
> 
> I think what Numpy does is normal enough to not justify a warning, but
> leave this to others because as I pointed out in the beginning I don't
> work with complex numbers.
> 
> Kindest regards,
> Tim

But I'm not proposing to change numpy's result, which I'm sure would raise many 
objections.  I'm just asking to give a warning, because I think in most cases
this is actually a mistake on the user's part.  Just like the warning currently
given when complex data are truncated to real part.



From ralf.gommers at gmail.com  Tue Dec 31 11:24:05 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 31 Dec 2013 17:24:05 +0100
Subject: [Numpy-discussion] proposal: min,
	max of complex should give warning
In-Reply-To: <l9up83$549$1@ger.gmane.org>
References: <l9s708$fof$1@ger.gmane.org>
	<CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg@mail.gmail.com>
	<l9up83$549$1@ger.gmane.org>
Message-ID: <CABL7CQh9Fc0Uh36W9p16mzAR-oYjJ7_k7rU_Dwq+eZND6YrbDA@mail.gmail.com>

On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com> wrote:

> Cera, Tim wrote:
>
> > I don't work with complex numbers, but just sampling what others do:
> >
> >
> > Python: no ordering, results in TypeError
> >
> > Matlab: sorts by magnitude
> > http://www.mathworks.com/help/matlab/ref/sort.html
> >
> > R: sorts first by real, then by imaginary
> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> >
> > Numpy: sorts first by real, then by imaginary (the documentation link
> > below calls this sort 'lexicographical' which I don't think is
> > correct)
> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> >
> >
> > I would think that the Matlab sort might be more useful, but easy
> > enough by using the absolute value.
> >
> > I think what Numpy does is normal enough to not justify a warning, but
> > leave this to others because as I pointed out in the beginning I don't
> > work with complex numbers.
> >
> > Kindest regards,
> > Tim
>
> But I'm not proposing to change numpy's result, which I'm sure would raise
> many
> objections.  I'm just asking to give a warning, because I think in most
> cases
> this is actually a mistake on the user's part.  Just like the warning
> currently
> given when complex data are truncated to real part.
>

Keep in mind that warnings can be highly annoying. If you're a user who
uses this functionality regularly (and you know what you're doing), then
you're going to be very unhappy to have to wrap each function call in:
    olderr = np.seterr(all='ignore')
    max(...)
    np.seterr(**olderr)
or in:
    with warnings.catch_warnings():
        warnings.filterwarnings('ignore', ...)
        max(...)

The actual behavior isn't documented now it looks like, so that should be
done. In the Notes section of max/min probably.

As for your proposal, it would be good to know if adding a warning would
actually catch any bugs. For the truncation warning it caught several in
scipy and other libs IIRC.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131231/add729d8/attachment.html>

From ndbecker2 at gmail.com  Tue Dec 31 11:45:08 2013
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 31 Dec 2013 11:45:08 -0500
Subject: [Numpy-discussion] proposal: min,
	max of complex should give warning
References: <l9s708$fof$1@ger.gmane.org>
	<CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg@mail.gmail.com>
	<l9up83$549$1@ger.gmane.org>
	<CABL7CQh9Fc0Uh36W9p16mzAR-oYjJ7_k7rU_Dwq+eZND6YrbDA@mail.gmail.com>
Message-ID: <l9usa8$7tr$1@ger.gmane.org>

Ralf Gommers wrote:

> On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> 
>> Cera, Tim wrote:
>>
>> > I don't work with complex numbers, but just sampling what others do:
>> >
>> >
>> > Python: no ordering, results in TypeError
>> >
>> > Matlab: sorts by magnitude
>> > http://www.mathworks.com/help/matlab/ref/sort.html
>> >
>> > R: sorts first by real, then by imaginary
>> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
>> >
>> > Numpy: sorts first by real, then by imaginary (the documentation link
>> > below calls this sort 'lexicographical' which I don't think is
>> > correct)
>> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
>> >
>> >
>> > I would think that the Matlab sort might be more useful, but easy
>> > enough by using the absolute value.
>> >
>> > I think what Numpy does is normal enough to not justify a warning, but
>> > leave this to others because as I pointed out in the beginning I don't
>> > work with complex numbers.
>> >
>> > Kindest regards,
>> > Tim
>>
>> But I'm not proposing to change numpy's result, which I'm sure would raise
>> many
>> objections.  I'm just asking to give a warning, because I think in most
>> cases
>> this is actually a mistake on the user's part.  Just like the warning
>> currently
>> given when complex data are truncated to real part.
>>
> 
> Keep in mind that warnings can be highly annoying. If you're a user who
> uses this functionality regularly (and you know what you're doing), then
> you're going to be very unhappy to have to wrap each function call in:
>     olderr = np.seterr(all='ignore')
>     max(...)
>     np.seterr(**olderr)
> or in:
>     with warnings.catch_warnings():
>         warnings.filterwarnings('ignore', ...)
>         max(...)
> 
> The actual behavior isn't documented now it looks like, so that should be
> done. In the Notes section of max/min probably.
> 
> As for your proposal, it would be good to know if adding a warning would
> actually catch any bugs. For the truncation warning it caught several in
> scipy and other libs IIRC.
> 
> Ralf

I tripped over it yesterday, which is what prompted my suggestion.



From jtaylor.debian at googlemail.com  Tue Dec 31 11:57:18 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Tue, 31 Dec 2013 17:57:18 +0100
Subject: [Numpy-discussion] ANN:  NumPy 1.7.2 release
Message-ID: <52C2F76E.9010509@googlemail.com>

Hello,

I'm happy to announce the of Numpy 1.7.2.
This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3.

More than 42 issues were fixed, the most important issues are listed in
the release notes:
https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst

Compared to the last release candidate four additional minor issues have
been fixed and compatibility with python 3.4b1 improved.

Source tarballs, installers and release notes can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.7.2

Cheers,
Julian Taylor


From charlesr.harris at gmail.com  Tue Dec 31 12:47:44 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 31 Dec 2013 10:47:44 -0700
Subject: [Numpy-discussion] ANN: NumPy 1.7.2 release
In-Reply-To: <52C2F76E.9010509@googlemail.com>
References: <52C2F76E.9010509@googlemail.com>
Message-ID: <CAB6mnxJBWn+FM25=abrqm4DNRG7f6-1keU_hPd253O64d0-Yhw@mail.gmail.com>

On Tue, Dec 31, 2013 at 9:57 AM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> Hello,
>
> I'm happy to announce the of Numpy 1.7.2.
> This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3.
>
> More than 42 issues were fixed, the most important issues are listed in
> the release notes:
> https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst
>
> Compared to the last release candidate four additional minor issues have
> been fixed and compatibility with python 3.4b1 improved.
>
> Source tarballs, installers and release notes can be found at
> https://sourceforge.net/projects/numpy/files/NumPy/1.7.2
>
>
Congrats on the release.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131231/946abcb9/attachment.html>

From d.l.goldsmith at gmail.com  Tue Dec 31 14:43:49 2013
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Tue, 31 Dec 2013 11:43:49 -0800
Subject: [Numpy-discussion] proposal: min,
	max of complex should give warning (Ralf Gommers)
Message-ID: <CAFtPsZr9FN6wwr85ScDNhqdkWZhG7C+=rWU6EVuBMG+mY-XJdA@mail.gmail.com>

>
> As for your proposal, it would be good to know if adding a warning would
> actually catch any bugs. For the truncation warning it caught several in
> scipy and other libs IIRC.
>
> Ralf
>

In light of this, perhaps the pertinent unit tests should be modified (even
if the warning suggestion isn't adopted, about which I'm neutral...but I'm
a little surprised that there isn't a generic way to globally turn off
specific warnings).

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20131231/ac17f43e/attachment.html>