From sturla.molden at gmail.com  Wed Nov  2 18:38:13 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 2 Nov 2016 22:38:13 +0000 (UTC)
Subject: [Numpy-discussion] missing from contributor list?
Message-ID: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>

Why am I missing from the contributor hist here?

https://github.com/numpy/numpy/blob/master/numpy/_build_utils/src/apple_sgemv_fix.c


Sturla



From robert.kern at gmail.com  Wed Nov  2 18:46:01 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 2 Nov 2016 15:46:01 -0700
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAF6FJisncUTBxt1KTkDYPyMeFNGzgPQrUN2mrqgXZ56Yutrc1g@mail.gmail.com>

Because Github (or maybe git) doesn't track the history of the file through
all of the renames. It is only reporting the contributors of changes to the
file at its current location. If you go back to the time just prior to the
commit that renamed the file, you do show up in the list:

https://github.com/numpy/numpy/blob/f179ec92d8ddb0dc5f7445255022be5c4765a704/numpy/build_utils/src/apple_sgemv_fix.c

On Wed, Nov 2, 2016 at 3:38 PM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> Why am I missing from the contributor hist here?
>
> https://github.com/numpy/numpy/blob/master/numpy/_
> build_utils/src/apple_sgemv_fix.c
>
>
> Sturla
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161102/a5469544/attachment.html>

From evgeny.burovskiy at gmail.com  Wed Nov  2 18:50:33 2016
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Thu, 3 Nov 2016 01:50:33 +0300
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <CAF6FJisncUTBxt1KTkDYPyMeFNGzgPQrUN2mrqgXZ56Yutrc1g@mail.gmail.com>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
 <CAF6FJisncUTBxt1KTkDYPyMeFNGzgPQrUN2mrqgXZ56Yutrc1g@mail.gmail.com>
Message-ID: <CAMRo0itwGkt=YRL8bdxx6Ocsp5bcwpvwsmCd3fdfNYb0agNMuA@mail.gmail.com>

br at duneyrr:~/repos/numpy/numpy/_build_utils$ git blame -M -C -C
apple_accelerate.py
4743f3b4 numpy/_build_utils/apple_accelerate.py (Charles Harris
2015-12-05 19:16:00 -0700  1) from __future__ import division, a
4743f3b4 numpy/_build_utils/apple_accelerate.py (Charles Harris
2015-12-05 19:16:00 -0700  2)
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  3) import os
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  4) import sys
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  5) import re
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  6)
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  7) __all__ = ['uses_accelerate_framew
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  8)
cf307769 numpy/build_utils/apple_accelerate.py  (Sturla Molden
2014-10-27 07:45:57 +0100  9) def uses_accelerate_framework(info
<snip>


On Thu, Nov 3, 2016 at 1:46 AM, Robert Kern <robert.kern at gmail.com> wrote:
> Because Github (or maybe git) doesn't track the history of the file through
> all of the renames. It is only reporting the contributors of changes to the
> file at its current location. If you go back to the time just prior to the
> commit that renamed the file, you do show up in the list:
>
> https://github.com/numpy/numpy/blob/f179ec92d8ddb0dc5f7445255022be5c4765a704/numpy/build_utils/src/apple_sgemv_fix.c
>
> On Wed, Nov 2, 2016 at 3:38 PM, Sturla Molden <sturla.molden at gmail.com>
> wrote:
>>
>> Why am I missing from the contributor hist here?
>>
>>
>> https://github.com/numpy/numpy/blob/master/numpy/_build_utils/src/apple_sgemv_fix.c
>>
>>
>> Sturla
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> --
> Robert Kern
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From jtaylor.debian at googlemail.com  Wed Nov  2 18:52:16 2016
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 2 Nov 2016 23:52:16 +0100
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
Message-ID: <da0d6955-c8ea-85c1-577b-19e00b975196@googlemail.com>

On 02.11.2016 23:38, Sturla Molden wrote:
> Why am I missing from the contributor hist here?
> 
> https://github.com/numpy/numpy/blob/master/numpy/_build_utils/src/apple_sgemv_fix.c
> 
> 

Probably because the file was moved and github does not track this. This
is something you should ask gihub about and not this list.

fwiw. git blame -C still shows it



From charlesr.harris at gmail.com  Wed Nov  2 18:59:58 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 2 Nov 2016 16:59:58 -0600
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAB6mnxL9dHu-PSzCsO+9xRQzDSoxcLPBwL8dzy-+5BT4w_8X+g@mail.gmail.com>

On Wed, Nov 2, 2016 at 4:38 PM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> Why am I missing from the contributor hist here?
>
> https://github.com/numpy/numpy/blob/master/numpy/_
> build_utils/src/apple_sgemv_fix.c
>
>
>
You still show up in the commit log of if you follow the file
```
git log --follow numpy/_build_utils/apple_accelerate.py
```

So I have to agree with others that the problem is on the github end.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161102/cdca1ff7/attachment.html>

From mail at telenczuk.pl  Thu Nov  3 06:10:22 2016
From: mail at telenczuk.pl (mail at telenczuk.pl)
Date: Thu, 03 Nov 2016 11:10:22 +0100
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <CAB6mnxL9dHu-PSzCsO+9xRQzDSoxcLPBwL8dzy-+5BT4w_8X+g@mail.gmail.com>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
 <CAB6mnxL9dHu-PSzCsO+9xRQzDSoxcLPBwL8dzy-+5BT4w_8X+g@mail.gmail.com>
Message-ID: <581b0d0e1261f_131115123bc5f@Pct-EqAlain-Z30.notmuch>

Hi,

I had the same problem when I changed my email address. It seems that github matches user profiles to commits using the email address, so you have to make sure that the email you sign your commits with is registered in your github account.

Cheers,

Bartosz

Charles R Harris wrote:
> On Wed, Nov 2, 2016 at 4:38 PM, Sturla Molden <sturla.molden at gmail.com>
> wrote:
> 
> > Why am I missing from the contributor hist here?
> >
> > https://github.com/numpy/numpy/blob/master/numpy/_
> > build_utils/src/apple_sgemv_fix.c
> >
> >
> >
> You still show up in the commit log of if you follow the file
> ```
> git log --follow numpy/_build_utils/apple_accelerate.py
> ```
> 
> So I have to agree with others that the problem is on the github end.
> 
> Chuck




From jtaylor.debian at googlemail.com  Thu Nov  3 10:24:24 2016
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Thu, 3 Nov 2016 15:24:24 +0100
Subject: [Numpy-discussion] missing from contributor list?
In-Reply-To: <581b0d0e1261f_131115123bc5f@Pct-EqAlain-Z30.notmuch>
References: <66871716499819023.492381sturla.molden-gmail.com@news.gmane.org>
 <CAB6mnxL9dHu-PSzCsO+9xRQzDSoxcLPBwL8dzy-+5BT4w_8X+g@mail.gmail.com>
 <581b0d0e1261f_131115123bc5f@Pct-EqAlain-Z30.notmuch>
Message-ID: <f994bef9-56b5-3ba8-f03d-179536032dfe@googlemail.com>

you can add multiple email addresses to your github settings so commits 
are properly attributed to your account.


On 11/03/2016 11:10 AM, mail at telenczuk.pl wrote:
> Hi,
>
> I had the same problem when I changed my email address. It seems that github matches user profiles to commits using the email address, so you have to make sure that the email you sign your commits with is registered in your github account.
>
> Cheers,
>
> Bartosz
>
> Charles R Harris wrote:
>> On Wed, Nov 2, 2016 at 4:38 PM, Sturla Molden <sturla.molden at gmail.com>
>> wrote:
>>
>>> Why am I missing from the contributor hist here?
>>>
>>> https://github.com/numpy/numpy/blob/master/numpy/_
>>> build_utils/src/apple_sgemv_fix.c
>>>
>>>
>>>
>> You still show up in the commit log of if you follow the file
>> ```
>> git log --follow numpy/_build_utils/apple_accelerate.py
>> ```
>>
>> So I have to agree with others that the problem is on the github end.
>>
>> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From yuri.sukhov at gmail.com  Thu Nov  3 11:17:26 2016
From: yuri.sukhov at gmail.com (Yuri Sukhov)
Date: Thu, 3 Nov 2016 18:17:26 +0300
Subject: [Numpy-discussion] Using numpy.rint() with scalars
Message-ID: <CADbd57fsPtko1FsryU4QRjpgkH-f_g9N06Dqoouj+4HeCgRT1Q@mail.gmail.com>

Hi all,

According to the documentation for numpy.rint() (https://docs.scipy.org/doc/
numpy/reference/generated/numpy.rint.html), it's a ufunc that accepts an
array-like object as an input.

But it also works with scalar inputs. Could anyone clarify if such use case
is considered to be common and acceptable? It it just the documentation
that does not cover one of the possible scenarios, or is it a side effect
and such use case should be avoided?

I want to use rint() with scalars as I have not found an alternative with
the same behavior in cPython, but if it's not how it should be used, I
don't want to rewrite the app when that behavior will change.

Thanks!

Yuri.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161103/2431ea93/attachment.html>

From charlesr.harris at gmail.com  Thu Nov  3 11:31:18 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 3 Nov 2016 09:31:18 -0600
Subject: [Numpy-discussion] Using numpy.rint() with scalars
In-Reply-To: <CADbd57fsPtko1FsryU4QRjpgkH-f_g9N06Dqoouj+4HeCgRT1Q@mail.gmail.com>
References: <CADbd57fsPtko1FsryU4QRjpgkH-f_g9N06Dqoouj+4HeCgRT1Q@mail.gmail.com>
Message-ID: <CAB6mnx+L0MarASitDeaGrXg8US_SkVG5-BTz-RK5jy6GFVBxZw@mail.gmail.com>

On Thu, Nov 3, 2016 at 9:17 AM, Yuri Sukhov <yuri.sukhov at gmail.com> wrote:

> Hi all,
>
> According to the documentation for numpy.rint() (
> https://docs.scipy.org/doc/numpy/reference/generated/numpy.rint.html),
> it's a ufunc that accepts an array-like object as an input.
>
> But it also works with scalar inputs. Could anyone clarify if such use
> case is considered to be common and acceptable? It it just the
> documentation that does not cover one of the possible scenarios, or is it a
> side effect and such use case should be avoided?
>
> I want to use rint() with scalars as I have not found an alternative with
> the same behavior in cPython, but if it's not how it should be used, I
> don't want to rewrite the app when that behavior will change.
>
>
Scalars are array_like, they can be converted to arrays.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161103/35d31ccf/attachment.html>

From charlesr.harris at gmail.com  Thu Nov  3 13:04:45 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 3 Nov 2016 11:04:45 -0600
Subject: [Numpy-discussion] Branching NumPy 1.12.x
Message-ID: <CAB6mnxLB88+GBc6_meAOf82iDDgyqb6dpqjH4bgZmxaGgGJRJA@mail.gmail.com>

Hi All,

I'm thinking that it is time to branch NumPy 1.12.x. I haven't got
everything in it that I would have liked, in particular __numpy_ufunc__,
but I think there is plenty of material and not branching is holding up
some of the more risky stuff.  My current thoughts on __numpy_ufunc__ is
that it would be best to work it out over the 1.13.0 release cycle,
starting with enabling it again right after the branch. Julian's work on
avoiding temporary copies and Pauli's overlap handling PR are two other
changes I've been putting off but don't want to delay further. There are
some other smaller things that I had scheduled for 1.12.0, but would like
to spend more time looking at. If there are some things that you think just
have to be in 1.12.0, please mention them, but I'd rather aim at getting
1.13.0 out in a timely manner.

Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161103/6a5a1fe7/attachment.html>

From Nicolas.Rougier at inria.fr  Thu Nov  3 14:29:10 2016
From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier)
Date: Thu, 3 Nov 2016 19:29:10 +0100
Subject: [Numpy-discussion] Useless but tricky exercise
Message-ID: <6652346A-AC95-4A71-82BE-9A0A0F254303@inria.fr>


Hi all,

Given an array V that is a view of a base B, I was wondering if it is possible
to find a (string) index such that `V = B[index]`

For example:

```
B = np.arange(8*8).reshape(8,8)
V = B[::2,::2]

index = find_view(B,V)
print(index)
"::2,::2"

print(np.allclose(V, eval("B[%s]" % index)))
True
```


I wrote a solution at https://gist.github.com/rougier/b8c2256434b3a4a4271260cd4cc6cbc7 
(not very thoroughly tested) but maybe there are better ways to do that (like a magic numpy call ?)


(no use-case at all, only for teaching)


Nicolas



From ralf.gommers at gmail.com  Fri Nov  4 04:05:41 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 4 Nov 2016 21:05:41 +1300
Subject: [Numpy-discussion] Branching NumPy 1.12.x
In-Reply-To: <CAB6mnxLB88+GBc6_meAOf82iDDgyqb6dpqjH4bgZmxaGgGJRJA@mail.gmail.com>
References: <CAB6mnxLB88+GBc6_meAOf82iDDgyqb6dpqjH4bgZmxaGgGJRJA@mail.gmail.com>
Message-ID: <CABL7CQgf7JXQozhuak6n99x6KKEZQNKf5Mg+YoT91GRsmyfznA@mail.gmail.com>

On Fri, Nov 4, 2016 at 6:04 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

> Hi All,
>
> I'm thinking that it is time to branch NumPy 1.12.x. I haven't got
> everything in it that I would have liked, in particular __numpy_ufunc__,
> but I think there is plenty of material and not branching is holding up
> some of the more risky stuff.  My current thoughts on __numpy_ufunc__ is
> that it would be best to work it out over the 1.13.0 release cycle,
> starting with enabling it again right after the branch. Julian's work on
> avoiding temporary copies and Pauli's overlap handling PR are two other
> changes I've been putting off but don't want to delay further. There are
> some other smaller things that I had scheduled for 1.12.0, but would like
> to spend more time looking at. If there are some things that you think just
> have to be in 1.12.0, please mention them, but I'd rather aim at getting
> 1.13.0 out in a timely manner.
>
> Thoughts?
>

That's a really good plan I think.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/fa5061ba/attachment.html>

From ndbecker2 at gmail.com  Fri Nov  4 08:06:32 2016
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 04 Nov 2016 08:06:32 -0400
Subject: [Numpy-discussion] array comprehension
Message-ID: <nvhtk5$s93$1@blaine.gmane.org>

I find I often write:
np.array ([some list comprehension])

mainly because list comprehensions are just so sweet.

But I imagine this isn't particularly efficient.

I wonder if numpy has a "better" way, and if not, maybe it would be a nice 
addition?



From faltet at gmail.com  Fri Nov  4 08:26:10 2016
From: faltet at gmail.com (Francesc Alted)
Date: Fri, 4 Nov 2016 13:26:10 +0100
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <nvhtk5$s93$1@blaine.gmane.org>
References: <nvhtk5$s93$1@blaine.gmane.org>
Message-ID: <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>

2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:

> I find I often write:
> np.array ([some list comprehension])
>
> mainly because list comprehensions are just so sweet.
>
> But I imagine this isn't particularly efficient.
>

Right.  Using a generator and np.fromiter() will avoid the creation of the
intermediate list.  Something like:

np.fromiter((i for i in range(x)))  # use xrange for Python 2


>
> I wonder if numpy has a "better" way, and if not, maybe it would be a nice
> addition?
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/3f4e91e0/attachment.html>

From ndbecker2 at gmail.com  Fri Nov  4 09:36:02 2016
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 04 Nov 2016 09:36:02 -0400
Subject: [Numpy-discussion] array comprehension
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
Message-ID: <nvi2rv$190$1@blaine.gmane.org>

Francesc Alted wrote:

> 2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:
> 
>> I find I often write:
>> np.array ([some list comprehension])
>>
>> mainly because list comprehensions are just so sweet.
>>
>> But I imagine this isn't particularly efficient.
>>
> 
> Right.  Using a generator and np.fromiter() will avoid the creation of the
> intermediate list.  Something like:
> 
> np.fromiter((i for i in range(x)))  # use xrange for Python 2
> 
> 
Does this generalize to >1 dimensions?



From faltet at gmail.com  Fri Nov  4 10:12:11 2016
From: faltet at gmail.com (Francesc Alted)
Date: Fri, 4 Nov 2016 15:12:11 +0100
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <nvi2rv$190$1@blaine.gmane.org>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
Message-ID: <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>

2016-11-04 14:36 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:

> Francesc Alted wrote:
>
> > 2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:
> >
> >> I find I often write:
> >> np.array ([some list comprehension])
> >>
> >> mainly because list comprehensions are just so sweet.
> >>
> >> But I imagine this isn't particularly efficient.
> >>
> >
> > Right.  Using a generator and np.fromiter() will avoid the creation of
> the
> > intermediate list.  Something like:
> >
> > np.fromiter((i for i in range(x)))  # use xrange for Python 2
> >
> >
> Does this generalize to >1 dimensions?
>

A reshape() is not enough?  What do you want to do exactly?


>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/55a4297b/attachment.html>

From shoyer at gmail.com  Fri Nov  4 11:04:13 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Fri, 4 Nov 2016 08:04:13 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
 <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>
Message-ID: <CAEQ_TvfCf12cLs-X264idOpeUmN77p_0bRwiD=k0DbWJ4pNV6A@mail.gmail.com>

On Fri, Nov 4, 2016 at 7:12 AM, Francesc Alted <faltet at gmail.com> wrote:

> Does this generalize to >1 dimensions?
>>
>
> A reshape() is not enough?  What do you want to do exactly?
>

np.fromiter takes scalar input and only builds a 1D array. So it actually
can't combine multiple values at once unless they are flattened out in
Python. It could be nice to add support for non-scalar inputs, stacking
them similarly to np.array. Likewise, it could be nice to add an axis
argument, so it can work similarly to np.stack.

More generally, you might want to iterate and rebuild over arbitrary
dimension(s) of an array. Something like
np.stack([x for x in np.unstack(y, axis)], axis)

But, we also don't have an unstack function. This would mostly be syntactic
sugar, but I think it would be a nice addition. Such a function actually
exists in TensorFlow:
https://g3doc.corp.google.com/third_party/tensorflow/g3doc/api_docs/python/array_ops.md?cl=head#unstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/6cdaae34/attachment.html>

From davidmenhur at gmail.com  Fri Nov  4 11:11:51 2016
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Fri, 4 Nov 2016 16:11:51 +0100
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAEQ_TvfCf12cLs-X264idOpeUmN77p_0bRwiD=k0DbWJ4pNV6A@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
 <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>
 <CAEQ_TvfCf12cLs-X264idOpeUmN77p_0bRwiD=k0DbWJ4pNV6A@mail.gmail.com>
Message-ID: <CAJhcF=2zr=MOM+xHkk2DKBe_Z-4LE1+98i4JsXohq6YYUqaWHQ@mail.gmail.com>

On 4 November 2016 at 16:04, Stephan Hoyer <shoyer at gmail.com> wrote:
>
> But, we also don't have an unstack function. This would mostly be syntactic
> sugar, but I think it would be a nice addition. Such a function actually
> exists in TensorFlow:
> https://g3doc.corp.google.com/third_party/tensorflow/g3doc/api_docs/python/array_ops.md?cl=head#unstack

That link is behind a login wall. This is the public version:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/array_ops.md


From rmay31 at gmail.com  Fri Nov  4 11:33:22 2016
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 4 Nov 2016 09:33:22 -0600
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAEQ_TvfCf12cLs-X264idOpeUmN77p_0bRwiD=k0DbWJ4pNV6A@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
 <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>
 <CAEQ_TvfCf12cLs-X264idOpeUmN77p_0bRwiD=k0DbWJ4pNV6A@mail.gmail.com>
Message-ID: <CAKH0P+X7XLKcnq++=i4_OsFHbrRprmpXewVkhcF6+S-iuDgcQw@mail.gmail.com>

On Fri, Nov 4, 2016 at 9:04 AM, Stephan Hoyer <shoyer at gmail.com> wrote:

> On Fri, Nov 4, 2016 at 7:12 AM, Francesc Alted <faltet at gmail.com> wrote:
>
>> Does this generalize to >1 dimensions?
>>>
>>
>> A reshape() is not enough?  What do you want to do exactly?
>>
>
> np.fromiter takes scalar input and only builds a 1D array. So it actually
> can't combine multiple values at once unless they are flattened out in
> Python. It could be nice to add support for non-scalar inputs, stacking
> them similarly to np.array. Likewise, it could be nice to add an axis
> argument, so it can work similarly to np.stack.
>

 itertools.product, itertools.permutation, etc. with np.fromiter (and
reshape) is probably also useful here, though it doesn't solve the
non-scalar problem.

Ryan

-- 
Ryan May
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/9a13d8ce/attachment.html>

From ndbecker2 at gmail.com  Fri Nov  4 11:56:39 2016
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 04 Nov 2016 11:56:39 -0400
Subject: [Numpy-discussion] array comprehension
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
 <CAFrp1vpvtb6ezzJHCS45uaRkhykrGs47sAiE2zi7sPOwZQha4g@mail.gmail.com>
Message-ID: <nvib3j$e4g$1@blaine.gmane.org>

Francesc Alted wrote:

> 2016-11-04 14:36 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:
> 
>> Francesc Alted wrote:
>>
>> > 2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:
>> >
>> >> I find I often write:
>> >> np.array ([some list comprehension])
>> >>
>> >> mainly because list comprehensions are just so sweet.
>> >>
>> >> But I imagine this isn't particularly efficient.
>> >>
>> >
>> > Right.  Using a generator and np.fromiter() will avoid the creation of
>> the
>> > intermediate list.  Something like:
>> >
>> > np.fromiter((i for i in range(x)))  # use xrange for Python 2
>> >
>> >
>> Does this generalize to >1 dimensions?
>>
> 
> A reshape() is not enough?  What do you want to do exactly?
> 

I was thinking about:
x = np.array ([[L1] L2]) where L1,L2 take the form of a list comprehension,
as a means to create a 2-D array (in this example)



From robert.kern at gmail.com  Fri Nov  4 11:59:23 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 4 Nov 2016 08:59:23 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <nvi2rv$190$1@blaine.gmane.org>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <nvi2rv$190$1@blaine.gmane.org>
Message-ID: <CAF6FJiv5rM6eNH0ZuEKkF5RUht50AJACXu-ueQyBBiPRvf=w6Q@mail.gmail.com>

On Fri, Nov 4, 2016 at 6:36 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> Francesc Alted wrote:
>
> > 2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:
> >
> >> I find I often write:
> >> np.array ([some list comprehension])
> >>
> >> mainly because list comprehensions are just so sweet.
> >>
> >> But I imagine this isn't particularly efficient.
> >>
> >
> > Right.  Using a generator and np.fromiter() will avoid the creation of
the
> > intermediate list.  Something like:
> >
> > np.fromiter((i for i in range(x)))  # use xrange for Python 2
> >
> >
> Does this generalize to >1 dimensions?

No.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/b524c92e/attachment.html>

From harrigan.matthew at gmail.com  Fri Nov  4 13:11:36 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Fri, 4 Nov 2016 13:11:36 -0400
Subject: [Numpy-discussion]  ufunc for sum of squared difference
Message-ID: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>

I was reading this
<http://notes-on-cython.readthedocs.io/en/latest/std_dev.html> and got
thinking about if a ufunc could compute the sum of squared differences in a
single pass without a temporary array.  The python code below demonstrates
a possible approach.

import numpy as np
x = np.arange(10)
c = 1.0
def add_square_diff(x1, x2):
    return x1 + (x2-c)**2
ufunc = np.frompyfunc(add_square_diff, 2, 1)
print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
print(np.sum(np.square(x-c)))

I have (at least) 4 questions:
1. Is it possible to pass run time constants to a ufunc written in C for
use in its inner loop, and if so how?
2. Is it possible to pass an initial value to reduce to avoid the clean up
required for the first element?
3. Does that ufunc work, or are there special cases which cause it to fall
apart?
4. Would a very specialized ufunc such as this be considered for
incorporating in numpy since it would help reduce time and memory of
functions already in numpy?

Thank you,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/49bb2f9a/attachment.html>

From jorisvandenbossche at gmail.com  Fri Nov  4 13:14:55 2016
From: jorisvandenbossche at gmail.com (Joris Van den Bossche)
Date: Fri, 4 Nov 2016 18:14:55 +0100
Subject: [Numpy-discussion] ANN: pandas v0.19.1 released!
Message-ID: <CALQtMBYeEqeoGGMp1rXwGJCJFzfeS5s_zDUopZgRfwW95fEovw@mail.gmail.com>

Hi all,

I'm pleased to announce the release of pandas 0.19.1.
This is a bug-fix release from 0.19.0 and includes some small regression
fixes, bug fixes and performance improvements. We recommend that all users
upgrade to this version.

See the v0.19.1 Whatsnew page
<http://pandas.pydata.org/pandas-docs/version/0.19.1/whatsnew.html> for an
overview of all bugs that have been fixed in 0.19.1.

Thanks to all contributors!

Joris

---

*How to get it:*

Source tarballs and windows/mac/linux wheels are available on PyPI (thanks
to Christoph Gohlke for the windows wheels, and to Matthew Brett for
setting up the mac/linux wheels).
Conda packages are already available via the conda-forge channel (conda
install pandas -c conda-forge). It will be available on the main channel
shortly.

*Issues:*

Please report any issues on our issue tracker: https://github.com/pydata/
pandas/issues

*Thanks to all the contributors of the 0.19.1 release:*

   - Adam Chainz
   - Anthonios Partheniou
   - Arash Rouhani
   - Ben Kandel
   - Brandon M. Burroughs
   - Chris
   - chris-b1
   - Chris Warth
   - David Krych
   - dubourg
   - gfyoung
   - Iv?n Vall?s P?rez
   - Jeff Reback
   - Joe Jevnik
   - Jon M. Mease
   - Joris Van den Bossche
   - Josh Owen
   - Keshav Ramaswamy
   - Larry Ren
   - mattrijk
   - Michael Felt
   - paul-mannino
   - Piotr Chromiec
   - Robert Bradshaw
   - Sinhrks
   - Thiago Serafim
   - Tom Bird
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/a0554690/attachment.html>

From njs at pobox.com  Fri Nov  4 13:24:22 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 4 Nov 2016 10:24:22 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAPJVwBkFEKDgj9iMk-aLhcVwxCGW9RTb9Ob=eitpBB7+Q9ET=A@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <CAPJVwB=C1u0rkqhvc_Wbc=Tpcu6wBvvHpWccfVdhU9j5xFq4KQ@mail.gmail.com>
 <CAPJVwBkFEKDgj9iMk-aLhcVwxCGW9RTb9Ob=eitpBB7+Q9ET=A@mail.gmail.com>
Message-ID: <CAPJVwBm0xtQcM5__e0CyH49cFL4CV1=-qJ=gns=71h+v__cjHA@mail.gmail.com>

Are you sure fromiter doesn't make an intermediate list or equivalent? It
has to collect all the values before it can know the shape or dtype of the
array to put them in.

On Nov 4, 2016 5:26 AM, "Francesc Alted" <faltet at gmail.com> wrote:



2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2 at gmail.com>:

> I find I often write:
> np.array ([some list comprehension])
>
> mainly because list comprehensions are just so sweet.
>
> But I imagine this isn't particularly efficient.
>

Right.  Using a generator and np.fromiter() will avoid the creation of the
intermediate list.  Something like:

np.fromiter((i for i in range(x)))  # use xrange for Python 2


>
> I wonder if numpy has a "better" way, and if not, maybe it would be a nice
> addition?
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Francesc Alted

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/8c6cf9b8/attachment.html>

From shoyer at gmail.com  Fri Nov  4 13:32:05 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Fri, 4 Nov 2016 10:32:05 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAPJVwBm0xtQcM5__e0CyH49cFL4CV1=-qJ=gns=71h+v__cjHA@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <CAPJVwB=C1u0rkqhvc_Wbc=Tpcu6wBvvHpWccfVdhU9j5xFq4KQ@mail.gmail.com>
 <CAPJVwBkFEKDgj9iMk-aLhcVwxCGW9RTb9Ob=eitpBB7+Q9ET=A@mail.gmail.com>
 <CAPJVwBm0xtQcM5__e0CyH49cFL4CV1=-qJ=gns=71h+v__cjHA@mail.gmail.com>
Message-ID: <CAEQ_TvdACUL5VwoJW1kNe1OXy4-pEzm9-xr5ZvuacXEH1-E4cA@mail.gmail.com>

On Fri, Nov 4, 2016 at 10:24 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Are you sure fromiter doesn't make an intermediate list or equivalent? It
> has to collect all the values before it can know the shape or dtype of the
> array to put them in.
>
fromiter dynamically resizes a NumPy array, like a Python list, except with
a growth factor of 1.5 (rather than 1.25):
https://github.com/numpy/numpy/blob/bb59409abf5237c155a1dc4c4d5b31e4acf32fbe/numpy/core/src/multiarray/ctors.c#L3721
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/ac62334a/attachment.html>

From njs at pobox.com  Fri Nov  4 13:36:06 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 4 Nov 2016 10:36:06 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAEQ_TvdACUL5VwoJW1kNe1OXy4-pEzm9-xr5ZvuacXEH1-E4cA@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <CAPJVwB=C1u0rkqhvc_Wbc=Tpcu6wBvvHpWccfVdhU9j5xFq4KQ@mail.gmail.com>
 <CAPJVwBkFEKDgj9iMk-aLhcVwxCGW9RTb9Ob=eitpBB7+Q9ET=A@mail.gmail.com>
 <CAPJVwBm0xtQcM5__e0CyH49cFL4CV1=-qJ=gns=71h+v__cjHA@mail.gmail.com>
 <CAEQ_TvdACUL5VwoJW1kNe1OXy4-pEzm9-xr5ZvuacXEH1-E4cA@mail.gmail.com>
Message-ID: <CAPJVwB=0DFNUGw3J3yi1SAvn0pHYiRrs7qFCcHqBec8U_GgSVg@mail.gmail.com>

On Nov 4, 2016 10:32 AM, "Stephan Hoyer" <shoyer at gmail.com> wrote:
>
> On Fri, Nov 4, 2016 at 10:24 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Are you sure fromiter doesn't make an intermediate list or equivalent?
It has to collect all the values before it can know the shape or dtype of
the array to put them in.
>
> fromiter dynamically resizes a NumPy array, like a Python list, except
with a growth factor of 1.5 (rather than 1.25):
>
https://github.com/numpy/numpy/blob/bb59409abf5237c155a1dc4c4d5b31e4acf32fbe/numpy/core/src/multiarray/ctors.c#L3721


Oh, right, and the dtype argument is mandatory, which is what makes this
possible.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/7d3ac5e5/attachment.html>

From sebastian at sipsolutions.net  Fri Nov  4 13:56:48 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 04 Nov 2016 18:56:48 +0100
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
Message-ID: <1478282208.861.29.camel@sipsolutions.net>

On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> I was reading this and got thinking about if a ufunc could compute
> the sum of squared differences in a single pass without a temporary
> array.? The python code below demonstrates a possible approach.
> 
> import numpy as np
> x = np.arange(10)
> c = 1.0
> def add_square_diff(x1, x2):
> ??? return x1 + (x2-c)**2
> ufunc = np.frompyfunc(add_square_diff, 2, 1)
> print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> print(np.sum(np.square(x-c)))
> 
> I have (at least) 4 questions:
> 1. Is it possible to pass run time constants to a ufunc written in C
> for use in its inner loop, and if so how?

I don't think its anticipated, since a ufunc could in most cases use a
third argument, but a 3 arg ufunc can't be reduced. Not sure if there
might be some trickery possible.

> 2. Is it possible to pass an initial value to reduce to avoid the
> clean up required for the first element?

This is the identity normally. But the identity can only be 0, 1 or -1
right now I think. The identity is what the output array gets
initialized with (which effectively makes it the first value passed
into the inner loop).

> 3. Does that ufunc work, or are there special cases which cause it to
> fall apart?
> 4. Would a very specialized ufunc such as this be considered for
> incorporating in numpy since it would help reduce time and memory of
> functions already in numpy?
> 

Might be mixing up things, however, IIRC the single pass approach has a
bad numerical accuracy, so that I doubt that it is a good default
algorithm.

- Sebastian


> Thank you,
> Matt
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/f3530a37/attachment.sig>

From harrigan.matthew at gmail.com  Fri Nov  4 15:42:25 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Fri, 4 Nov 2016 15:42:25 -0400
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <1478282208.861.29.camel@sipsolutions.net>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
Message-ID: <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>

I didn't notice identity before.  Seems like frompyfunc always sets it to
None.  If it were zero maybe it would work as desired here.

In the writing your own ufunc doc, I was wondering if the pointer to data
could be used to get a constant at runtime.  If not, what could that be
used for?

static void double_logit(char **args, npy_intp *dimensions,
                            npy_intp* steps, void* data)

Why would the numerical accuracy be any different?  The subtraction and
square operations look identical and I thought np.sum just calls
np.add.reduce, so the reduction step uses the same code and would therefore
have the same accuracy.

Thanks

On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> > I was reading this and got thinking about if a ufunc could compute
> > the sum of squared differences in a single pass without a temporary
> > array.  The python code below demonstrates a possible approach.
> >
> > import numpy as np
> > x = np.arange(10)
> > c = 1.0
> > def add_square_diff(x1, x2):
> >     return x1 + (x2-c)**2
> > ufunc = np.frompyfunc(add_square_diff, 2, 1)
> > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> > print(np.sum(np.square(x-c)))
> >
> > I have (at least) 4 questions:
> > 1. Is it possible to pass run time constants to a ufunc written in C
> > for use in its inner loop, and if so how?
>
> I don't think its anticipated, since a ufunc could in most cases use a
> third argument, but a 3 arg ufunc can't be reduced. Not sure if there
> might be some trickery possible.
>
> > 2. Is it possible to pass an initial value to reduce to avoid the
> > clean up required for the first element?
>
> This is the identity normally. But the identity can only be 0, 1 or -1
> right now I think. The identity is what the output array gets
> initialized with (which effectively makes it the first value passed
> into the inner loop).
>
> > 3. Does that ufunc work, or are there special cases which cause it to
> > fall apart?
> > 4. Would a very specialized ufunc such as this be considered for
> > incorporating in numpy since it would help reduce time and memory of
> > functions already in numpy?
> >
>
> Might be mixing up things, however, IIRC the single pass approach has a
> bad numerical accuracy, so that I doubt that it is a good default
> algorithm.
>
> - Sebastian
>
>
> > Thank you,
> > Matt
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/812f6b67/attachment.html>

From chris.barker at noaa.gov  Fri Nov  4 16:08:44 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 4 Nov 2016 13:08:44 -0700
Subject: [Numpy-discussion] array comprehension
In-Reply-To: <CAPJVwB=0DFNUGw3J3yi1SAvn0pHYiRrs7qFCcHqBec8U_GgSVg@mail.gmail.com>
References: <nvhtk5$s93$1@blaine.gmane.org>
 <CAFrp1vqqkXXTOkJgo7VA-HCibXNunSPr8d8Wqs8rnaewxyNjcQ@mail.gmail.com>
 <CAPJVwB=C1u0rkqhvc_Wbc=Tpcu6wBvvHpWccfVdhU9j5xFq4KQ@mail.gmail.com>
 <CAPJVwBkFEKDgj9iMk-aLhcVwxCGW9RTb9Ob=eitpBB7+Q9ET=A@mail.gmail.com>
 <CAPJVwBm0xtQcM5__e0CyH49cFL4CV1=-qJ=gns=71h+v__cjHA@mail.gmail.com>
 <CAEQ_TvdACUL5VwoJW1kNe1OXy4-pEzm9-xr5ZvuacXEH1-E4cA@mail.gmail.com>
 <CAPJVwB=0DFNUGw3J3yi1SAvn0pHYiRrs7qFCcHqBec8U_GgSVg@mail.gmail.com>
Message-ID: <CALGmxEJop7zpSFvmrQ+-FbYhNhpgcFQBSDEmbshEN5kW-CKWvg@mail.gmail.com>

On Fri, Nov 4, 2016 at 10:36 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Nov 4, 2016 10:32 AM, "Stephan Hoyer" <shoyer at gmail.com> wrote:
> > fromiter dynamically resizes a NumPy array, like a Python list, except
> with a growth factor of 1.5
>


> Oh, right, and the dtype argument is mandatory, which is what makes this
> possible.
>
Couldn't it determine the dtype from the first element, and then barf later
if an incompatible one shows up?

And then we could adapt this code to np.array() and get nice performance
with no extra functions to think about calling...

And off the top of my head, I can't think of why it couldn't be generalized
to the nd case as well.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/0a4378e4/attachment.html>

From sebastian at sipsolutions.net  Fri Nov  4 17:33:05 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 04 Nov 2016 22:33:05 +0100
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
Message-ID: <1478295185.4986.0.camel@sipsolutions.net>

On Fr, 2016-11-04 at 15:42 -0400, Matthew Harrigan wrote:
> I didn't notice identity before.? Seems like frompyfunc always sets
> it to None.? If it were zero maybe it would work as desired here.
> 
> In the writing your own ufunc doc, I was wondering if the pointer to
> data could be used to get a constant at runtime.? If not, what could
> that be used for?
> static void double_logit(char **args, npy_intp *dimensions,
> ????????????????????????????npy_intp* steps, void* data)
> Why would the numerical accuracy be any different?? The subtraction
> and square operations look identical and I thought np.sum just calls
> np.add.reduce, so the reduction step uses the same code and would
> therefore have the same accuracy.
> 

Sorry, did not read it carefully, I guess `c` is the mean, so you are
doing the two pass method.

- Sebastian


> Thanks
> 
> On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg <sebastian at sipsolution
> s.net> wrote:
> > On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> > > I was reading this and got thinking about if a ufunc could
> > compute
> > > the sum of squared differences in a single pass without a
> > temporary
> > > array.? The python code below demonstrates a possible approach.
> > >
> > > import numpy as np
> > > x = np.arange(10)
> > > c = 1.0
> > > def add_square_diff(x1, x2):
> > > ??? return x1 + (x2-c)**2
> > > ufunc = np.frompyfunc(add_square_diff, 2, 1)
> > > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> > > print(np.sum(np.square(x-c)))
> > >
> > > I have (at least) 4 questions:
> > > 1. Is it possible to pass run time constants to a ufunc written
> > in C
> > > for use in its inner loop, and if so how?
> > 
> > I don't think its anticipated, since a ufunc could in most cases
> > use a
> > third argument, but a 3 arg ufunc can't be reduced. Not sure if
> > there
> > might be some trickery possible.
> > 
> > > 2. Is it possible to pass an initial value to reduce to avoid the
> > > clean up required for the first element?
> > 
> > This is the identity normally. But the identity can only be 0, 1 or
> > -1
> > right now I think. The identity is what the output array gets
> > initialized with (which effectively makes it the first value passed
> > into the inner loop).
> > 
> > > 3. Does that ufunc work, or are there special cases which cause
> > it to
> > > fall apart?
> > > 4. Would a very specialized ufunc such as this be considered for
> > > incorporating in numpy since it would help reduce time and memory
> > of
> > > functions already in numpy?
> > >
> > 
> > Might be mixing up things, however, IIRC the single pass approach
> > has a
> > bad numerical accuracy, so that I doubt that it is a good default
> > algorithm.
> > 
> > - Sebastian
> > 
> > 
> > > Thank you,
> > > Matt
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at scipy.org
> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161104/2095b7dc/attachment.sig>

From charlesr.harris at gmail.com  Sat Nov  5 19:04:06 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 5 Nov 2016 17:04:06 -0600
Subject: [Numpy-discussion] Numpy 1.12.x branched
Message-ID: <CAB6mnx+1ZCB4eM99a1dJqsORCduKy1prUHbgNWd0nt=UhruhUw@mail.gmail.com>

Hi All,

Numpy 1.12.x has been branched and the 1.13 development branch is open. It
would be helpful if folks could review the release notes as it is likely
I've missed something.  I'd like to make the first beta release in a couple
of days.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161105/7b0a939e/attachment.html>

From sebastian at sipsolutions.net  Sun Nov  6 11:56:12 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sun, 06 Nov 2016 17:56:12 +0100
Subject: [Numpy-discussion] Numpy 1.12.x branched
In-Reply-To: <CAB6mnx+1ZCB4eM99a1dJqsORCduKy1prUHbgNWd0nt=UhruhUw@mail.gmail.com>
References: <CAB6mnx+1ZCB4eM99a1dJqsORCduKy1prUHbgNWd0nt=UhruhUw@mail.gmail.com>
Message-ID: <1478451372.3875.5.camel@sipsolutions.net>

On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote:
> Hi All,
> 
> Numpy 1.12.x has been branched and the 1.13 development branch is
> open. It would be helpful if folks could review the release notes as
> it is likely I've missed something.? I'd like to make the first beta
> release in a couple of days.
> 

Very cool, thanks for all the hard work!

- Sebastian


> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161106/344262ea/attachment.sig>

From charlesr.harris at gmail.com  Sun Nov  6 13:44:04 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 6 Nov 2016 11:44:04 -0700
Subject: [Numpy-discussion] __numpy_ufunc__
Message-ID: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>

Hi All,

For those interested in continuing the __numpy_ufunc__ saga, there is a pull
request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely we
will want to make some changes up front before merging that, so some
discussion is in order.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161106/0ee45ace/attachment.html>

From charlesr.harris at gmail.com  Sun Nov  6 15:10:25 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 6 Nov 2016 13:10:25 -0700
Subject: [Numpy-discussion] __numpy_ufunc__
In-Reply-To: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
References: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
Message-ID: <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>

On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> Hi All,
>
> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
> request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely we
> will want to make some changes up front before merging that, so some
> discussion is in order.
>
>
As a first order of business, let's decide whether to remove the index and
rename `__numpy_ufunc__`. The motivation for this is discussed in issue
#5986. <https://github.com/numpy/numpy/issues/5986>
If we decide positive on that (I'm in favor), I would be happy with the
proposed name `__array_ufunc__`, although something more descriptive like
`__handle_ufunc__` might be better. This is a wonderful opportunity for
bike shedding for those so inclined ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161106/01d17c6d/attachment.html>

From nathan12343 at gmail.com  Sun Nov  6 15:21:08 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Sun, 6 Nov 2016 14:21:08 -0600
Subject: [Numpy-discussion] __numpy_ufunc__
In-Reply-To: <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
References: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
 <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
Message-ID: <CAJXewOkyud6UMx-nfALpj5Gsg5nfeNb1=iUAQfEAr+JyotSoaQ@mail.gmail.com>

I'm interested in this, but was never able to dive in to the lengthy
discussions on this. I'm curious if there is a summary of the current state
of things somewhere that I could read to catch up.

On Sunday, November 6, 2016, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
> charlesr.harris at gmail.com
> <javascript:_e(%7B%7D,'cvml','charlesr.harris at gmail.com');>> wrote:
>
>> Hi All,
>>
>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>> request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely
>> we will want to make some changes up front before merging that, so some
>> discussion is in order.
>>
>>
> As a first order of business, let's decide whether to remove the index and
> rename `__numpy_ufunc__`. The motivation for this is discussed in issue
> #5986. <https://github.com/numpy/numpy/issues/5986>
> If we decide positive on that (I'm in favor), I would be happy with the
> proposed name `__array_ufunc__`, although something more descriptive like
> `__handle_ufunc__` might be better. This is a wonderful opportunity for
> bike shedding for those so inclined ;)
>
> Chuck
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161106/4cdcdcff/attachment.html>

From ralf.gommers at gmail.com  Mon Nov  7 03:08:54 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 7 Nov 2016 21:08:54 +1300
Subject: [Numpy-discussion] __numpy_ufunc__
In-Reply-To: <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
References: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
 <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
Message-ID: <CABL7CQigiTMZ3kYgx5DGX9ru4qN35H1CwVsOh=nYo7+iLfWXGw@mail.gmail.com>

On Mon, Nov 7, 2016 at 9:10 AM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> Hi All,
>>
>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>> request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely
>> we will want to make some changes up front before merging that, so some
>> discussion is in order.
>>
>>
> As a first order of business, let's decide whether to remove the index and
> rename `__numpy_ufunc__`. The motivation for this is discussed in issue
> #5986. <https://github.com/numpy/numpy/issues/5986>
> If we decide positive on that (I'm in favor),
>

It seems like everyone on that issue is in favor or at least +0. So +1 from
me too.


> I would be happy with the proposed name `__array_ufunc__`, although
> something more descriptive like `__handle_ufunc__` might be better.
>



> This is a wonderful opportunity for bike shedding for those so inclined ;)
>

pass :)

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161107/aac1a1c3/attachment.html>

From ralf.gommers at gmail.com  Mon Nov  7 03:19:20 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 7 Nov 2016 21:19:20 +1300
Subject: [Numpy-discussion] __numpy_ufunc__
In-Reply-To: <CAJXewOkyud6UMx-nfALpj5Gsg5nfeNb1=iUAQfEAr+JyotSoaQ@mail.gmail.com>
References: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
 <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
 <CAJXewOkyud6UMx-nfALpj5Gsg5nfeNb1=iUAQfEAr+JyotSoaQ@mail.gmail.com>
Message-ID: <CABL7CQgWBURp=W_+K90ypTjR7nu6P+s6Li=DLkCD9X+dO60_Dg@mail.gmail.com>

On Mon, Nov 7, 2016 at 9:21 AM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> I'm interested in this, but was never able to dive in to the lengthy
> discussions on this. I'm curious if there is a summary of the current
> state of things somewhere that I could read to catch up.


Old proposal, original implementation:
https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst

Main issue for discussion: https://github.com/numpy/numpy/issues/5844. That
really needs a summary, but I'm not sure there's an up-to-date one.

Ralf



> On Sunday, November 6, 2016, Charles R Harris <charlesr.harris at gmail.com>
> wrote:
>
>>
>>
>> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>>> request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely
>>> we will want to make some changes up front before merging that, so some
>>> discussion is in order.
>>>
>>>
>> As a first order of business, let's decide whether to remove the index
>> and rename `__numpy_ufunc__`. The motivation for this is discussed in issue
>> #5986. <https://github.com/numpy/numpy/issues/5986>
>> If we decide positive on that (I'm in favor), I would be happy with the
>> proposed name `__array_ufunc__`, although something more descriptive like
>> `__handle_ufunc__` might be better. This is a wonderful opportunity for
>> bike shedding for those so inclined ;)
>>
>> Chuck
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161107/9acaee47/attachment.html>

From matti.picus at gmail.com  Mon Nov  7 13:32:54 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 7 Nov 2016 20:32:54 +0200
Subject: [Numpy-discussion] Numpy 1.12.x branched
In-Reply-To: <mailman.100854.1478506763.2321.numpy-discussion@scipy.org>
References: <mailman.100854.1478506763.2321.numpy-discussion@scipy.org>
Message-ID: <638b75a9-7d9b-5ddf-6b3a-fadf5a12c65a@gmail.com>

On 07/11/16 10:19, numpy-discussion-request at scipy.org wrote:
> Date: Sun, 06 Nov 2016 17:56:12 +0100
> From: Sebastian Berg<sebastian at sipsolutions.net>
> To:numpy-discussion at scipy.org
> Subject: Re: [Numpy-discussion] Numpy 1.12.x branched
> Message-ID:<1478451372.3875.5.camel at sipsolutions.net>
> Content-Type: text/plain; charset="utf-8"
>
> On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote:
>> >Hi All,
>> >
>> >Numpy 1.12.x has been branched and the 1.13 development branch is
>> >open. It would be helpful if folks could review the release notes as
>> >it is likely I've missed something.? I'd like to make the first beta
>> >release in a couple of days.
>> >
> Very cool, thanks for all the hard work!
>
> - Sebastian
>
>
>> >Chuck
Thanks for managing this. I don't know where, but it would be nice if 
the release notes could mention the PyPy support - we are down to only a 
few failures on the test suite, the only real oustanding issue is nditer 
using UPDATEIFCOPY which depends on refcounting semantics to trigger the 
copy. Other than that PyPy + NumPy 1.12 is a working thing, we (PyPy 
devs) will soon try to make it work faster :).
Matti


From charlesr.harris at gmail.com  Mon Nov  7 14:57:29 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 7 Nov 2016 12:57:29 -0700
Subject: [Numpy-discussion] __numpy_ufunc__
In-Reply-To: <CABL7CQigiTMZ3kYgx5DGX9ru4qN35H1CwVsOh=nYo7+iLfWXGw@mail.gmail.com>
References: <CAB6mnxLivh54Su+zcyuW_jo0u1ss4XSid_UvV2UY1eP+QfDiTA@mail.gmail.com>
 <CAB6mnx+nwpFWmBKcs7-FS70pkyoBKwSGzEUQhE_M89d5nXeWmg@mail.gmail.com>
 <CABL7CQigiTMZ3kYgx5DGX9ru4qN35H1CwVsOh=nYo7+iLfWXGw@mail.gmail.com>
Message-ID: <CAB6mnxKXtKuCJCQ5H8Y5Fhn40cFq_ysHS1SkAqKGXKFa7K_6bw@mail.gmail.com>

On Mon, Nov 7, 2016 at 1:08 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Mon, Nov 7, 2016 at 9:10 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Sun, Nov 6, 2016 at 11:44 AM, Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> For those interested in continuing the __numpy_ufunc__ saga, there is a pull
>>> request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely
>>> we will want to make some changes up front before merging that, so some
>>> discussion is in order.
>>>
>>>
>> As a first order of business, let's decide whether to remove the index
>> and rename `__numpy_ufunc__`. The motivation for this is discussed in issue
>> #5986. <https://github.com/numpy/numpy/issues/5986>
>> If we decide positive on that (I'm in favor),
>>
>
> It seems like everyone on that issue is in favor or at least +0. So +1
> from me too.
>
>
>> I would be happy with the proposed name `__array_ufunc__`, although
>> something more descriptive like `__handle_ufunc__` might be better.
>>
>
>
>
>> This is a wonderful opportunity for bike shedding for those so inclined ;)
>>
>
Let me try to break things down an bit.


*Uncontroversial*


   - Rename __numpy_ufunc__ to __array_ufunc__
   - Remove index
   - Out is always a tuple


I think this much is useful even if nothing else is done.

*Goals*


   - Deprecate __array_priority__
   - Ufuncs should succeed or error, never return NotImplemented
   - Add __array_ufunc__ stub to ndarray.


I don't think these are controversial either, but they are longer term
except possibly the last. Note that never returning NotImplemented
disentangles ufuncs from ndarray binops, which I think is a good thing.

*Binops*


Here we come to the crux of the last arguments. The functions used for
binops can currently be set dynamically, the method that is currently used
to set them when the ufunc module is loaded. I think we want to do away
with that at some point and fix a set of ufuncs with which they are
implemented. This allows folks to overide the binop behavior using
__array_ufunc__. I think that is mostly of interest to people who are
subclassing ndarray and with that restriction doesn't bother me except that
it entangles ufuncs with binops. However, what I'd like to see as well is
an opt out for objects that don't care about ufuncs, but want to override
the python numeric operators, something simple like `__me_me_me__`, or,
more seriously, `__array_opt_out__`  that will only come into play if the
defining object is on the right hand side of an instance of ndarray. In
that case the binop would return NotImplemented so as to defer to the
Python machinery.  Note that  __array_priority__ is currently (ab)used for
this.

*Numpy scalars*

Numpy scalars default to the corresponding PyArray_Type or
PyGenericArrType_Type unless both arguments can be converted to the same c
type as the calling scalar, so I don't think there is a problem there. Note
that they do check _array_priority_ before trying to convert unknown
objects to array scalars in a fallback case.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161107/554755bd/attachment.html>

From charlesr.harris at gmail.com  Mon Nov  7 14:59:48 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 7 Nov 2016 12:59:48 -0700
Subject: [Numpy-discussion] Numpy 1.12.x branched
In-Reply-To: <638b75a9-7d9b-5ddf-6b3a-fadf5a12c65a@gmail.com>
References: <mailman.100854.1478506763.2321.numpy-discussion@scipy.org>
 <638b75a9-7d9b-5ddf-6b3a-fadf5a12c65a@gmail.com>
Message-ID: <CAB6mnxLcbnJQHm9Q9XSt-je3466YsgcFq_fam0QqY3vQu_Pvjw@mail.gmail.com>

On Mon, Nov 7, 2016 at 11:32 AM, Matti Picus <matti.picus at gmail.com> wrote:

> On 07/11/16 10:19, numpy-discussion-request at scipy.org wrote:
>
>> Date: Sun, 06 Nov 2016 17:56:12 +0100
>> From: Sebastian Berg<sebastian at sipsolutions.net>
>> To:numpy-discussion at scipy.org
>> Subject: Re: [Numpy-discussion] Numpy 1.12.x branched
>> Message-ID:<1478451372.3875.5.camel at sipsolutions.net>
>> Content-Type: text/plain; charset="utf-8"
>>
>> On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote:
>>
>>> >Hi All,
>>> >
>>> >Numpy 1.12.x has been branched and the 1.13 development branch is
>>> >open. It would be helpful if folks could review the release notes as
>>> >it is likely I've missed something.? I'd like to make the first beta
>>> >release in a couple of days.
>>> >
>>>
>> Very cool, thanks for all the hard work!
>>
>> - Sebastian
>>
>>
>> >Chuck
>>>
>> Thanks for managing this. I don't know where, but it would be nice if the
> release notes could mention the PyPy support - we are down to only a few
> failures on the test suite, the only real oustanding issue is nditer using
> UPDATEIFCOPY which depends on refcounting semantics to trigger the copy.
> Other than that PyPy + NumPy 1.12 is a working thing, we (PyPy devs) will
> soon try to make it work faster :).
>

A PR updating the release notes would be welcome. This might be one of the
highlights for those interested in PyPy.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161107/3f23239c/attachment.html>

From frederic.bastien at gmail.com  Thu Nov 10 11:06:34 2016
From: frederic.bastien at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCYXN0aWVu?=)
Date: Thu, 10 Nov 2016 11:06:34 -0500
Subject: [Numpy-discussion] Numpy 1.12.x branched
In-Reply-To: <CAB6mnxLcbnJQHm9Q9XSt-je3466YsgcFq_fam0QqY3vQu_Pvjw@mail.gmail.com>
References: <mailman.100854.1478506763.2321.numpy-discussion@scipy.org>
 <638b75a9-7d9b-5ddf-6b3a-fadf5a12c65a@gmail.com>
 <CAB6mnxLcbnJQHm9Q9XSt-je3466YsgcFq_fam0QqY3vQu_Pvjw@mail.gmail.com>
Message-ID: <CADKKbtiFspvE-PNaC6GnXJWvvewv9_zf38vE5hjCfa+9GCGgJA@mail.gmail.com>

My change about numpy.mean in float16 aren't in the doc.

Should I make a PR again numpy master or maintenance/1.12.x?

Fred

On Mon, Nov 7, 2016 at 2:59 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Mon, Nov 7, 2016 at 11:32 AM, Matti Picus <matti.picus at gmail.com>
> wrote:
>
>> On 07/11/16 10:19, numpy-discussion-request at scipy.org wrote:
>>
>>> Date: Sun, 06 Nov 2016 17:56:12 +0100
>>> From: Sebastian Berg<sebastian at sipsolutions.net>
>>> To:numpy-discussion at scipy.org
>>> Subject: Re: [Numpy-discussion] Numpy 1.12.x branched
>>> Message-ID:<1478451372.3875.5.camel at sipsolutions.net>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote:
>>>
>>>> >Hi All,
>>>> >
>>>> >Numpy 1.12.x has been branched and the 1.13 development branch is
>>>> >open. It would be helpful if folks could review the release notes as
>>>> >it is likely I've missed something.? I'd like to make the first beta
>>>> >release in a couple of days.
>>>> >
>>>>
>>> Very cool, thanks for all the hard work!
>>>
>>> - Sebastian
>>>
>>>
>>> >Chuck
>>>>
>>> Thanks for managing this. I don't know where, but it would be nice if
>> the release notes could mention the PyPy support - we are down to only a
>> few failures on the test suite, the only real oustanding issue is nditer
>> using UPDATEIFCOPY which depends on refcounting semantics to trigger the
>> copy. Other than that PyPy + NumPy 1.12 is a working thing, we (PyPy devs)
>> will soon try to make it work faster :).
>>
>
> A PR updating the release notes would be welcome. This might be one of the
> highlights for those interested in PyPy.
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161110/e4e1c78d/attachment.html>

From charlesr.harris at gmail.com  Thu Nov 10 13:30:13 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 10 Nov 2016 11:30:13 -0700
Subject: [Numpy-discussion] Numpy 1.12.x branched
In-Reply-To: <CADKKbtiFspvE-PNaC6GnXJWvvewv9_zf38vE5hjCfa+9GCGgJA@mail.gmail.com>
References: <mailman.100854.1478506763.2321.numpy-discussion@scipy.org>
 <638b75a9-7d9b-5ddf-6b3a-fadf5a12c65a@gmail.com>
 <CAB6mnxLcbnJQHm9Q9XSt-je3466YsgcFq_fam0QqY3vQu_Pvjw@mail.gmail.com>
 <CADKKbtiFspvE-PNaC6GnXJWvvewv9_zf38vE5hjCfa+9GCGgJA@mail.gmail.com>
Message-ID: <CAB6mnxJDkMTV4KuuJmyDXk6YdKMXA90U7oiWGL0a3QC7dWZtjA@mail.gmail.com>

On Thu, Nov 10, 2016 at 9:06 AM, Fr?d?ric Bastien <
frederic.bastien at gmail.com> wrote:

> My change about numpy.mean in float16 aren't in the doc.
>
> Should I make a PR again numpy master or maintenance/1.12.x?
>

Make it against master. I may cut and paste the content into a bigger PR I
will merge before the first beta.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161110/3535dc07/attachment.html>

From harrigan.matthew at gmail.com  Fri Nov 11 11:25:58 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Fri, 11 Nov 2016 11:25:58 -0500
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <1478295185.4986.0.camel@sipsolutions.net>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
Message-ID: <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>

I started a ufunc to compute the sum of square differences here
<https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.
It is about 4x faster and uses half the memory compared to
np.sum(np.square(x-c)).  This could significantly speed up common
operations like std and var (where c=np.mean(x).  It faster because its a
single pass instead of 3, and also because the inner loop is specialized
for the common reduce case, which might be an optimization considering more
generally.

I think I have answered my question #2&3 above.

Can someone please point me to an example where "data" was used in a ufunc
inner loop?  How can that value be set at runtime?  Thanks

On Fri, Nov 4, 2016 at 5:33 PM, Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Fr, 2016-11-04 at 15:42 -0400, Matthew Harrigan wrote:
> > I didn't notice identity before.  Seems like frompyfunc always sets
> > it to None.  If it were zero maybe it would work as desired here.
> >
> > In the writing your own ufunc doc, I was wondering if the pointer to
> > data could be used to get a constant at runtime.  If not, what could
> > that be used for?
> > static void double_logit(char **args, npy_intp *dimensions,
> >                             npy_intp* steps, void* data)
> > Why would the numerical accuracy be any different?  The subtraction
> > and square operations look identical and I thought np.sum just calls
> > np.add.reduce, so the reduction step uses the same code and would
> > therefore have the same accuracy.
> >
>
> Sorry, did not read it carefully, I guess `c` is the mean, so you are
> doing the two pass method.
>
> - Sebastian
>
>
> > Thanks
> >
> > On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg <sebastian at sipsolution
> > s.net> wrote:
> > > On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> > > > I was reading this and got thinking about if a ufunc could
> > > compute
> > > > the sum of squared differences in a single pass without a
> > > temporary
> > > > array.  The python code below demonstrates a possible approach.
> > > >
> > > > import numpy as np
> > > > x = np.arange(10)
> > > > c = 1.0
> > > > def add_square_diff(x1, x2):
> > > >     return x1 + (x2-c)**2
> > > > ufunc = np.frompyfunc(add_square_diff, 2, 1)
> > > > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> > > > print(np.sum(np.square(x-c)))
> > > >
> > > > I have (at least) 4 questions:
> > > > 1. Is it possible to pass run time constants to a ufunc written
> > > in C
> > > > for use in its inner loop, and if so how?
> > >
> > > I don't think its anticipated, since a ufunc could in most cases
> > > use a
> > > third argument, but a 3 arg ufunc can't be reduced. Not sure if
> > > there
> > > might be some trickery possible.
> > >
> > > > 2. Is it possible to pass an initial value to reduce to avoid the
> > > > clean up required for the first element?
> > >
> > > This is the identity normally. But the identity can only be 0, 1 or
> > > -1
> > > right now I think. The identity is what the output array gets
> > > initialized with (which effectively makes it the first value passed
> > > into the inner loop).
> > >
> > > > 3. Does that ufunc work, or are there special cases which cause
> > > it to
> > > > fall apart?
> > > > 4. Would a very specialized ufunc such as this be considered for
> > > > incorporating in numpy since it would help reduce time and memory
> > > of
> > > > functions already in numpy?
> > > >
> > >
> > > Might be mixing up things, however, IIRC the single pass approach
> > > has a
> > > bad numerical accuracy, so that I doubt that it is a good default
> > > algorithm.
> > >
> > > - Sebastian
> > >
> > >
> > > > Thank you,
> > > > Matt
> > > > _______________________________________________
> > > > NumPy-Discussion mailing list
> > > > NumPy-Discussion at scipy.org
> > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at scipy.org
> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161111/338b4178/attachment.html>

From jtaylor.debian at googlemail.com  Fri Nov 11 13:38:51 2016
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 11 Nov 2016 19:38:51 +0100
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
Message-ID: <db2e2f96-ddc0-b25e-0645-0631bcbb0572@googlemail.com>

here is an old unfinished PR adding max_abs which has a similar purpose
as yours:
https://github.com/numpy/numpy/pull/5509

So far I know data is set during runtime construction and is part of the
ufunc object, so it is not really very suitable to pass in constants on
call.
Its intended purpose is pass in functions to be called in generic loops
like PyUFunc_d_d.

Having an option to mark arguments of a ufunc as special in reductions
could be useful, e.g. it would allow a potential
(fused-)multiply-and-add ufunc to be used to implement a weighted sum.

On 11.11.2016 17:25, Matthew Harrigan wrote:
> I started a ufunc to compute the sum of square differences here
> <https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.  It
> is about 4x faster and uses half the memory compared to
> np.sum(np.square(x-c)).  This could significantly speed up common
> operations like std and var (where c=np.mean(x).  It faster because its
> a single pass instead of 3, and also because the inner loop is
> specialized for the common reduce case, which might be an optimization
> considering more generally.
> 
> I think I have answered my question #2&3 above.
> 
> Can someone please point me to an example where "data" was used in a
> ufunc inner loop?  How can that value be set at runtime?  Thanks
> 
> On Fri, Nov 4, 2016 at 5:33 PM, Sebastian Berg
> <sebastian at sipsolutions.net <mailto:sebastian at sipsolutions.net>> wrote:
> 
>     On Fr, 2016-11-04 at 15:42 -0400, Matthew Harrigan wrote:
>     > I didn't notice identity before.  Seems like frompyfunc always sets
>     > it to None.  If it were zero maybe it would work as desired here.
>     >
>     > In the writing your own ufunc doc, I was wondering if the pointer to
>     > data could be used to get a constant at runtime.  If not, what could
>     > that be used for?
>     > static void double_logit(char **args, npy_intp *dimensions,
>     >                             npy_intp* steps, void* data)
>     > Why would the numerical accuracy be any different?  The subtraction
>     > and square operations look identical and I thought np.sum just calls
>     > np.add.reduce, so the reduction step uses the same code and would
>     > therefore have the same accuracy.
>     >
> 
>     Sorry, did not read it carefully, I guess `c` is the mean, so you are
>     doing the two pass method.
> 
>     - Sebastian
> 
> 
>     > Thanks
>     >
>     > On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg <sebastian at sipsolution
>     > s.net <http://s.net>> wrote:
>     > > On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
>     > > > I was reading this and got thinking about if a ufunc could
>     > > compute
>     > > > the sum of squared differences in a single pass without a
>     > > temporary
>     > > > array.  The python code below demonstrates a possible approach.
>     > > >
>     > > > import numpy as np
>     > > > x = np.arange(10)
>     > > > c = 1.0
>     > > > def add_square_diff(x1, x2):
>     > > >     return x1 + (x2-c)**2
>     > > > ufunc = np.frompyfunc(add_square_diff, 2, 1)
>     > > > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
>     > > > print(np.sum(np.square(x-c)))
>     > > >
>     > > > I have (at least) 4 questions:
>     > > > 1. Is it possible to pass run time constants to a ufunc written
>     > > in C
>     > > > for use in its inner loop, and if so how?
>     > >
>     > > I don't think its anticipated, since a ufunc could in most cases
>     > > use a
>     > > third argument, but a 3 arg ufunc can't be reduced. Not sure if
>     > > there
>     > > might be some trickery possible.
>     > >
>     > > > 2. Is it possible to pass an initial value to reduce to avoid the
>     > > > clean up required for the first element?
>     > >
>     > > This is the identity normally. But the identity can only be 0, 1 or
>     > > -1
>     > > right now I think. The identity is what the output array gets
>     > > initialized with (which effectively makes it the first value passed
>     > > into the inner loop).
>     > >
>     > > > 3. Does that ufunc work, or are there special cases which cause
>     > > it to
>     > > > fall apart?
>     > > > 4. Would a very specialized ufunc such as this be considered for
>     > > > incorporating in numpy since it would help reduce time and memory
>     > > of
>     > > > functions already in numpy?
>     > > >
>     > >
>     > > Might be mixing up things, however, IIRC the single pass approach
>     > > has a
>     > > bad numerical accuracy, so that I doubt that it is a good default
>     > > algorithm.
>     > >
>     > > - Sebastian
>     > >
>     > >
>     > > > Thank you,
>     > > > Matt
>     > > > _______________________________________________
>     > > > NumPy-Discussion mailing list
>     > > > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
>     > > _______________________________________________
>     > > NumPy-Discussion mailing list
>     > > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
>     > >
>     > _______________________________________________
>     > NumPy-Discussion mailing list
>     > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> 
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     https://mail.scipy.org/mailman/listinfo/numpy-discussion
>     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> 
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 



From harrigan.matthew at gmail.com  Fri Nov 11 16:06:15 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Fri, 11 Nov 2016 16:06:15 -0500
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <db2e2f96-ddc0-b25e-0645-0631bcbb0572@googlemail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <db2e2f96-ddc0-b25e-0645-0631bcbb0572@googlemail.com>
Message-ID: <CAOfRF=g3YnsbVM6tTKh-qYB7qAFmKD9dShZ7Tfhz6rhUt4qEWw@mail.gmail.com>

My possibly poorly thought out API would be something like
np.add_sq_diff.set_c(42.0).reduce(x), basically add a setter method which
returned self.  Thanks for explaining what data was intended to do.  All of
this is really just a dance around the requirement that reduce only works
on binary functions.  Would there be any interest in extending reduce to
allow for arbitrary numbers of input and output arguments?

On Fri, Nov 11, 2016 at 1:38 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> here is an old unfinished PR adding max_abs which has a similar purpose
> as yours:
> https://github.com/numpy/numpy/pull/5509
>
> So far I know data is set during runtime construction and is part of the
> ufunc object, so it is not really very suitable to pass in constants on
> call.
> Its intended purpose is pass in functions to be called in generic loops
> like PyUFunc_d_d.
>
> Having an option to mark arguments of a ufunc as special in reductions
> could be useful, e.g. it would allow a potential
> (fused-)multiply-and-add ufunc to be used to implement a weighted sum.
>
> On 11.11.2016 17:25, Matthew Harrigan wrote:
> > I started a ufunc to compute the sum of square differences here
> > <https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.
> It
> > is about 4x faster and uses half the memory compared to
> > np.sum(np.square(x-c)).  This could significantly speed up common
> > operations like std and var (where c=np.mean(x).  It faster because its
> > a single pass instead of 3, and also because the inner loop is
> > specialized for the common reduce case, which might be an optimization
> > considering more generally.
> >
> > I think I have answered my question #2&3 above.
> >
> > Can someone please point me to an example where "data" was used in a
> > ufunc inner loop?  How can that value be set at runtime?  Thanks
> >
> > On Fri, Nov 4, 2016 at 5:33 PM, Sebastian Berg
> > <sebastian at sipsolutions.net <mailto:sebastian at sipsolutions.net>> wrote:
> >
> >     On Fr, 2016-11-04 at 15:42 -0400, Matthew Harrigan wrote:
> >     > I didn't notice identity before.  Seems like frompyfunc always sets
> >     > it to None.  If it were zero maybe it would work as desired here.
> >     >
> >     > In the writing your own ufunc doc, I was wondering if the pointer
> to
> >     > data could be used to get a constant at runtime.  If not, what
> could
> >     > that be used for?
> >     > static void double_logit(char **args, npy_intp *dimensions,
> >     >                             npy_intp* steps, void* data)
> >     > Why would the numerical accuracy be any different?  The subtraction
> >     > and square operations look identical and I thought np.sum just
> calls
> >     > np.add.reduce, so the reduction step uses the same code and would
> >     > therefore have the same accuracy.
> >     >
> >
> >     Sorry, did not read it carefully, I guess `c` is the mean, so you are
> >     doing the two pass method.
> >
> >     - Sebastian
> >
> >
> >     > Thanks
> >     >
> >     > On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg
> <sebastian at sipsolution
> >     > s.net <http://s.net>> wrote:
> >     > > On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> >     > > > I was reading this and got thinking about if a ufunc could
> >     > > compute
> >     > > > the sum of squared differences in a single pass without a
> >     > > temporary
> >     > > > array.  The python code below demonstrates a possible approach.
> >     > > >
> >     > > > import numpy as np
> >     > > > x = np.arange(10)
> >     > > > c = 1.0
> >     > > > def add_square_diff(x1, x2):
> >     > > >     return x1 + (x2-c)**2
> >     > > > ufunc = np.frompyfunc(add_square_diff, 2, 1)
> >     > > > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> >     > > > print(np.sum(np.square(x-c)))
> >     > > >
> >     > > > I have (at least) 4 questions:
> >     > > > 1. Is it possible to pass run time constants to a ufunc written
> >     > > in C
> >     > > > for use in its inner loop, and if so how?
> >     > >
> >     > > I don't think its anticipated, since a ufunc could in most cases
> >     > > use a
> >     > > third argument, but a 3 arg ufunc can't be reduced. Not sure if
> >     > > there
> >     > > might be some trickery possible.
> >     > >
> >     > > > 2. Is it possible to pass an initial value to reduce to avoid
> the
> >     > > > clean up required for the first element?
> >     > >
> >     > > This is the identity normally. But the identity can only be 0, 1
> or
> >     > > -1
> >     > > right now I think. The identity is what the output array gets
> >     > > initialized with (which effectively makes it the first value
> passed
> >     > > into the inner loop).
> >     > >
> >     > > > 3. Does that ufunc work, or are there special cases which cause
> >     > > it to
> >     > > > fall apart?
> >     > > > 4. Would a very specialized ufunc such as this be considered
> for
> >     > > > incorporating in numpy since it would help reduce time and
> memory
> >     > > of
> >     > > > functions already in numpy?
> >     > > >
> >     > >
> >     > > Might be mixing up things, however, IIRC the single pass approach
> >     > > has a
> >     > > bad numerical accuracy, so that I doubt that it is a good default
> >     > > algorithm.
> >     > >
> >     > > - Sebastian
> >     > >
> >     > >
> >     > > > Thank you,
> >     > > > Matt
> >     > > > _______________________________________________
> >     > > > NumPy-Discussion mailing list
> >     > > > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> >     > > _______________________________________________
> >     > > NumPy-Discussion mailing list
> >     > > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> >     > >
> >     > _______________________________________________
> >     > NumPy-Discussion mailing list
> >     > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> >
> >     _______________________________________________
> >     NumPy-Discussion mailing list
> >     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> >
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161111/8d83ff1d/attachment.html>

From oleksandr.pavlyk at intel.com  Sat Nov 12 12:00:07 2016
From: oleksandr.pavlyk at intel.com (Pavlyk, Oleksandr)
Date: Sat, 12 Nov 2016 17:00:07 +0000
Subject: [Numpy-discussion] Ensuring one can operate on array-like argument
	in place
Message-ID: <4C9EDA7282E297428F3986994EB0FBD387DFB8@ORSMSX110.amr.corp.intel.com>

Hi,

In my Cython code a function processes it argument x as follows:

    x_arr = PyArray_CheckFromAny(
          x, NULL, 0, 0,
          cnp.NPY_ELEMENTSTRIDES | cnp.NPY_ENSUREARRAY | cnp.NPY_NOTSWAPPED, NULL)

    if x_arr is not x:
       in_place = 1  # a copy was made, so we can work in place.

The logic is of the last line turns out to be incorrect, because the input x can be a class with an array interface:

class FakeArray(object):
    def __init__(self, data):
        self._data = data
        self.__array_interface__ = data.__array_interface__

Feeding my function FakeArray(xx),  x_arr will point into the content of xx, resulting in unwarranted content
overwrite of xx.

I am trying to replace that condition with

    if x_arr is not x and cnp.PyArray_Check(x):
       # a copy was made, so we can work in place.
       in_place = 1 if cnp.PyArray_CHKFLAGS(x_arr, cnp.NPY_WRITEABLE) else 0

I am wondering if I perhaps overlooked some case.

Thank you,
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161112/6de13f31/attachment.html>

From pav at iki.fi  Sun Nov 13 09:29:06 2016
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 13 Nov 2016 14:29:06 +0000 (UTC)
Subject: [Numpy-discussion] Ensuring one can operate on array-like
	argument	in place
References: <4C9EDA7282E297428F3986994EB0FBD387DFB8@ORSMSX110.amr.corp.intel.com>
Message-ID: <o09tbi$c33$1@blaine.gmane.org>

Sat, 12 Nov 2016 17:00:07 +0000, Pavlyk, Oleksandr kirjoitti:
[clip]
>     if x_arr is not x:
>        in_place = 1  # a copy was made, so we can work in place.
> 
> The logic is of the last line turns out to be incorrect, because the
> input x can be a class with an array interface.

Please see:

https://github.com/scipy/scipy/blob/master/scipy/linalg/misc.py#L169

This probably can be translated to equivalent Numpy C API calls.



From Jerome.Kieffer at esrf.fr  Mon Nov 14 03:38:09 2016
From: Jerome.Kieffer at esrf.fr (Jerome Kieffer)
Date: Mon, 14 Nov 2016 09:38:09 +0100
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
Message-ID: <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>

On Fri, 11 Nov 2016 11:25:58 -0500
Matthew Harrigan <harrigan.matthew at gmail.com> wrote:

> I started a ufunc to compute the sum of square differences here
> <https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.
> It is about 4x faster and uses half the memory compared to
> np.sum(np.square(x-c)). 

Hi Matt,

Using *blas* you win already a factor two (maybe more depending on you blas implementation):

% python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))" "np.sum(np.square(x-2.))"
10 loops, best of 3: 135 msec per loop

% python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))" "y=x-2.;np.dot(y,y)"
10 loops, best of 3: 70.2 msec per loop


Cheers,
-- 
J?r?me Kieffer


From e.antero.tammi at gmail.com  Mon Nov 14 15:38:25 2016
From: e.antero.tammi at gmail.com (eat)
Date: Mon, 14 Nov 2016 22:38:25 +0200
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
Message-ID: <CAKa=AYQhZ5kMxWeJMpHMj=AkS7btvBijvg1wynt7wf5iAe2c1w@mail.gmail.com>

Yeah,

but it's not so obvious what's happening "under the hoods". Consider this
(with an old Win7 machine):
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64
bit (AMD64)]
np.__version__
'1.11.1'

On Mon, Nov 14, 2016 at 10:38 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr>
wrote:

> On Fri, 11 Nov 2016 11:25:58 -0500
> Matthew Harrigan <harrigan.matthew at gmail.com> wrote:
>
> > I started a ufunc to compute the sum of square differences here
> > <https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.
> > It is about 4x faster and uses half the memory compared to
> > np.sum(np.square(x-c)).
>
> Hi Matt,
>
> Using *blas* you win already a factor two (maybe more depending on you
> blas implementation):
>
> % python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))"
> "np.sum(np.square(x-2.))"
> 10 loops, best of 3: 135 msec per loop
>
> % python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))"
> "y=x-2.;np.dot(y,y)"
> 10 loops, best of 3: 70.2 msec per loop
>
x= np.linspace(0, 1, int(1e6))

timeit np.sum(np.square(x- 2.))
10 loops, best of 3: 23 ms per loop

y= x- 2.

timeit np.dot(y, y)
The slowest run took 18.60 times longer than the fastest. This could mean
that an intermediate result is being cached.
1000 loops, best of 3: 1.78 ms per loop

timeit np.dot(y, y)
1000 loops, best of 3: 1.73 ms per loop

Best,
eat

>
>
> Cheers,
> --
> J?r?me Kieffer
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161114/432242a2/attachment.html>

From chris.barker at noaa.gov  Mon Nov 14 18:20:53 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 14 Nov 2016 15:20:53 -0800
Subject: [Numpy-discussion] Ensuring one can operate on array-like
 argument in place
In-Reply-To: <4C9EDA7282E297428F3986994EB0FBD387DFB8@ORSMSX110.amr.corp.intel.com>
References: <4C9EDA7282E297428F3986994EB0FBD387DFB8@ORSMSX110.amr.corp.intel.com>
Message-ID: <CALGmxEL+JwRqm5oYbWnzs9nB+SH0-XfLrXnNopaumdN3n5Xjbg@mail.gmail.com>

I tend to use ndarray.copy() in python code -- no reason you couldn't do
the same in Cython.

If you want to take any array-like object that may not have a copy()
method, you could call asanyarray() first:

-CHB





On Sat, Nov 12, 2016 at 9:00 AM, Pavlyk, Oleksandr <
oleksandr.pavlyk at intel.com> wrote:

> Hi,
>
>
>
> In my Cython code a function processes it argument x as follows:
>
>
>
>     x_arr = PyArray_CheckFromAny(
>
>           x, NULL, 0, 0,
>
>           cnp.NPY_ELEMENTSTRIDES | cnp.NPY_ENSUREARRAY |
> cnp.NPY_NOTSWAPPED, NULL)
>
>
>
>     if x_arr is not x:
>
>        in_place = 1  # a copy was made, so we can work in place.
>
>
>
> The logic is of the last line turns out to be incorrect, because the input
> x can be a class with an array interface:
>
>
>
> class FakeArray(object):
>
>     def __init__(self, data):
>
>         self._data = data
>
>         self.__array_interface__ = data.__array_interface__
>
>
>
> Feeding my function FakeArray(xx),  x_arr will point into the content of
> xx, resulting in unwarranted content
> overwrite of xx.
>
>
>
> I am trying to replace that condition with
>
>
>
>     if x_arr is not x and cnp.PyArray_Check(x):
>
>        # a copy was made, so we can work in place.
>
>        in_place = 1 if cnp.PyArray_CHKFLAGS(x_arr, cnp.NPY_WRITEABLE) else
> 0
>
>
>
> I am wondering if I perhaps overlooked some case.
>
>
>
> Thank you,
>
> Sasha
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161114/189f23cd/attachment.html>

From harrigan.matthew at gmail.com  Mon Nov 14 20:40:39 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Mon, 14 Nov 2016 20:40:39 -0500
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
Message-ID: <CAOfRF=g4kdsK0jLTkkAPAVi-bcprqkQ4JPLgB1-rARdBfnxPfA@mail.gmail.com>

Still slower and worse uses 2x the memory for the intermediate temporary
array.

I propose allowing implicit reductions with ufuncs.  Specifically if out is
provided with shape[axis] = 1, then pass it on to the ufunc with a stride
of 0.  That should allow this to work:

x = np.arange(10)
def add_square_diff(x1, x2, x3):
    return x1 + (x2-x3)**2
result  =np.zeros(1)
np.frompyfunc(add_square_diff, 3, 1)(result, x, np.mean(x), result)

Essentially it creates a reduce for a function which isn't binary.  I think
this would be generally useful.  For instance, finding the min and max in
one pass would be nice:

def minmax(x1, x2, x3):
    return min(x1,x3), max(x2,x3)
minn = np.array([np.inf])
maxx = np.array([-np.inf])
np.frompyfunc(minmax, 3, 2)(minn, maxx, x, minn, maxx)

Note it also allows for arbitrary initial values or identity to be
specified, possibly determined at run time.  I think this would make ufuncs
even more universal.

On Mon, Nov 14, 2016 at 3:38 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr>
wrote:

> On Fri, 11 Nov 2016 11:25:58 -0500
> Matthew Harrigan <harrigan.matthew at gmail.com> wrote:
>
> > I started a ufunc to compute the sum of square differences here
> > <https://gist.github.com/mattharrigan/6f678b3d6df5efd236fc23bfb59fd3bd>.
> > It is about 4x faster and uses half the memory compared to
> > np.sum(np.square(x-c)).
>
> Hi Matt,
>
> Using *blas* you win already a factor two (maybe more depending on you
> blas implementation):
>
> % python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))"
> "np.sum(np.square(x-2.))"
> 10 loops, best of 3: 135 msec per loop
>
> % python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))"
> "y=x-2.;np.dot(y,y)"
> 10 loops, best of 3: 70.2 msec per loop
>
>
> Cheers,
> --
> J?r?me Kieffer
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161114/34dc4ee0/attachment.html>

From shoyer at gmail.com  Mon Nov 14 20:49:30 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 14 Nov 2016 17:49:30 -0800
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAOfRF=g4kdsK0jLTkkAPAVi-bcprqkQ4JPLgB1-rARdBfnxPfA@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
 <CAOfRF=g4kdsK0jLTkkAPAVi-bcprqkQ4JPLgB1-rARdBfnxPfA@mail.gmail.com>
Message-ID: <CAEQ_Tvfzux-twtkB+75ceR6hOE34t-0QOaN_ushM4R5hmEZEWg@mail.gmail.com>

On Mon, Nov 14, 2016 at 5:40 PM, Matthew Harrigan <
harrigan.matthew at gmail.com> wrote:

> Essentially it creates a reduce for a function which isn't binary.  I
> think this would be generally useful.
>

NumPy already has a generic enough interface for creating such ufuncs. In
fact, it's called a "generalized ufunc":
https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html

I think you could already write "implicit reductions" using gufuncs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161114/a5873e74/attachment.html>

From stuart at stuartreynolds.net  Tue Nov 15 12:37:42 2016
From: stuart at stuartreynolds.net (Stuart Reynolds)
Date: Tue, 15 Nov 2016 09:37:42 -0800
Subject: [Numpy-discussion] subclassing ndarray and keeping same ufunc
	behavior
Message-ID: <CAAy-kdkeSvkg3KC5N-zFPFGw+yKQ=EsGsbAgGe4FonxdndURBQ@mail.gmail.com>

I'm trying to subclass an ndarray so that I can add some additional fields.
When I do this however, I get new odd behavior when my object is passed to
a variety of numpy functions. For example nanmin returns now return an
object of the type of my new array class, whereas previously I'd get a
float64. Why? Is this a bug with nanmin or my class?

import numpy as np
class NDArrayWithColumns(np.ndarray):
    def __new__(cls, obj, columns=None):
        obj = obj.view(cls)
        obj.columns = tuple(columns)
        return obj

    def __array_finalize__(self, obj):
        if obj is None: return
        self.columns = getattr(obj, 'columns', None)

NAN = float("nan")
r = np.array([1.,0.,1.,0.,1.,0.,1.,0.,NAN, 1., 1.])print "MIN",
np.nanmin(r), type(np.nanmin(r))

gives:

MIN 0.0 <type 'numpy.float64'>

but

>>> r = NDArrayWithColumns(r, ["a"])>>> print "MIN", np.nanmin(r), type(np.nanmin(r))
MIN 0.0 <class '__main__.NDArrayWithColumns'>>>> print r.shape   # ?!(11,)

Note the change in type, and also that str(np.nanmin(r)) shows 1 field, not
11 as indicated by its shape. This seems wrong. Is there a way to get my
subclass to behave more like an ndarray?

I realize from the docs that I can override __array_wrap__, but its not
clear me how how to use it to solve this issue. Or whether its the right
tool.

In case you're interested, I'm subclassing because I'd like to track column
names in matrices of a single type. This is pretty common wish in scikit
pipelines. Structured arrays and record type arrays allow for varying type.
Pandas provides this functionality, but dealing with numpy arrays is easier
(and more efficient) when writing cython extensions. Also, I think the
 structured arrays and record types are unlikely to play nice with cython
because they're more freely typed -- I want to deal exclusively with arrays
of doubles.

Any thoughts of how to subclass ndarray and keep original behavior in
ufuncs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161115/0e893cf5/attachment.html>

From m.h.vankerkwijk at gmail.com  Tue Nov 15 13:48:11 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Tue, 15 Nov 2016 13:48:11 -0500
Subject: [Numpy-discussion] subclassing ndarray and keeping same ufunc
	behavior
In-Reply-To: <CAAy-kdkeSvkg3KC5N-zFPFGw+yKQ=EsGsbAgGe4FonxdndURBQ@mail.gmail.com>
References: <CAAy-kdkeSvkg3KC5N-zFPFGw+yKQ=EsGsbAgGe4FonxdndURBQ@mail.gmail.com>
Message-ID: <CAJNV+9tuwGYDJcOwcU6TH-ckcD3MB7SHhAFbON9J0=H+Fv4h4A@mail.gmail.com>

Hi Stuart,

It certainly seems correct behaviour to return the subclass you
created: after all, you might want to keep the information on
`columns` (e.g., consider doing nanmin along a given axis). Indeed, we
certainly want to keep the unit in astropy's Quantity (which also is a
subclass of ndarray).

On the shape: shouldn't that be print(np.nanmin(r).shape)??

Overall, I think it is worth considering very carefully what exactly
you try to accomplish; if different elements along a given axis have
different meaning, I'm not sure it makes all that much sense to treat
them as a single array (e.g., np.sin might be useful for one column,
not not another). Even if pandas is slower, the advantage in clarity
of what is happening might well be more important in the long run.

All the best,

Marten

p.s. nanmin is not a ufunc; you can find it in numpy/lib/nan_functions.py


From nathan12343 at gmail.com  Tue Nov 15 13:52:35 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Tue, 15 Nov 2016 13:52:35 -0500
Subject: [Numpy-discussion] subclassing ndarray and keeping same ufunc
	behavior
In-Reply-To: <CAJNV+9tuwGYDJcOwcU6TH-ckcD3MB7SHhAFbON9J0=H+Fv4h4A@mail.gmail.com>
References: <CAAy-kdkeSvkg3KC5N-zFPFGw+yKQ=EsGsbAgGe4FonxdndURBQ@mail.gmail.com>
 <CAJNV+9tuwGYDJcOwcU6TH-ckcD3MB7SHhAFbON9J0=H+Fv4h4A@mail.gmail.com>
Message-ID: <CAJXewOmfN6EKh3j0FHv9ADfQDa1_1YPqUjPXXt2Dhts+_ZONvA@mail.gmail.com>

You might also want to consider writing a wrapper object that contains an
ndarray as a (possibly private) attribute and then presents different views
or interpretations of that array.

Subclassing ndarray is a pit of snakes, it's best to avoid it if you can (I
say as the author and maintainer of an ndarray subclass).

On Tue, Nov 15, 2016 at 1:48 PM, Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Hi Stuart,
>
> It certainly seems correct behaviour to return the subclass you
> created: after all, you might want to keep the information on
> `columns` (e.g., consider doing nanmin along a given axis). Indeed, we
> certainly want to keep the unit in astropy's Quantity (which also is a
> subclass of ndarray).
>
> On the shape: shouldn't that be print(np.nanmin(r).shape)??
>
> Overall, I think it is worth considering very carefully what exactly
> you try to accomplish; if different elements along a given axis have
> different meaning, I'm not sure it makes all that much sense to treat
> them as a single array (e.g., np.sin might be useful for one column,
> not not another). Even if pandas is slower, the advantage in clarity
> of what is happening might well be more important in the long run.
>
> All the best,
>
> Marten
>
> p.s. nanmin is not a ufunc; you can find it in numpy/lib/nan_functions.py
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161115/888e331b/attachment.html>

From stuart at stuartreynolds.net  Tue Nov 15 14:01:45 2016
From: stuart at stuartreynolds.net (Stuart Reynolds)
Date: Tue, 15 Nov 2016 11:01:45 -0800
Subject: [Numpy-discussion] subclassing ndarray and keeping same ufunc
	behavior
In-Reply-To: <CAJNV+9tuwGYDJcOwcU6TH-ckcD3MB7SHhAFbON9J0=H+Fv4h4A@mail.gmail.com>
References: <CAAy-kdkeSvkg3KC5N-zFPFGw+yKQ=EsGsbAgGe4FonxdndURBQ@mail.gmail.com>
 <CAJNV+9tuwGYDJcOwcU6TH-ckcD3MB7SHhAFbON9J0=H+Fv4h4A@mail.gmail.com>
Message-ID: <CAAy-kd==6LxO97KVFM4__1DTuwZOA28mX=-CnuRS5OE61axf0Q@mail.gmail.com>

Doh! Thanks for that.

On Tue, Nov 15, 2016 at 10:48 AM, Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Hi Stuart,
>
> It certainly seems correct behaviour to return the subclass you
> created: after all, you might want to keep the information on
> `columns` (e.g., consider doing nanmin along a given axis). Indeed, we
> certainly want to keep the unit in astropy's Quantity (which also is a
> subclass of ndarray).
>
> On the shape: shouldn't that be print(np.nanmin(r).shape)??
>
> Overall, I think it is worth considering very carefully what exactly
> you try to accomplish; if different elements along a given axis have
> different meaning, I'm not sure it makes all that much sense to treat
> them as a single array (e.g., np.sin might be useful for one column,
> not not another). Even if pandas is slower, the advantage in clarity
> of what is happening might well be more important in the long run.
>
> All the best,
>
> Marten
>
> p.s. nanmin is not a ufunc; you can find it in numpy/lib/nan_functions.py
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161115/d1cacb90/attachment.html>

From Jerome.Kieffer at esrf.fr  Wed Nov 16 03:29:24 2016
From: Jerome.Kieffer at esrf.fr (Jerome Kieffer)
Date: Wed, 16 Nov 2016 09:29:24 +0100
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAKa=AYQhZ5kMxWeJMpHMj=AkS7btvBijvg1wynt7wf5iAe2c1w@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
 <CAKa=AYQhZ5kMxWeJMpHMj=AkS7btvBijvg1wynt7wf5iAe2c1w@mail.gmail.com>
Message-ID: <20161116092924.79c2d82c@lintaillefer.esrf.fr>

On Mon, 14 Nov 2016 22:38:25 +0200
eat <e.antero.tammi at gmail.com> wrote:

> but it's not so obvious what's happening "under the hoods". Consider this
> (with an old Win7 machine):
> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64
> bit (AMD64)]
> np.__version__
> '1.11.1'

What matters is the "blas" library used under the hood, hence the
options passed to numpy at compile time. I notices 20x differences
depending on the blas version.

But more importantly:
* results are the same (at the limit of the numerical precision)
* dot() was always faster than sum(square()), varying from a bit to a lot.

I agree this may change in future version of numpy.

Cheers,
-- 
J?r?me Kieffer


From harrigan.matthew at gmail.com  Wed Nov 16 20:18:34 2016
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Wed, 16 Nov 2016 20:18:34 -0500
Subject: [Numpy-discussion] ufunc for sum of squared difference
In-Reply-To: <CAEQ_Tvfzux-twtkB+75ceR6hOE34t-0QOaN_ushM4R5hmEZEWg@mail.gmail.com>
References: <CAOfRF=i9siiU2H09MLLhAnyPJsG3JmMyv_CEs2iHwCheLV+ohg@mail.gmail.com>
 <1478282208.861.29.camel@sipsolutions.net>
 <CAOfRF=iHjuaOo==VRBuT8DUin6p79mE9x66AWgC+__=Z+uEkyQ@mail.gmail.com>
 <1478295185.4986.0.camel@sipsolutions.net>
 <CAOfRF=gw9kZii+uJD2mTK5S-0pjr2fYaB0S9KrFEXBqeQNfTOw@mail.gmail.com>
 <20161114093809.6ea2b3a0@lintaillefer.esrf.fr>
 <CAOfRF=g4kdsK0jLTkkAPAVi-bcprqkQ4JPLgB1-rARdBfnxPfA@mail.gmail.com>
 <CAEQ_Tvfzux-twtkB+75ceR6hOE34t-0QOaN_ushM4R5hmEZEWg@mail.gmail.com>
Message-ID: <CAOfRF=i-qi2kmFDChrhR84=HFdxZ-q7HHcj9zuD+bGy=J41pog@mail.gmail.com>

Thanks for pointing me to that.  I think its a great match for a 1D case,
like the inner product of two arrays in terms of signature.  I don't see
how it works with higher dimensional arrays, esp with something like the
axis parameter in ufunc.reduce.  With what I proposed for an array of shape
(M, N) for example, result.shape could be (1,) or (M, 1) or (1, N) for
reducing over the flattened array or either axis.  Can you do something
like that with gufunc or do you need to iterate gufunc calls over
slices/views?  Thanks again.

On Mon, Nov 14, 2016 at 8:49 PM, Stephan Hoyer <shoyer at gmail.com> wrote:

> On Mon, Nov 14, 2016 at 5:40 PM, Matthew Harrigan <
> harrigan.matthew at gmail.com> wrote:
>
>> Essentially it creates a reduce for a function which isn't binary.  I
>> think this would be generally useful.
>>
>
> NumPy already has a generic enough interface for creating such ufuncs. In
> fact, it's called a "generalized ufunc":
> https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html
>
> I think you could already write "implicit reductions" using gufuncs?
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161116/7d9d28b9/attachment.html>

From charlesr.harris at gmail.com  Thu Nov 17 00:47:39 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 16 Nov 2016 22:47:39 -0700
Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
Message-ID: <CAB6mnxLi87AsWQgK0M0_jhnTdnQQP-o3rp+hMNvGwQzjBLAi3w@mail.gmail.com>

Hi All,

I'm pleased to annouce the release of NumPy 1.12.0b1. This release
supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests
submitted by 133 contributors. It is quite sizeable and rather than put the
release notes inline I've attached them as a file and they may also be
viewed at Github <https://github.com/numpy/numpy/releases/tag/v1.12.0b1>.
Zip files and tarballs may also be found the Github link. Wheels and source
archives may be downloaded from PyPI, which is the recommended method.

This release is a large collection of fixes, enhancements, and improvements
and it is difficult to select just a few as highlights. However, the
following enhancements may be of particular interest

   - Order of operations in ``np.einsum`` now can be optimized for large
   speed improvements.
   - New ``signature`` argument to ``np.vectorize`` for vectorizing with
   core dimensions.
   - The ``keepdims`` argument was added to many functions.
   - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this
   is a milestone for PyPy's C-API compatibility layer.

Thanks to all,

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161116/0a07229b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1.12.0-notes.rst
Type: text/prs.fallenstein.rst
Size: 64965 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161116/0a07229b/attachment.bin>

From agballen189 at aol.com  Thu Nov 17 11:49:29 2016
From: agballen189 at aol.com (Andres Guzman-Ballen)
Date: Thu, 17 Nov 2016 11:49:29 -0500
Subject: [Numpy-discussion] The --compilers flag is ignored for build_ext
Message-ID: <15873316e96-5cdc-4a06@webprd-a11.mail.aol.com>

Hello Numpy community!


First time poster here. I was recommended by Charles Harris to present a question I have to you guys regarding using --compiler=intelem being ignored when I use build_ext.


I am trying to compile Numpy v1.11.2 that I am getting from https://pypi.python.org/packages/16/f5/b432f028134dd30cfbf6f21b8264a9938e5e0f75204e72453af08d67eb0b/numpy-1.11.2.tar.gz, using Python 2.7.12 on my Ubuntu 10.04 LTS Machine. I specifically want to compile this with the Intel compiler. I have sourced the compilervars.sh so the Intel compiler is in the path. This is the command that is failing:


'python setup.py build_ext --inplace --debug --force --compiler=intelem --fcompiler=intelem'. 


The reason it fails is because I have added compiler flags to umath's configuration extension code in numpy/core/setup.py as extra_compiler_arg and they are specifically meant for Intel. Since the compiler setup is ignored (unlike in setup.py build, where config_cc is correctly called and takes care of this), it defaults to GCC. This is a snippet of what you get for 'build_ext':
running build_clib
running build_src
build_src
building py_modules sources
building library "npymath" sources
creating build
creating build/src.linux-x86_64-2.7
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config


This is what you get for using 'build':
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
creating build
creating build/src.linux-x86_64-2.7
creating build/src.linux-x86_64-2.7/numpy
creating build/src.linux-x86_64-2.7/numpy/distutils
building library "npymath" sources
Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/icc
Could not locate executable ecc
customize IntelEM64TFCompiler
Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/ifort
customize IntelEM64TFCompiler using config


I do not experience this problem with 'python setup.py build --debug --force --compiler=intelem --fcompiler=intelem'.  because the configuration function is correctly called here: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/build.py#L11 (that is where I got the output from above). However, I do not see anything like this in build_ext: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/build_ext.py#L34. This of course could be because build_ext is doing something else below in the run command on line 109.


This is most likely not the right answer, but what harm does this do?

diff --git a/numpy/distutils/command/build_ext.py b/numpy/distutils/command/build_ext.py
index 0fa52a2..ffa88ae 100644
--- a/numpy/distutils/command/build_ext.py
+++ b/numpy/distutils/command/build_ext.py
@@ -79,6 +79,8 @@ class build_ext (old_build_ext):
             return


         # Make sure that extension sources are complete.
+        self.run_command('config_cc')
+        self.run_command('config_fc')
         self.run_command('build_src')


         if self.distribution.has_c_libraries():


I am able to get build_ext to behave properly by adding this snippet, but I'm pretty sure this is the wrong approach because there seems to be code below line 109 in numpy/distutils/command/build_ext.py that sets compiler-related things up.


This is the ticket I created under Numpy's issues page: https://github.com/numpy/numpy/issues/8283. Any feedback is very well appreciated and I'd be more than happy to help contribute and get this fixed for the community. I don't want to just throw a problem at you guys and expect someone else to fix it for me. Thanks!


Andres
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161117/6f36745f/attachment.html>

From matti.picus at gmail.com  Thu Nov 17 18:24:19 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 18 Nov 2016 01:24:19 +0200
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
Message-ID: <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>

Congrats to all on the release.Two questions:

Is there a guide to building standard wheels for NumPy?

Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and 
OSX64, how can I get them blessed and uploaded to PyPI?

Matti


On 17/11/16 07:47, numpy-discussion-request at scipy.org wrote:
> Date: Wed, 16 Nov 2016 22:47:39 -0700
> From: Charles R Harris<charlesr.harris at gmail.com>
> To: numpy-discussion<numpy-discussion at scipy.org>, SciPy Users List
> 	<scipy-user at scipy.org>,  SciPy Developers List<scipy-dev at scipy.org>,
> 	python-announce-list at python.org
> Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
>
> Hi All,
>
> I'm pleased to annouce the release of NumPy 1.12.0b1. This release
> supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests
> submitted by 133 contributors. It is quite sizeable and rather than put the
> release notes inline I've attached them as a file and they may also be
> viewed at Github<https://github.com/numpy/numpy/releases/tag/v1.12.0b1>.
> Zip files and tarballs may also be found the Github link. Wheels and source
> archives may be downloaded from PyPI, which is the recommended method.
>
> This release is a large collection of fixes, enhancements, and improvements
> and it is difficult to select just a few as highlights. However, the
> following enhancements may be of particular interest
>
>     - Order of operations in ``np.einsum`` now can be optimized for large
>     speed improvements.
>     - New ``signature`` argument to ``np.vectorize`` for vectorizing with
>     core dimensions.
>     - The ``keepdims`` argument was added to many functions.
>     - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this
>     is a milestone for PyPy's C-API compatibility layer.
>
> Thanks to all,
>
> Chuck



From izgurskii at gmail.com  Fri Nov 18 02:43:05 2016
From: izgurskii at gmail.com (Dmitrii Izgurskii)
Date: Fri, 18 Nov 2016 08:43:05 +0100
Subject: [Numpy-discussion] Status of NAs for integer arrays
Message-ID: <CAO3kOC1-TBjSJdQ_DHOx9kZ9eZ_BbZGeQL5i=pZjHrTSC+AmuQ@mail.gmail.com>

Hi,

I would like to know where I can follow the progress of what's being done
on implementing missing values in integer arrays. Some threads I found are
from 2012. Is really nothing being done on this issue?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/0c13933b/attachment.html>

From matthew.brett at gmail.com  Fri Nov 18 03:08:25 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 18 Nov 2016 00:08:25 -0800
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
Message-ID: <CAH6Pt5qba9XbnCeyqyj4meS-owea=Lb4kqomoMQsjpcYY-c71w@mail.gmail.com>

Hi,

On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <matti.picus at gmail.com> wrote:
> Congrats to all on the release.Two questions:
>
> Is there a guide to building standard wheels for NumPy?

I don't think so - there is a repository that we use to build the
wheels, that has the Windows, OSX and manyllinux recipes for the
standard CPython build:

https://github.com/MacPython/numpy-wheelso

If you can work out a way to automate the PyPy builds and tests -
especially using the same repo - that would be very useful.

> Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> OSX64, how can I get them blessed and uploaded to PyPI?

If you can automate the build and tests, I'm guessing there will be no
objections - but it's not my call...

Cheers,

Matthew


From ralf.gommers at gmail.com  Fri Nov 18 03:41:24 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 18 Nov 2016 21:41:24 +1300
Subject: [Numpy-discussion] Status of NAs for integer arrays
In-Reply-To: <CAO3kOC1-TBjSJdQ_DHOx9kZ9eZ_BbZGeQL5i=pZjHrTSC+AmuQ@mail.gmail.com>
References: <CAO3kOC1-TBjSJdQ_DHOx9kZ9eZ_BbZGeQL5i=pZjHrTSC+AmuQ@mail.gmail.com>
Message-ID: <CABL7CQhJMq7kV=kH1A+nK8Q4zmqRTT_zFtgHj67rghPH0S6U+w@mail.gmail.com>

On Fri, Nov 18, 2016 at 8:43 PM, Dmitrii Izgurskii <izgurskii at gmail.com>
wrote:

> Hi,
>
> I would like to know where I can follow the progress of what's being done
> on implementing missing values in integer arrays. Some threads I found are
> from 2012. Is really nothing being done on this issue?
>

Not that I'm aware of, no.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/3cc92d1e/attachment.html>

From ralf.gommers at gmail.com  Fri Nov 18 04:14:45 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 18 Nov 2016 22:14:45 +1300
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <CAH6Pt5qba9XbnCeyqyj4meS-owea=Lb4kqomoMQsjpcYY-c71w@mail.gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
 <CAH6Pt5qba9XbnCeyqyj4meS-owea=Lb4kqomoMQsjpcYY-c71w@mail.gmail.com>
Message-ID: <CABL7CQj5g0OmLZUeu5G7za5jnApzphFZ8+nLcecmoin7oOMY2Q@mail.gmail.com>

On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi,
>
> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
> > Congrats to all on the release.Two questions:
> >
> > Is there a guide to building standard wheels for NumPy?
>
> I don't think so - there is a repository that we use to build the
> wheels, that has the Windows, OSX and manyllinux recipes for the
> standard CPython build:
>
> https://github.com/MacPython/numpy-wheelso
>
> If you can work out a way to automate the PyPy builds and tests -
> especially using the same repo - that would be very useful.
>
> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> > OSX64, how can I get them blessed and uploaded to PyPI?
>
> If you can automate the build and tests, I'm guessing there will be no
> objections - but it's not my call...
>

I'm in favor, assuming that the wheel tags and PyPy backwards compatibility
situation is OK. Can't really find any examples. What I mean is that for
CPython wheels contain tags like "cp27" or "cp35". PyPy wheels should have
tags "pp<something>". Are the PyPy cpyext layer and the <something> defined
such that a new PyPy release won't break older wheels?

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/4f8fc3f9/attachment.html>

From sylvain.corlay at gmail.com  Fri Nov 18 05:34:21 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Fri, 18 Nov 2016 11:34:21 +0100
Subject: [Numpy-discussion] ANN: xtensor and xtensor-python
Message-ID: <CAK=Phk6XAX8XvHurF_cq0y27DK2E=TXKeiHOZLULJNen_1LVRg@mail.gmail.com>

Hi all,

We are pleased to announce the release of the xtensor
<https://github.com/QuantStack/xtensor> library and its python bindings
xtensor-python <https://github.com/QuantStack/xtensor-python>, by Johan
Mabille (@JohanMabille) and Sylvain Corlay (@SylvainCorlay).
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#1-xtensor>
1. xtensor

xtensor <https://github.com/QuantStack/xtensor> is a C++ template library
for manipulating multi-dimensional array expressions.

It provides

   - an API following the idioms of the *C++ standard library*.
   - an extensible expression system enabling lazy broadcasting and universal
   functions.
   - python bindings for operating inplace on numpy arrays in your C++
   thanks to Python's buffer protocol and pybind.

More details on lazy computing with xtensor are available below.
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#2-xtensor-cookiecutter>
2. xtensor-cookiecutter

Besides, xtensor <https://github.com/QuantStack/xtensor> and xtensor-python
<https://github.com/QuantStack/xtensor-python>, we provide a cookiecutter
template project to help extension authors get started. The
xtensor-cookiecutter <https://github.com/QuantStack/xtensor-python> generates
a simple project for a Python C++ extension with

   - a working setup.py compiling the extension module
   - a few example of functions making use of xtensor exposed to python,
   and the example of a vectorized function exposed to python.
   - all the boilerplate for the setup of the unit testing and generation
   of the HTML documentation for these examples.

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#3-you-can-try-it-now>3.
You can try it now!

You can try the xtensor live on the project website
<http://quantstack.net/xtensor>. The Try it Now button is powered by

   - The Cling C++ interpreter.
   - The Jupyter notebook.
   - The Binder project.

------------------------------
------------------------------
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#getting-started>Getting
started

xtensor requires a modern C++ compiler supporting C++14. The following C+
compilers are supported:

   - On Windows platforms, Visual C++ 2015 Update 2, or more recent
   - On Unix platforms, gcc 4.9 or a recent version of Clang

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#installation>
Installation

xtensor and xtensor-python are header-only libraries. We provide packages
for the conda package manager.

conda install -c conda-forge xtensor         # installs xtensor
conda install -c conda-forge xtensor-python  # installs xtensor and
xtensor-python

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#usage>
Usage
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#basic-usage>Basic
Usage

Initialize a 2-D array and compute the sum of one of its rows and a 1-D
array.

#include <iostream>
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray<double> arr1
  {{1.0, 2.0, 3.0},
   {2.0, 5.0, 7.0},
   {2.0, 5.0, 7.0}};

xt::xarray<double> arr2
  {5.0, 6.0, 7.0};

xt::xarray<double> res = xt::make_xview(arr1, 1) + arr2;

std::cout << res;

Outputs:

{7, 11, 14}

Initialize a 1-D array and reshape it inplace.

#include <iostream>
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray<int> arr
  {1, 2, 3, 4, 5, 6, 7, 8, 9};

arr.reshape({3, 3});

std::cout << arr;

Outputs:

{{1, 2, 3},
 {4, 5, 6},
 {7, 8, 9}}

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#lazy-broadcasting-with-xtensor>Lazy
Broadcasting with xtensor

We can operate on arrays of different shapes of dimensions in an
elementwise fashion. Broadcasting rules of xtensor are similar to those of
numpy <http://www.numpy.org/> and libdynd <http://libdynd.org/>.
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#broadcasting-rules>Broadcasting
rules

In an operation involving two arrays of different dimensions, the array
with the lesser dimensions is broadcast across the leading dimensions of
the other.

For example, if A has shape (2, 3), and B has shape (4, 2, 3), the result
of a broadcasted operation with A and B has shape (4, 2, 3).

   (2, 3) # A
(4, 2, 3) # B
---------
(4, 2, 3) # Result

The same rule holds for scalars, which are handled as 0-D expressions. If
matched up dimensions of two input arrays are different, and one of them
has size 1, it is broadcast to match the size of the other. Let's say B has
the shape (4, 2, 1) in the previous example, so the broadcasting happens as
follows:

   (2, 3) # A
(4, 2, 1) # B
---------
(4, 2, 3) # Result

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#universal-functions-laziness-and-vectorization>Universal
functions, Laziness and Vectorization

With xtensor, if x, y and z are arrays of *broadcastable shapes*, the
return type of an expression such as x + y * sin(z) is not an array. It is
an xexpression object offering the same interface as an N-dimensional
array, which does not hold the result. Values are only computed upon access
or when the expression is assigned to an xarray object. This allows to
operate symbolically on very large arrays and only compute the result for
the indices of interest.

We provide utilities to vectorize any scalar function (taking multiple
scalar arguments) into a function that will perform on xexpressions,
applying the lazy broadcasting rules which we just described. These
functions are called *xfunction*s. They are xtensor's counterpart to
numpy's universal functions.

In xtensor, arithmetic operations (+, -, *, /) and all special functions
are *xfunction*s.
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#iterating-over-xexpressions-and-broadcasting-iterators>Iterating
over xexpressions and Broadcasting Iterators

All xexpressions offer two sets of functions to retrieve iterator pairs
(and their const counterpart).

   - begin() and end() provide instances of xiterators which can be used to
   iterate over all the elements of the expression. The order in which
   elements are listed is row-major in that the index of last dimension is
   incremented first.
   - xbegin(shape) and xend(shape) are similar but take a *broadcasting
   shape* as an argument. Elements are iterated upon in a row-major way,
   but certain dimensions are repeated to match the provided shape as per the
   rules described above. For an expression e, e.xbegin(e.shape()) and
   e.begin() are equivalent.

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#fixed-dimension-and-dynamic-dimension>
Fixed-dimension *and* Dynamic dimension

Two container classes implementing multi-dimensional arrays are provided:
xarray and xtensor.

   - xarray can be reshaped dynamically to any number of dimensions. It is
   the container that is the most similar to numpy arrays.
   - xtensor has a dimension set at compilation time, which enables many
   optimizations. For example, shapes and strides of xtensor instances are
   allocated on the stack instead of the heap.

xarray and xtensor container are both xexpressions and can be involved and
mixed in universal functions, assigned to each other etc...

Besides, two access operators are provided:

   - The variadic template operator() which can take multiple integral
   arguments or none.
   - And the operator[] which takes a single multi-index argument, which
   can be of size determined at runtime. operator[] also supports access
   with braced initializers.

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#python-bindings>Python
bindings

The python bindings are built upon the pybind11
<https://github.com/pybind/pybind11> library, a lightweight header-only for
creating bindings between the Python and C++ programming languages.
<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#example-1-use-an-algorithm-of-the-c-library-on-a-numpy-array-inplace>Example
1: Use an algorithm of the C++ library on a numpy array inplace.

C++ code

#include <numeric>                    // std::accumulate
#include "pybind11/pybind11.h"        // pybind11
#include "xtensor/xmath.hpp"          // C++ universal functions
#include "xtensor-python/pyarray.hpp" // numpy bindings
double sum_of_sines(xt::pyarray<double> &m)
{
    auto sines = xt::sin(m); // sines does not hold any value
    return std::accumulate(sines.begin(), sines.end(), 0.0);
}
PYBIND11_PLUGIN(xtensor_python_test)
{
    pybind11::module m("xtensor_python_test",
                       "Test module for xtensor python bindings");

    m.def("sum_of_sines",
          sum_of_sines,
          "Return the sum of the sines");

    return m.ptr();
}

Python Code

import numpy as npimport xtensor_python_test as xt

a = np.arange(15).reshape(3, 5)
s = xt.sum_of_sines(v)
s

Outputs

1.2853996391883833

<https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#example-2-create-a-universal-function-from-a-c-scalar-function>Example
2: Create a universal function from a C++ scalar function

C++ code

#include "pybind11/pybind11.h"
#include "xtensor-python/pyvectorize.hpp"
#include <numeric>
#include <cmath>
namespace py = pybind11;
double scalar_func(double i, double j)
{
    return std::sin(i) - std::cos(j);
}
PYBIND11_PLUGIN(xtensor_python_test)
{
    py::module m("xtensor_python_test",
                 "Test module for xtensor python bindings");

    m.def("vectorized_func", xt::pyvectorize(scalar_func), "");

    return m.ptr();
}

Python Code

import numpy as npimport xtensor_python_test as xt

x = np.arange(15).reshape(3, 5)
y = [1, 2, 3]
z = xt.vectorized_func(x, y)
z

Outputs

[[-1.     ,  0.30116,  1.32544,  1.13111, -0.10315],
 [-1.95892, -0.81971,  1.07313,  1.97935,  1.06576],
 [-1.54402, -1.54029, -0.12042,  1.41016,  1.64425]]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/8dc6f21e/attachment.html>

From p.j.a.cock at googlemail.com  Fri Nov 18 09:30:33 2016
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Fri, 18 Nov 2016 14:30:33 +0000
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
Message-ID: <CAKVJ-_7zp9+u6Dkm7=p+qwKnL5OGs5ZxNSz_6M5ADZoP1EsQJg@mail.gmail.com>

I have a related question to Matti's,

Do you have any recommendations for building standard wheels
for 3rd party Python libraries which use both the NumPy Python
and C API?

e.g. Do we need to do anything special given the NumPy C API
itself is versioned? Does it matter compiler chain should we use?

Thanks

Peter

On Thu, Nov 17, 2016 at 11:24 PM, Matti Picus <matti.picus at gmail.com> wrote:
> Congrats to all on the release.Two questions:
>
> Is there a guide to building standard wheels for NumPy?
>
> Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> OSX64, how can I get them blessed and uploaded to PyPI?
>
> Matti
>
>
> On 17/11/16 07:47, numpy-discussion-request at scipy.org wrote:
>>
>> Date: Wed, 16 Nov 2016 22:47:39 -0700
>> From: Charles R Harris<charlesr.harris at gmail.com>
>> To: numpy-discussion<numpy-discussion at scipy.org>, SciPy Users List
>>         <scipy-user at scipy.org>,  SciPy Developers
>> List<scipy-dev at scipy.org>,
>>         python-announce-list at python.org
>> Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
>>
>> Hi All,
>>
>> I'm pleased to annouce the release of NumPy 1.12.0b1. This release
>> supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests
>> submitted by 133 contributors. It is quite sizeable and rather than put
>> the
>> release notes inline I've attached them as a file and they may also be
>> viewed at Github<https://github.com/numpy/numpy/releases/tag/v1.12.0b1>.
>> Zip files and tarballs may also be found the Github link. Wheels and
>> source
>> archives may be downloaded from PyPI, which is the recommended method.
>>
>> This release is a large collection of fixes, enhancements, and
>> improvements
>> and it is difficult to select just a few as highlights. However, the
>> following enhancements may be of particular interest
>>
>>     - Order of operations in ``np.einsum`` now can be optimized for large
>>     speed improvements.
>>     - New ``signature`` argument to ``np.vectorize`` for vectorizing with
>>     core dimensions.
>>     - The ``keepdims`` argument was added to many functions.
>>     - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this
>>     is a milestone for PyPy's C-API compatibility layer.
>>
>> Thanks to all,
>>
>> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion


From nathan12343 at gmail.com  Fri Nov 18 09:46:41 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Fri, 18 Nov 2016 14:46:41 +0000
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <CAKVJ-_7zp9+u6Dkm7=p+qwKnL5OGs5ZxNSz_6M5ADZoP1EsQJg@mail.gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
 <CAKVJ-_7zp9+u6Dkm7=p+qwKnL5OGs5ZxNSz_6M5ADZoP1EsQJg@mail.gmail.com>
Message-ID: <CAJXewOkWHLpUofs-K4=rT_b0Pzr-DCVAeLyasAXg_EoJbHDXPA@mail.gmail.com>

Since the NumPy API is forwards compatible, you should use the oldest
version of NumPy you would like to support to build your wheels with. The
wheels will then work with any future NumPy versions.

On Fri, Nov 18, 2016 at 9:30 AM Peter Cock <p.j.a.cock at googlemail.com>
wrote:

> I have a related question to Matti's,
>
> Do you have any recommendations for building standard wheels
> for 3rd party Python libraries which use both the NumPy Python
> and C API?
>
> e.g. Do we need to do anything special given the NumPy C API
> itself is versioned? Does it matter compiler chain should we use?
>
> Thanks
>
> Peter
>
> On Thu, Nov 17, 2016 at 11:24 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
> > Congrats to all on the release.Two questions:
> >
> > Is there a guide to building standard wheels for NumPy?
> >
> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> > OSX64, how can I get them blessed and uploaded to PyPI?
> >
> > Matti
> >
> >
> > On 17/11/16 07:47, numpy-discussion-request at scipy.org wrote:
> >>
> >> Date: Wed, 16 Nov 2016 22:47:39 -0700
> >> From: Charles R Harris<charlesr.harris at gmail.com>
> >> To: numpy-discussion<numpy-discussion at scipy.org>, SciPy Users List
> >>         <scipy-user at scipy.org>,  SciPy Developers
> >> List<scipy-dev at scipy.org>,
> >>         python-announce-list at python.org
> >> Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
> >>
> >> Hi All,
> >>
> >> I'm pleased to annouce the release of NumPy 1.12.0b1. This release
> >> supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull
> requests
> >> submitted by 133 contributors. It is quite sizeable and rather than put
> >> the
> >> release notes inline I've attached them as a file and they may also be
> >> viewed at Github<https://github.com/numpy/numpy/releases/tag/v1.12.0b1
> >.
> >> Zip files and tarballs may also be found the Github link. Wheels and
> >> source
> >> archives may be downloaded from PyPI, which is the recommended method.
> >>
> >> This release is a large collection of fixes, enhancements, and
> >> improvements
> >> and it is difficult to select just a few as highlights. However, the
> >> following enhancements may be of particular interest
> >>
> >>     - Order of operations in ``np.einsum`` now can be optimized for
> large
> >>     speed improvements.
> >>     - New ``signature`` argument to ``np.vectorize`` for vectorizing
> with
> >>     core dimensions.
> >>     - The ``keepdims`` argument was added to many functions.
> >>     - Support for PyPy 2.7 v5.6.0 has been added. While not complete,
> this
> >>     is a milestone for PyPy's C-API compatibility layer.
> >>
> >> Thanks to all,
> >>
> >> Chuck
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/942c6d9b/attachment.html>

From p.j.a.cock at googlemail.com  Fri Nov 18 10:42:33 2016
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Fri, 18 Nov 2016 15:42:33 +0000
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <CAJXewOkWHLpUofs-K4=rT_b0Pzr-DCVAeLyasAXg_EoJbHDXPA@mail.gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
 <CAKVJ-_7zp9+u6Dkm7=p+qwKnL5OGs5ZxNSz_6M5ADZoP1EsQJg@mail.gmail.com>
 <CAJXewOkWHLpUofs-K4=rT_b0Pzr-DCVAeLyasAXg_EoJbHDXPA@mail.gmail.com>
Message-ID: <CAKVJ-_6G9OeUVPa4uoz++K7wphwueqiBVRwgGHuo9GZ8mnRBbQ@mail.gmail.com>

Thanks Nathan,

That makes sense (compile using the oldest version of NumPy
we wish to support).

The information on https://github.com/MacPython/numpy-wheels
will probably be very useful too (I've been meaning to try out
appveyor at some point for Windows builds/testing).

Regards,

Peter

On Fri, Nov 18, 2016 at 2:46 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Since the NumPy API is forwards compatible, you should use the oldest
> version of NumPy you would like to support to build your wheels with. The
> wheels will then work with any future NumPy versions.
>
> On Fri, Nov 18, 2016 at 9:30 AM Peter Cock <p.j.a.cock at googlemail.com>
> wrote:
>>
>> I have a related question to Matti's,
>>
>> Do you have any recommendations for building standard wheels
>> for 3rd party Python libraries which use both the NumPy Python
>> and C API?
>>
>> e.g. Do we need to do anything special given the NumPy C API
>> itself is versioned? Does it matter compiler chain should we use?
>>
>> Thanks
>>
>> Peter


From njs at pobox.com  Fri Nov 18 11:24:10 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 18 Nov 2016 08:24:10 -0800
Subject: [Numpy-discussion] NumPy 1.12.0b1 released
In-Reply-To: <CABL7CQj5g0OmLZUeu5G7za5jnApzphFZ8+nLcecmoin7oOMY2Q@mail.gmail.com>
References: <mailman.103586.1479361663.2321.numpy-discussion@scipy.org>
 <85707776-c19f-3b1e-ba75-9ca4ed2b209e@gmail.com>
 <CAH6Pt5qba9XbnCeyqyj4meS-owea=Lb4kqomoMQsjpcYY-c71w@mail.gmail.com>
 <CABL7CQj5g0OmLZUeu5G7za5jnApzphFZ8+nLcecmoin7oOMY2Q@mail.gmail.com>
Message-ID: <CAPJVwB=02e2YPfmWGsD8SEBifcxwSg7kcjbWzabkgcJ+po8=kg@mail.gmail.com>

On Nov 18, 2016 01:14, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
>
>
>
> On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:
>>
>> Hi,
>>
>> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <matti.picus at gmail.com>
wrote:
>> > Congrats to all on the release.Two questions:
>> >
>> > Is there a guide to building standard wheels for NumPy?
>>
>> I don't think so - there is a repository that we use to build the
>> wheels, that has the Windows, OSX and manyllinux recipes for the
>> standard CPython build:
>>
>> https://github.com/MacPython/numpy-wheelso
>>
>> If you can work out a way to automate the PyPy builds and tests -
>> especially using the same repo - that would be very useful.
>>
>> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
>> > OSX64, how can I get them blessed and uploaded to PyPI?
>>
>> If you can automate the build and tests, I'm guessing there will be no
>> objections - but it's not my call...
>
>
> I'm in favor, assuming that the wheel tags and PyPy backwards
compatibility situation is OK. Can't really find any examples. What I mean
is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels
should have tags "pp<something>". Are the PyPy cpyext layer and the
<something> defined such that a new PyPy release won't break older wheels?

Another thing to think about is that 1.12 on pypy won't pass its test suite
(though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/c81256d4/attachment.html>

From ralf.gommers at gmail.com  Fri Nov 18 18:30:19 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 19 Nov 2016 12:30:19 +1300
Subject: [Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)
Message-ID: <CABL7CQiRJ_znutsxfEjm7DngUb0Nb7CcO2prMAS+6wVNCJSq2Q@mail.gmail.com>

On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Another thing to think about is that 1.12 on pypy won't pass its test
> suite (though it's close), and we're not yet testing new PRs on pypy, so no
> guarantees about 1.13 yet. I think on balance these probably aren't reasons
> *not* to upload wheels, but it's a funny place where we're talking about
> providing "official" builds even though it's not an "officially supported
> platform". So we will at least want to be clear about that. And someone
> will have to handle the bug reports about the test suite failing :-).
>

Those are good points. We could run PyPy on TravisCI; the PyPy install and
numpy build aren't difficult anymore.

Handling bug reports is mostly checking if it's PyPy specific, and if so
refer to the PyPy tracker I'd think. It is some work, but given that PyPy
has finally chosen a way to support Numpy that's not a dead end and has
come quite a long way quite quickly, taking on that bit of extra work as
Numpy maintainers is a good time investment imho.

Many bug reports will go straight to PyPy though I expect, because often
that is the obvious place to go. This is what I just got from downloading
the OS X PyPy binary and pip installing numpy master:

Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with
GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
nose version 1.3.7
.................................S.......................................................E........................................................................................S.............................................................RPython
traceback:
  File "pypy_interpreter.c", line 43348, in
BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_4.c", line 16627, in
generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17
Fatal RPython error: AssertionError
Abort trap: 6

So guess I'll go find out where that issue tracker is:)

Ralf


On Nov 18, 2016 01:14, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
> >
> >
> >
> > On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
> >> > Congrats to all on the release.Two questions:
> >> >
> >> > Is there a guide to building standard wheels for NumPy?
> >>
> >> I don't think so - there is a repository that we use to build the
> >> wheels, that has the Windows, OSX and manyllinux recipes for the
> >> standard CPython build:
> >>
> >> https://github.com/MacPython/numpy-wheelso
> >>
> >> If you can work out a way to automate the PyPy builds and tests -
> >> especially using the same repo - that would be very useful.
> >>
> >> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32
> and
> >> > OSX64, how can I get them blessed and uploaded to PyPI?
> >>
> >> If you can automate the build and tests, I'm guessing there will be no
> >> objections - but it's not my call...
> >
> >
> > I'm in favor, assuming that the wheel tags and PyPy backwards
> compatibility situation is OK. Can't really find any examples. What I mean
> is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels
> should have tags "pp<something>". Are the PyPy cpyext layer and the
> <something> defined such that a new PyPy release won't break older wheels?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161119/6af7f7ff/attachment.html>

From njs at pobox.com  Fri Nov 18 20:53:04 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 18 Nov 2016 17:53:04 -0800
Subject: [Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)
In-Reply-To: <CABL7CQiRJ_znutsxfEjm7DngUb0Nb7CcO2prMAS+6wVNCJSq2Q@mail.gmail.com>
References: <CABL7CQiRJ_znutsxfEjm7DngUb0Nb7CcO2prMAS+6wVNCJSq2Q@mail.gmail.com>
Message-ID: <CAPJVwB=3GhHcTuRDTeuYPMLwOOG9OifkQTf6h0kGy2XrfdiW=g@mail.gmail.com>

On Nov 18, 2016 3:30 PM, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Another thing to think about is that 1.12 on pypy won't pass its test
suite (though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).
>
>
> Those are good points. We could run PyPy on TravisCI; the PyPy install
and numpy build aren't difficult anymore.

I'm not sure how useful this until the test suite is passing, though, just
because of how travis-ci works.

The main outstanding issue needs some work on the numpy side: basically
UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
because it assumes that Python objects get deterministically deallocated as
soon as their reference count drops to zero, and pypy doesn't have
reference counts.

I think fixing this should be straightforward enough, in case anyone wants
to work on it. Every user of UPDATEIFCOPY already has to be aware of the
reference counts and know when the pseudo-"view" array is supposed to be
deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
acts like the current UPDATEIFCOPY, except that instead of using __del__ to
do the writeback, there's an explicit API call you have to make, like
PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
writeback if set. Then we should transition to using this internally, and
probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
it entirely).

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161118/89276fe0/attachment.html>

From ralf.gommers at gmail.com  Fri Nov 18 21:14:06 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 19 Nov 2016 15:14:06 +1300
Subject: [Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)
In-Reply-To: <CAPJVwB=3GhHcTuRDTeuYPMLwOOG9OifkQTf6h0kGy2XrfdiW=g@mail.gmail.com>
References: <CABL7CQiRJ_znutsxfEjm7DngUb0Nb7CcO2prMAS+6wVNCJSq2Q@mail.gmail.com>
 <CAPJVwB=3GhHcTuRDTeuYPMLwOOG9OifkQTf6h0kGy2XrfdiW=g@mail.gmail.com>
Message-ID: <CABL7CQgYJ-Gw5se18Xw-30hChz6Dus9=KHY-xD+QrXrQMmKzqA@mail.gmail.com>

On Sat, Nov 19, 2016 at 2:53 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Nov 18, 2016 3:30 PM, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
> >
> >
> >
> > On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> Another thing to think about is that 1.12 on pypy won't pass its test
> suite (though it's close), and we're not yet testing new PRs on pypy, so no
> guarantees about 1.13 yet. I think on balance these probably aren't reasons
> *not* to upload wheels, but it's a funny place where we're talking about
> providing "official" builds even though it's not an "officially supported
> platform". So we will at least want to be clear about that. And someone
> will have to handle the bug reports about the test suite failing :-).
> >
> >
> > Those are good points. We could run PyPy on TravisCI; the PyPy install
> and numpy build aren't difficult anymore.
>
> I'm not sure how useful this until the test suite is passing, though, just
> because of how travis-ci works.
>
It's not hard to skip the currently failing tests only on a single run in
the build matrix. That would keep TravisCI green and ensure there's no
regressions.

Ralf



> The main outstanding issue needs some work on the numpy side: basically
> UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
> because it assumes that Python objects get deterministically deallocated as
> soon as their reference count drops to zero, and pypy doesn't have
> reference counts.
>
> I think fixing this should be straightforward enough, in case anyone wants
> to work on it. Every user of UPDATEIFCOPY already has to be aware of the
> reference counts and know when the pseudo-"view" array is supposed to be
> deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
> acts like the current UPDATEIFCOPY, except that instead of using __del__ to
> do the writeback, there's an explicit API call you have to make, like
> PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
> writeback if set. Then we should transition to using this internally, and
> probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
> it entirely).
>
> -n
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161119/0ebe0c7d/attachment.html>

From leonardo.bianconi at eldorado.org.br  Wed Nov 23 09:47:18 2016
From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi)
Date: Wed, 23 Nov 2016 14:47:18 +0000
Subject: [Numpy-discussion] Wheel files for PPC64
Message-ID: <77b02d9343004f4997364a2e8a9ffc55@serv030.corp.eldorado.org.br>

Hi all!

I realized that there is no "whl" files for PPC64 (https://pypi.python.org/pypi/numpy/1.12.0b1), which makes numpy be compiled when installing it with pip. It is causing a low performance on this architecture when the libopenblas is not previously installed, while for x86_64 it is installed as a lib, which is in the "whl" file.

What must to be done to add a build of numpy for PPC64, and upload the "whl" files to pypi link?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161123/a76388b8/attachment.html>

From matthew.brett at gmail.com  Wed Nov 23 13:08:51 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 23 Nov 2016 10:08:51 -0800
Subject: [Numpy-discussion] Wheel files for PPC64
In-Reply-To: <77b02d9343004f4997364a2e8a9ffc55@serv030.corp.eldorado.org.br>
References: <77b02d9343004f4997364a2e8a9ffc55@serv030.corp.eldorado.org.br>
Message-ID: <CAH6Pt5q2W5ZYJ6nOPcHd+C05GE=2Jj1esoe8qzHGUSTfX+_BwA@mail.gmail.com>

Hi,

On Wed, Nov 23, 2016 at 6:47 AM, Leonardo Bianconi
<leonardo.bianconi at eldorado.org.br> wrote:
> Hi all!
>
>
>
> I realized that there is no ?whl? files for PPC64
> (https://pypi.python.org/pypi/numpy/1.12.0b1), which makes numpy be compiled
> when installing it with pip. It is causing a low performance on this
> architecture when the libopenblas is not previously installed, while for
> x86_64 it is installed as a lib, which is in the ?whl? file.
>
>
>
> What must to be done to add a build of numpy for PPC64, and upload the ?whl?
> files to pypi link?

The problem for wheel files is agreeing to some standard for binary
compatibility that guarantees the wheel will work on more than a
single Unix distribution / version.

The work to do that for Intel Linux is over at
https://www.python.org/dev/peps/pep-0513 .  As you can see, that PEP
defines a standard set of libraries and library symbol versions to
build against, such that the resulting wheel will be compatible with a
large number of Intel Linux distributions.    There's also a standard
docker container to build with, containing the correct libraries /
symbols.

Otherwise, you'll have to build the wheels for your platform only, and
put them up somewhere in a web directory, for others with the same
platform.  It doesn't make sense to put them on pypi, because, unless
you have done the standardization work, they will likely fail for most
people.

Best,

Matthew


From leonardo.bianconi at eldorado.org.br  Tue Nov 29 12:22:09 2016
From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi)
Date: Tue, 29 Nov 2016 17:22:09 +0000
Subject: [Numpy-discussion] Wheel files for PPC64
Message-ID: <69e5934e8fb04301828e28577d5c4bd8@serv030.corp.eldorado.org.br>

Hi.
Can anyone help me with this?

>
> From: Leonardo Bianconi 
> Sent: quarta-feira, 23 de novembro de 2016 12:47
> To: 'numpy-discussion at scipy.org' <numpy-discussion at scipy.org>
> Subject: Wheel files for PPC64
>
> Hi all!
>
> I realized that there is no "whl" files for PPC64 (https://pypi.python.org/pypi/numpy/1.12.0b1),
> which makes numpy be compiled when installing it with pip. It is causing a low performance on
> this architecture when the libopenblas is not previously installed, while for x86_64 it is installed
> as a lib, which is in the "whl" file.
>
> What must to be done to add a build of numpy for PPC64, and upload the "whl" files to pypi
> link?


From nathan12343 at gmail.com  Tue Nov 29 12:29:00 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Tue, 29 Nov 2016 11:29:00 -0600
Subject: [Numpy-discussion] Wheel files for PPC64
In-Reply-To: <69e5934e8fb04301828e28577d5c4bd8@serv030.corp.eldorado.org.br>
References: <69e5934e8fb04301828e28577d5c4bd8@serv030.corp.eldorado.org.br>
Message-ID: <CAJXewOnf5XwNcjhwmpSe6MNvpRBaWqN7=b1_9_Sx8SMWZvNjEg@mail.gmail.com>

Hi Leonardo,

Matthew Brett replied to you on Nov 23:

https://mail.scipy.org/pipermail/numpy-discussion/2016-November/076290.html

It seems you will need to take this up with the upstream python packaging
community to come up with an alternate to manylinux1 that will work on ppc
architectures.

-Nathan

On Tue, Nov 29, 2016 at 11:22 AM, Leonardo Bianconi <
leonardo.bianconi at eldorado.org.br> wrote:

> Hi.
> Can anyone help me with this?
>
> >
> > From: Leonardo Bianconi
> > Sent: quarta-feira, 23 de novembro de 2016 12:47
> > To: 'numpy-discussion at scipy.org' <numpy-discussion at scipy.org>
> > Subject: Wheel files for PPC64
> >
> > Hi all!
> >
> > I realized that there is no "whl" files for PPC64 (
> https://pypi.python.org/pypi/numpy/1.12.0b1),
> > which makes numpy be compiled when installing it with pip. It is causing
> a low performance on
> > this architecture when the libopenblas is not previously installed,
> while for x86_64 it is installed
> > as a lib, which is in the "whl" file.
> >
> > What must to be done to add a build of numpy for PPC64, and upload the
> "whl" files to pypi
> > link?
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161129/ea85920d/attachment.html>

From weihuayi at xtu.edu.cn  Wed Nov 30 16:54:16 2016
From: weihuayi at xtu.edu.cn (Wei, Huayi)
Date: Thu, 1 Dec 2016 05:54:16 +0800
Subject: [Numpy-discussion] Does numpy.bincount support numpy.float128 type
	weights?
Message-ID: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>

Hi, There,

Here is a sample code using `numpy.bincount`

     import numpy as np
     a = np.array([1.0, 2.0, 3.0], dtype=np.float128)
     b = np.array([1, 2, 0], dtype=np.int)
     c = np.bincount(b, weights=a)

If run it, I get the following error report:

     ----> 1 c = np.bincount(b, weights=a)
     TypeError: Cannot cast array data from dtype('float128') to 
dtype('float64') according to the rule 'safe'

Is it a bug of `np.bincount`? Does there exist any similar function 
which I can use to do the similar thing with numpy.float128 type weights?

Best

Huayi





From nathan12343 at gmail.com  Wed Nov 30 16:59:03 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Wed, 30 Nov 2016 15:59:03 -0600
Subject: [Numpy-discussion] Does numpy.bincount support numpy.float128
 type weights?
In-Reply-To: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>
References: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>
Message-ID: <CAJXewOnFudMnC46GgryxsbHS90mGLCBsQSqa3zGYRHY8RhaRsA@mail.gmail.com>

I think this is a deficiency in the current implementation of bincount,
which always casts the weights to float64. This WIP pull request should
probably fix it:

https://github.com/numpy/numpy/pull/7464

On Wed, Nov 30, 2016 at 3:54 PM, Wei, Huayi <weihuayi at xtu.edu.cn> wrote:

> Hi, There,
>
> Here is a sample code using `numpy.bincount`
>
>     import numpy as np
>     a = np.array([1.0, 2.0, 3.0], dtype=np.float128)
>     b = np.array([1, 2, 0], dtype=np.int)
>     c = np.bincount(b, weights=a)
>
> If run it, I get the following error report:
>
>     ----> 1 c = np.bincount(b, weights=a)
>     TypeError: Cannot cast array data from dtype('float128') to
> dtype('float64') according to the rule 'safe'
>
> Is it a bug of `np.bincount`? Does there exist any similar function which
> I can use to do the similar thing with numpy.float128 type weights?
>
> Best
>
> Huayi
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161130/96e66163/attachment.html>

From jtaylor.debian at googlemail.com  Wed Nov 30 17:16:29 2016
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 30 Nov 2016 23:16:29 +0100
Subject: [Numpy-discussion] Does numpy.bincount support numpy.float128
 type weights?
In-Reply-To: <CAJXewOnFudMnC46GgryxsbHS90mGLCBsQSqa3zGYRHY8RhaRsA@mail.gmail.com>
References: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>
 <CAJXewOnFudMnC46GgryxsbHS90mGLCBsQSqa3zGYRHY8RhaRsA@mail.gmail.com>
Message-ID: <63be9f0d-e563-6e1a-a9a2-26c9a18a6cd0@googlemail.com>

Also note that float128 is rarely what you want.
It is not a quad precision value, it maps to C long double which is 80
bit on x86 and less on stuff like arm.

On 30.11.2016 22:59, Nathan Goldbaum wrote:
> I think this is a deficiency in the current implementation of bincount,
> which always casts the weights to float64. This WIP pull request should
> probably fix it:
> 
> https://github.com/numpy/numpy/pull/7464
> 
> On Wed, Nov 30, 2016 at 3:54 PM, Wei, Huayi <weihuayi at xtu.edu.cn
> <mailto:weihuayi at xtu.edu.cn>> wrote:
> 
>     Hi, There,
> 
>     Here is a sample code using `numpy.bincount`
> 
>         import numpy as np
>         a = np.array([1.0, 2.0, 3.0], dtype=np.float128)
>         b = np.array([1, 2, 0], dtype=np.int <http://np.int>)
>         c = np.bincount(b, weights=a)
> 
>     If run it, I get the following error report:
> 
>         ----> 1 c = np.bincount(b, weights=a)
>         TypeError: Cannot cast array data from dtype('float128') to
>     dtype('float64') according to the rule 'safe'
> 
>     Is it a bug of `np.bincount`? Does there exist any similar function
>     which I can use to do the similar thing with numpy.float128 type
>     weights?
> 
>     Best
> 
>     Huayi
> 
> 
> 
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     https://mail.scipy.org/mailman/listinfo/numpy-discussion
>     <https://mail.scipy.org/mailman/listinfo/numpy-discussion>
> 
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 



From sebastian at sipsolutions.net  Wed Nov 30 17:22:45 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 30 Nov 2016 23:22:45 +0100
Subject: [Numpy-discussion] Does numpy.bincount support numpy.float128
 type weights?
In-Reply-To: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>
References: <3b453ff5-dc3c-f084-4a55-0a98aedb243f@xtu.edu.cn>
Message-ID: <1480544565.816.7.camel@sipsolutions.net>

Fist off, a word of caution. float128 depends on your system and maps
to whatever longdouble is (IIRC) or may not even exist. So I hope you
don't expect IEEE 128 bit floats, if you are unsure, maybe check
`np.finfo`.

If speed does not matter
```
res = np.zeros(np.max(b), dtype=np.longdouble)
np.add.at(res, b, a)
```
will work, but do not expect it to be fast.

- Sebastian


On Do, 2016-12-01 at 05:54 +0800, Wei, Huayi wrote:
> Hi, There,
> 
> Here is a sample code using `numpy.bincount`
> 
> ?????import numpy as np
> ?????a = np.array([1.0, 2.0, 3.0], dtype=np.float128)
> ?????b = np.array([1, 2, 0], dtype=np.int)
> ?????c = np.bincount(b, weights=a)
> 
> If run it, I get the following error report:
> 
> ?????----> 1 c = np.bincount(b, weights=a)
> ?????TypeError: Cannot cast array data from dtype('float128') to?
> dtype('float64') according to the rule 'safe'
> 
> Is it a bug of `np.bincount`? Does there exist any similar function?
> which I can use to do the similar thing with numpy.float128 type
> weights?
> 
> Best
> 
> Huayi
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161130/51c4c7b3/attachment.sig>