From charlesr.harris at gmail.com  Tue May  9 11:19:49 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 9 May 2017 09:19:49 -0600
Subject: [Numpy-discussion] Ready to branch 1.13
Message-ID: <CAB6mnx+uGNVY68r=pObq-Xe0qyVYkA0+AyVs2_g_U5MnR4kp_g@mail.gmail.com>

Hi All,

I am getting ready to branch 1.13 after updating the release notes and
.mailmap. If there is something that you think is essential to get into
1.13 that is not there, comment.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170509/41f01679/attachment.html>

From numpy at mspacek.mm.st  Tue May  9 12:46:25 2017
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Tue, 9 May 2017 18:46:25 +0200
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
Message-ID: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>

Hello,

I've opened up a pull request to add a function called np.search(), or something 
like it, to complement np.searchsorted():

https://github.com/numpy/numpy/pull/9055

There's also this issue I opened before starting the PR:

https://github.com/numpy/numpy/issues/9052

Proposed API changes require discussion on the list, so here I am!

This proposed function (and perhaps array method?) does the same as 
np.searchsorted(a, v), but doesn't require `a` to be sorted, and explicitly 
checks if all the values in `v` are a subset of those in `a`. If not, it 
currently raises an error, but that could be controlled via a kwarg.

As I mentioned in the PR, I often find myself abusing np.searchsorted() by not 
explicitly checking these assumptions. The temptation to use it is great, 
because it's such a fast and convenient function, and most of the time that I 
use it, the assumptions are indeed valid. Explicitly checking those assumptions 
each and every time before I use np.searchsorted() is tedious, and easy to 
forget to do. I wouldn't be surprised if many others abuse np.searchsorted() in 
the same way.

Looking at my own habits and uses, it seems to me that finding the indices of 
matching values of one array in another is a more common use case than finding 
insertion indices of one array into another sorted array. So, I propose that 
np.search(), or something like it, could be even more useful than np.searchsorted().

Thoughts?

Martin

From shoyer at gmail.com  Tue May  9 13:39:09 2017
From: shoyer at gmail.com (Stephan Hoyer)
Date: Tue, 9 May 2017 10:39:09 -0700
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
Message-ID: <CAEQ_TvfXnL5K=K9iu=tmS3qFnJd6sfFUOEB+_sT=XpPEaqu7KA@mail.gmail.com>

On Tue, May 9, 2017 at 9:46 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:

> Looking at my own habits and uses, it seems to me that finding the indices
> of matching values of one array in another is a more common use case than
> finding insertion indices of one array into another sorted array. So, I
> propose that np.search(), or something like it, could be even more useful
> than np.searchsorted().
>

The current version of this PR only returns the indices of the *first*
match (rather than all matches), which is an important detail. I would
strongly consider including that detail in the name (e.g., by calling this
"find_first" rather than "search"), because my naive expectation for a
method called "search" is to find all matches.

In any case, I agree that this functionality would be welcome. Getting the
details right for a high performance solution is tricky, and there is
strong evidence of interest given the 200+ upvotes on this StackOverflow
question:
http://stackoverflow.com/questions/432112/is-there-a-numpy-function-to-return-the-first-index-of-something-in-an-array
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170509/80c6b4a1/attachment.html>

From wieser.eric+numpy at gmail.com  Tue May  9 19:36:42 2017
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Tue, 09 May 2017 23:36:42 +0000
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <CAEQ_TvfXnL5K=K9iu=tmS3qFnJd6sfFUOEB+_sT=XpPEaqu7KA@mail.gmail.com>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
 <CAEQ_TvfXnL5K=K9iu=tmS3qFnJd6sfFUOEB+_sT=XpPEaqu7KA@mail.gmail.com>
Message-ID: <CAL1kJvAfVqjr_tm3L_Z8KbE6Cs0pzok6vW5LRT3F=DR2attq8g@mail.gmail.com>

I don?t think that np.search is really solving the same problem as
find_first would.

IMO, we should solve that problem with an argfirst(bool_array, axis=0,
keepdims=False) -> intp function, with almost the same semantics as argmax,
but special-casing an array of Falses, to return bool_array.shape[axis].

Eric
?

On Tue, 9 May 2017 at 18:39 Stephan Hoyer <shoyer at gmail.com> wrote:

> On Tue, May 9, 2017 at 9:46 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
>
>> Looking at my own habits and uses, it seems to me that finding the
>> indices of matching values of one array in another is a more common use
>> case than finding insertion indices of one array into another sorted array.
>> So, I propose that np.search(), or something like it, could be even more
>> useful than np.searchsorted().
>>
>
> The current version of this PR only returns the indices of the *first*
> match (rather than all matches), which is an important detail. I would
> strongly consider including that detail in the name (e.g., by calling this
> "find_first" rather than "search"), because my naive expectation for a
> method called "search" is to find all matches.
>
> In any case, I agree that this functionality would be welcome. Getting the
> details right for a high performance solution is tricky, and there is
> strong evidence of interest given the 200+ upvotes on this StackOverflow
> question:
>
> http://stackoverflow.com/questions/432112/is-there-a-numpy-function-to-return-the-first-index-of-something-in-an-array
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170509/a34bc7bc/attachment.html>

From charlesr.harris at gmail.com  Wed May 10 14:36:20 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 10 May 2017 12:36:20 -0600
Subject: [Numpy-discussion] Numpy 1.13.x branched
Message-ID: <CAB6mnxLHGLbOThM5ceQLpciJ4L0Tkp8A17vi7Lf+zjeMejKSDg@mail.gmail.com>

Hi All,

Just a quck note that 1.13.x has been branched and master is open for
1.14.0 development.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170510/164f4dac/attachment.html>

From charlesr.harris at gmail.com  Wed May 10 21:48:34 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 10 May 2017 19:48:34 -0600
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
Message-ID: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>

Hi All,

I'm please to announce the NumPy 1.13.0rc1 release. This release supports
Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
most ambitious releases in the last several years. Some of the highlights
and new functions are

*Highlights*

   - Operations like ``a + b + c`` will reuse temporaries on some
   platforms, resulting in less memory use and faster execution.
   - Inplace operations check if inputs overlap outputs and create
   temporaries to avoid problems.
   - New __array_ufunc__ attribute provides improved ability for classes to
   override default ufunc behavior.
   -  New np.block function for creating blocked arrays.


*New functions*

   - New ``np.positive`` ufunc.
   - New ``np.divmod`` ufunc provides more efficient divmod.
   - New ``np.isnat`` ufunc tests for NaT special values.
   - New ``np.heaviside`` ufunc computes the Heaviside function.
   - New ``np.isin`` function, improves on ``in1d``.
   - New ``np.block`` function for creating blocked arrays.
   - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.

Wheels for the pre-release are available on PyPI. Source tarballs,
zipfiles, release notes, and the Changelog are available on github
<https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.

A total of 100 people contributed to this release.  People with a "+" by
their
names contributed a patch for the first time.

   - A. Jesse Jiryu Davis +
   - Alessandro Pietro Bardelli +
   - Alex Rothberg +
   - Alexander Shadchin
   - Allan Haldane
   - Andres Guzman-Ballen +
   - Antoine Pitrou
   - Antony Lee
   - B R S Recht +
   - Baurzhan Muftakhidinov +
   - Ben Rowland
   - Benda Xu +
   - Blake Griffith
   - Bradley Wogsland +
   - Brandon Carter +
   - CJ Carey
   - Charles Harris
   - Danny Hermes +
   - Duke Vijitbenjaronk +
   - Egor Klenin +
   - Elliott Forney +
   - Elliott M Forney +
   - Endolith
   - Eric Wieser
   - Erik M. Bray
   - Eugene +
   - Evan Limanto +
   - Felix Berkenkamp +
   - Fran?ois Bissey +
   - Frederic Bastien
   - Greg Young
   - Gregory R. Lee
   - Importance of Being Ernest +
   - Jaime Fernandez
   - Jakub Wilk +
   - James Cowgill +
   - James Sanders
   - Jean Utke +
   - Jesse Thoren +
   - Jim Crist +
   - Joerg Behrmann +
   - John Kirkham
   - Jonathan Helmus
   - Jonathan L Long
   - Jonathan Tammo Siebert +
   - Joseph Fox-Rabinovitz
   - Joshua Loyal +
   - Juan Nunez-Iglesias +
   - Julian Taylor
   - Kirill Balunov +
   - Likhith Chitneni +
   - Lo?c Est?ve
   - Mads Ohm Larsen
   - Marein K?nings +
   - Marten van Kerkwijk
   - Martin Thoma
   - Martino Sorbaro +
   - Marvin Schmidt +
   - Matthew Brett
   - Matthias Bussonnier +
   - Matthias C. M. Troffaes +
   - Matti Picus
   - Michael Seifert
   - Mikhail Pak +
   - Mortada Mehyar
   - Nathaniel J. Smith
   - Nick Papior
   - Oscar Villellas +
   - Pauli Virtanen
   - Pavel Potocek
   - Pete Peeradej Tanruangporn +
   - Philipp A +
   - Ralf Gommers
   - Robert Kern
   - Roland Kaufmann +
   - Ronan Lamy
   - Sami Salonen +
   - Sanchez Gonzalez Alvaro
   - Sebastian Berg
   - Shota Kawabuchi
   - Simon Gibbons
   - Stefan Otte
   - Stefan Peterson +
   - Stephan Hoyer
   - S?ren Fuglede J?rgensen +
   - Takuya Akiba
   - Tom Boyd +
   - Ville Skytt? +
   - Warren Weckesser
   - Wendell Smith
   - Yu Feng
   - Zixu Zhao +
   - Z? Vin?cius +
   - aha66 +
   - davidjn +
   - drabach +
   - drlvk +
   - jsh9 +
   - solarjoe +
   - zengi +

Cheers,

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170510/350fdb69/attachment.html>

From nathan12343 at gmail.com  Wed May 10 22:06:50 2017
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Thu, 11 May 2017 02:06:50 +0000
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
Message-ID: <CAJXewOnEZ4O+=0uHfn1N4LtWFDmc_3LDkDMj9YmJmf-UwMBrTQ@mail.gmail.com>

Hi Chuck,

Is there a docs build for this release somewhere? I'd like to find an
authoritative reference about __array_ufunc__, which I'd hesistated on
looking into until now for fear about the API changing.

Nathan

On Wed, May 10, 2017 at 8:49 PM Charles R Harris <charlesr.harris at gmail.com>
wrote:

> Hi All,
>
> I'm please to announce the NumPy 1.13.0rc1 release. This release supports
> Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
> most ambitious releases in the last several years. Some of the highlights
> and new functions are
>
> *Highlights*
>
>    - Operations like ``a + b + c`` will reuse temporaries on some
>    platforms, resulting in less memory use and faster execution.
>    - Inplace operations check if inputs overlap outputs and create
>    temporaries to avoid problems.
>    - New __array_ufunc__ attribute provides improved ability for classes
>    to override default ufunc behavior.
>    -  New np.block function for creating blocked arrays.
>
>
> *New functions*
>
>    - New ``np.positive`` ufunc.
>    - New ``np.divmod`` ufunc provides more efficient divmod.
>    - New ``np.isnat`` ufunc tests for NaT special values.
>    - New ``np.heaviside`` ufunc computes the Heaviside function.
>    - New ``np.isin`` function, improves on ``in1d``.
>    - New ``np.block`` function for creating blocked arrays.
>    - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
>
> Wheels for the pre-release are available on PyPI. Source tarballs,
> zipfiles, release notes, and the Changelog are available on github
> <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.
>
> A total of 100 people contributed to this release.  People with a "+" by
> their
> names contributed a patch for the first time.
>
>    - A. Jesse Jiryu Davis +
>    - Alessandro Pietro Bardelli +
>    - Alex Rothberg +
>    - Alexander Shadchin
>    - Allan Haldane
>    - Andres Guzman-Ballen +
>    - Antoine Pitrou
>    - Antony Lee
>    - B R S Recht +
>    - Baurzhan Muftakhidinov +
>    - Ben Rowland
>    - Benda Xu +
>    - Blake Griffith
>    - Bradley Wogsland +
>    - Brandon Carter +
>    - CJ Carey
>    - Charles Harris
>    - Danny Hermes +
>    - Duke Vijitbenjaronk +
>    - Egor Klenin +
>    - Elliott Forney +
>    - Elliott M Forney +
>    - Endolith
>    - Eric Wieser
>    - Erik M. Bray
>    - Eugene +
>    - Evan Limanto +
>    - Felix Berkenkamp +
>    - Fran?ois Bissey +
>    - Frederic Bastien
>    - Greg Young
>    - Gregory R. Lee
>    - Importance of Being Ernest +
>    - Jaime Fernandez
>    - Jakub Wilk +
>    - James Cowgill +
>    - James Sanders
>    - Jean Utke +
>    - Jesse Thoren +
>    - Jim Crist +
>    - Joerg Behrmann +
>    - John Kirkham
>    - Jonathan Helmus
>    - Jonathan L Long
>    - Jonathan Tammo Siebert +
>    - Joseph Fox-Rabinovitz
>    - Joshua Loyal +
>    - Juan Nunez-Iglesias +
>    - Julian Taylor
>    - Kirill Balunov +
>    - Likhith Chitneni +
>    - Lo?c Est?ve
>    - Mads Ohm Larsen
>    - Marein K?nings +
>    - Marten van Kerkwijk
>    - Martin Thoma
>    - Martino Sorbaro +
>    - Marvin Schmidt +
>    - Matthew Brett
>    - Matthias Bussonnier +
>    - Matthias C. M. Troffaes +
>    - Matti Picus
>    - Michael Seifert
>    - Mikhail Pak +
>    - Mortada Mehyar
>    - Nathaniel J. Smith
>    - Nick Papior
>    - Oscar Villellas +
>    - Pauli Virtanen
>    - Pavel Potocek
>    - Pete Peeradej Tanruangporn +
>    - Philipp A +
>    - Ralf Gommers
>    - Robert Kern
>    - Roland Kaufmann +
>    - Ronan Lamy
>    - Sami Salonen +
>    - Sanchez Gonzalez Alvaro
>    - Sebastian Berg
>    - Shota Kawabuchi
>    - Simon Gibbons
>    - Stefan Otte
>    - Stefan Peterson +
>    - Stephan Hoyer
>    - S?ren Fuglede J?rgensen +
>    - Takuya Akiba
>    - Tom Boyd +
>    - Ville Skytt? +
>    - Warren Weckesser
>    - Wendell Smith
>    - Yu Feng
>    - Zixu Zhao +
>    - Z? Vin?cius +
>    - aha66 +
>    - davidjn +
>    - drabach +
>    - drlvk +
>    - jsh9 +
>    - solarjoe +
>    - zengi +
>
> Cheers,
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/9db90632/attachment-0001.html>

From sank.daniel at gmail.com  Wed May 10 22:09:55 2017
From: sank.daniel at gmail.com (Daniel Sank)
Date: Wed, 10 May 2017 19:09:55 -0700
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAJXewOnEZ4O+=0uHfn1N4LtWFDmc_3LDkDMj9YmJmf-UwMBrTQ@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <CAJXewOnEZ4O+=0uHfn1N4LtWFDmc_3LDkDMj9YmJmf-UwMBrTQ@mail.gmail.com>
Message-ID: <CAKN9TDKb61hSZ=FEp-2oJY8JiCCrRpzUdtjH5ZoGgXna8Om8=Q@mail.gmail.com>

+1

I've been looking for docs on __array_ufunc__ for about ten minutes...

On Wed, May 10, 2017 at 7:06 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> Hi Chuck,
>
> Is there a docs build for this release somewhere? I'd like to find an
> authoritative reference about __array_ufunc__, which I'd hesistated on
> looking into until now for fear about the API changing.
>
> Nathan
>
> On Wed, May 10, 2017 at 8:49 PM Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> Hi All,
>>
>> I'm please to announce the NumPy 1.13.0rc1 release. This release supports
>> Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
>> most ambitious releases in the last several years. Some of the highlights
>> and new functions are
>>
>> *Highlights*
>>
>>    - Operations like ``a + b + c`` will reuse temporaries on some
>>    platforms, resulting in less memory use and faster execution.
>>    - Inplace operations check if inputs overlap outputs and create
>>    temporaries to avoid problems.
>>    - New __array_ufunc__ attribute provides improved ability for classes
>>    to override default ufunc behavior.
>>    -  New np.block function for creating blocked arrays.
>>
>>
>> *New functions*
>>
>>    - New ``np.positive`` ufunc.
>>    - New ``np.divmod`` ufunc provides more efficient divmod.
>>    - New ``np.isnat`` ufunc tests for NaT special values.
>>    - New ``np.heaviside`` ufunc computes the Heaviside function.
>>    - New ``np.isin`` function, improves on ``in1d``.
>>    - New ``np.block`` function for creating blocked arrays.
>>    - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
>>
>> Wheels for the pre-release are available on PyPI. Source tarballs,
>> zipfiles, release notes, and the Changelog are available on github
>> <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.
>>
>> A total of 100 people contributed to this release.  People with a "+" by
>> their
>> names contributed a patch for the first time.
>>
>>    - A. Jesse Jiryu Davis +
>>    - Alessandro Pietro Bardelli +
>>    - Alex Rothberg +
>>    - Alexander Shadchin
>>    - Allan Haldane
>>    - Andres Guzman-Ballen +
>>    - Antoine Pitrou
>>    - Antony Lee
>>    - B R S Recht +
>>    - Baurzhan Muftakhidinov +
>>    - Ben Rowland
>>    - Benda Xu +
>>    - Blake Griffith
>>    - Bradley Wogsland +
>>    - Brandon Carter +
>>    - CJ Carey
>>    - Charles Harris
>>    - Danny Hermes +
>>    - Duke Vijitbenjaronk +
>>    - Egor Klenin +
>>    - Elliott Forney +
>>    - Elliott M Forney +
>>    - Endolith
>>    - Eric Wieser
>>    - Erik M. Bray
>>    - Eugene +
>>    - Evan Limanto +
>>    - Felix Berkenkamp +
>>    - Fran?ois Bissey +
>>    - Frederic Bastien
>>    - Greg Young
>>    - Gregory R. Lee
>>    - Importance of Being Ernest +
>>    - Jaime Fernandez
>>    - Jakub Wilk +
>>    - James Cowgill +
>>    - James Sanders
>>    - Jean Utke +
>>    - Jesse Thoren +
>>    - Jim Crist +
>>    - Joerg Behrmann +
>>    - John Kirkham
>>    - Jonathan Helmus
>>    - Jonathan L Long
>>    - Jonathan Tammo Siebert +
>>    - Joseph Fox-Rabinovitz
>>    - Joshua Loyal +
>>    - Juan Nunez-Iglesias +
>>    - Julian Taylor
>>    - Kirill Balunov +
>>    - Likhith Chitneni +
>>    - Lo?c Est?ve
>>    - Mads Ohm Larsen
>>    - Marein K?nings +
>>    - Marten van Kerkwijk
>>    - Martin Thoma
>>    - Martino Sorbaro +
>>    - Marvin Schmidt +
>>    - Matthew Brett
>>    - Matthias Bussonnier +
>>    - Matthias C. M. Troffaes +
>>    - Matti Picus
>>    - Michael Seifert
>>    - Mikhail Pak +
>>    - Mortada Mehyar
>>    - Nathaniel J. Smith
>>    - Nick Papior
>>    - Oscar Villellas +
>>    - Pauli Virtanen
>>    - Pavel Potocek
>>    - Pete Peeradej Tanruangporn +
>>    - Philipp A +
>>    - Ralf Gommers
>>    - Robert Kern
>>    - Roland Kaufmann +
>>    - Ronan Lamy
>>    - Sami Salonen +
>>    - Sanchez Gonzalez Alvaro
>>    - Sebastian Berg
>>    - Shota Kawabuchi
>>    - Simon Gibbons
>>    - Stefan Otte
>>    - Stefan Peterson +
>>    - Stephan Hoyer
>>    - S?ren Fuglede J?rgensen +
>>    - Takuya Akiba
>>    - Tom Boyd +
>>    - Ville Skytt? +
>>    - Warren Weckesser
>>    - Wendell Smith
>>    - Yu Feng
>>    - Zixu Zhao +
>>    - Z? Vin?cius +
>>    - aha66 +
>>    - davidjn +
>>    - drabach +
>>    - drlvk +
>>    - jsh9 +
>>    - solarjoe +
>>    - zengi +
>>
>> Cheers,
>>
>> Chuck
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>


-- 
Daniel Sank
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170510/7eea055d/attachment.html>

From njs at pobox.com  Wed May 10 22:15:56 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 10 May 2017 19:15:56 -0700
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAJXewOnEZ4O+=0uHfn1N4LtWFDmc_3LDkDMj9YmJmf-UwMBrTQ@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <CAJXewOnEZ4O+=0uHfn1N4LtWFDmc_3LDkDMj9YmJmf-UwMBrTQ@mail.gmail.com>
Message-ID: <CAPJVwB=ZRNT4OYw9_U+1cXebCw=XUj_3+2h0kO4M5E6bwqC4Qg@mail.gmail.com>

On Wed, May 10, 2017 at 7:06 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Hi Chuck,
>
> Is there a docs build for this release somewhere? I'd like to find an
> authoritative reference about __array_ufunc__, which I'd hesistated on
> looking into until now for fear about the API changing.

A sort-of-rendered version of the end-user docs can be seen here:
  https://github.com/numpy/numpy/blob/master/doc/source/reference/arrays.classes.rst

And the NEP has been updated to hopefully provide a more spec-like
description of the final version:
  https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst

Note that the API is "provisional" in 1.13, i.e. it *might* change in
backwards-incompatible ways:
  https://docs.python.org/3/glossary.html#term-provisional-api

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From nadavh at visionsense.com  Thu May 11 09:09:51 2017
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 11 May 2017 13:09:51 +0000
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
Message-ID: <BY2PR08MB095EEBCC6A9A7FE15381931CBED0@BY2PR08MB095.namprd08.prod.outlook.com>

There is a change to "expand_dims" function, that it is now does not allow axis = a.ndims.

This influences matplotlib function get_bending_matrices in triinterpolate.py


  Nadav

________________________________
From: NumPy-Discussion <numpy-discussion-bounces+nadavh=visionsense.com at python.org> on behalf of Charles R Harris <charlesr.harris at gmail.com>
Sent: 11 May 2017 04:48:34
To: numpy-discussion; SciPy-User; SciPy Developers List; python-announce-list at python.org
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.

Hi All,

I'm please to announce the NumPy 1.13.0rc1 release. This release supports Python 2.7 and 3.4-3.6 and contains many new features. It is one of the most ambitious releases in the last several years. Some of the highlights and new functions are

Highlights

  *   Operations like ``a + b + c`` will reuse temporaries on some platforms, resulting in less memory use and faster execution.
  *   Inplace operations check if inputs overlap outputs and create temporaries to avoid problems.
  *   New __array_ufunc__ attribute provides improved ability for classes to override default ufunc behavior.
  *    New np.block function for creating blocked arrays.

New functions

  *   New ``np.positive`` ufunc.
  *   New ``np.divmod`` ufunc provides more efficient divmod.
  *   New ``np.isnat`` ufunc tests for NaT special values.
  *   New ``np.heaviside`` ufunc computes the Heaviside function.
  *   New ``np.isin`` function, improves on ``in1d``.
  *   New ``np.block`` function for creating blocked arrays.
  *   New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.

Wheels for the pre-release are available on PyPI. Source tarballs, zipfiles, release notes, and the Changelog are available on github<https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.

A total of 100 people contributed to this release.  People with a "+" by their
names contributed a patch for the first time.

  *   A. Jesse Jiryu Davis +
  *   Alessandro Pietro Bardelli +
  *   Alex Rothberg +
  *   Alexander Shadchin
  *   Allan Haldane
  *   Andres Guzman-Ballen +
  *   Antoine Pitrou
  *   Antony Lee
  *   B R S Recht +
  *   Baurzhan Muftakhidinov +
  *   Ben Rowland
  *   Benda Xu +
  *   Blake Griffith
  *   Bradley Wogsland +
  *   Brandon Carter +
  *   CJ Carey
  *   Charles Harris
  *   Danny Hermes +
  *   Duke Vijitbenjaronk +
  *   Egor Klenin +
  *   Elliott Forney +
  *   Elliott M Forney +
  *   Endolith
  *   Eric Wieser
  *   Erik M. Bray
  *   Eugene +
  *   Evan Limanto +
  *   Felix Berkenkamp +
  *   Fran?ois Bissey +
  *   Frederic Bastien
  *   Greg Young
  *   Gregory R. Lee
  *   Importance of Being Ernest +
  *   Jaime Fernandez
  *   Jakub Wilk +
  *   James Cowgill +
  *   James Sanders
  *   Jean Utke +
  *   Jesse Thoren +
  *   Jim Crist +
  *   Joerg Behrmann +
  *   John Kirkham
  *   Jonathan Helmus
  *   Jonathan L Long
  *   Jonathan Tammo Siebert +
  *   Joseph Fox-Rabinovitz
  *   Joshua Loyal +
  *   Juan Nunez-Iglesias +
  *   Julian Taylor
  *   Kirill Balunov +
  *   Likhith Chitneni +
  *   Lo?c Est?ve
  *   Mads Ohm Larsen
  *   Marein K?nings +
  *   Marten van Kerkwijk
  *   Martin Thoma
  *   Martino Sorbaro +
  *   Marvin Schmidt +
  *   Matthew Brett
  *   Matthias Bussonnier +
  *   Matthias C. M. Troffaes +
  *   Matti Picus
  *   Michael Seifert
  *   Mikhail Pak +
  *   Mortada Mehyar
  *   Nathaniel J. Smith
  *   Nick Papior
  *   Oscar Villellas +
  *   Pauli Virtanen
  *   Pavel Potocek
  *   Pete Peeradej Tanruangporn +
  *   Philipp A +
  *   Ralf Gommers
  *   Robert Kern
  *   Roland Kaufmann +
  *   Ronan Lamy
  *   Sami Salonen +
  *   Sanchez Gonzalez Alvaro
  *   Sebastian Berg
  *   Shota Kawabuchi
  *   Simon Gibbons
  *   Stefan Otte
  *   Stefan Peterson +
  *   Stephan Hoyer
  *   S?ren Fuglede J?rgensen +
  *   Takuya Akiba
  *   Tom Boyd +
  *   Ville Skytt? +
  *   Warren Weckesser
  *   Wendell Smith
  *   Yu Feng
  *   Zixu Zhao +
  *   Z? Vin?cius +
  *   aha66 +
  *   davidjn +
  *   drabach +
  *   drlvk +
  *   jsh9 +
  *   solarjoe +
  *   zengi +

Cheers,

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/b0b865fa/attachment-0001.html>

From wieser.eric+numpy at gmail.com  Thu May 11 09:56:36 2017
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Thu, 11 May 2017 13:56:36 +0000
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <BY2PR08MB095EEBCC6A9A7FE15381931CBED0@BY2PR08MB095.namprd08.prod.outlook.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <BY2PR08MB095EEBCC6A9A7FE15381931CBED0@BY2PR08MB095.namprd08.prod.outlook.com>
Message-ID: <CAL1kJvANUVy0WuSs7cJX6hmsCkpEH=hYt-hQ_bVhCUnWNWh69w@mail.gmail.com>

Nadav: Can you provide a testcase that fails?

I don't think you're correct - it works just fine when `axis = a.ndims` -
the issue arises when `axis > a.ndims`, but I'd argue that in that case an
error is correct behaviour. But still a change, so perhaps needs a release
note entry

On Thu, 11 May 2017 at 14:25 Nadav Horesh <nadavh at visionsense.com> wrote:

> There is a change to "expand_dims" function, that it is now does not allow
> axis = a.ndims.
>
> This influences matplotlib function get_bending_matrices in
> triinterpolate.py
>
>
>   Nadav
> ------------------------------
> *From:* NumPy-Discussion <numpy-discussion-bounces+nadavh=
> visionsense.com at python.org> on behalf of Charles R Harris <
> charlesr.harris at gmail.com>
> *Sent:* 11 May 2017 04:48:34
> *To:* numpy-discussion; SciPy-User; SciPy Developers List;
> python-announce-list at python.org
> *Subject:* [Numpy-discussion] NumPy v1.13.0rc1 released.
>
> Hi All,
>
> I'm please to announce the NumPy 1.13.0rc1 release. This release supports
> Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
> most ambitious releases in the last several years. Some of the highlights
> and new functions are
>
> *Highlights*
>
>    - Operations like ``a + b + c`` will reuse temporaries on some
>    platforms, resulting in less memory use and faster execution.
>    - Inplace operations check if inputs overlap outputs and create
>    temporaries to avoid problems.
>    - New __array_ufunc__ attribute provides improved ability for classes
>    to override default ufunc behavior.
>    -  New np.block function for creating blocked arrays.
>
>
> *New functions*
>
>    - New ``np.positive`` ufunc.
>    - New ``np.divmod`` ufunc provides more efficient divmod.
>    - New ``np.isnat`` ufunc tests for NaT special values.
>    - New ``np.heaviside`` ufunc computes the Heaviside function.
>    - New ``np.isin`` function, improves on ``in1d``.
>    - New ``np.block`` function for creating blocked arrays.
>    - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
>
> Wheels for the pre-release are available on PyPI. Source tarballs,
> zipfiles, release notes, and the Changelog are available on github
> <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.
>
> A total of 100 people contributed to this release.  People with a "+" by
> their
> names contributed a patch for the first time.
>
>    - A. Jesse Jiryu Davis +
>    - Alessandro Pietro Bardelli +
>    - Alex Rothberg +
>    - Alexander Shadchin
>    - Allan Haldane
>    - Andres Guzman-Ballen +
>    - Antoine Pitrou
>    - Antony Lee
>    - B R S Recht +
>    - Baurzhan Muftakhidinov +
>    - Ben Rowland
>    - Benda Xu +
>    - Blake Griffith
>    - Bradley Wogsland +
>    - Brandon Carter +
>    - CJ Carey
>    - Charles Harris
>    - Danny Hermes +
>    - Duke Vijitbenjaronk +
>    - Egor Klenin +
>    - Elliott Forney +
>    - Elliott M Forney +
>    - Endolith
>    - Eric Wieser
>    - Erik M. Bray
>    - Eugene +
>    - Evan Limanto +
>    - Felix Berkenkamp +
>    - Fran?ois Bissey +
>    - Frederic Bastien
>    - Greg Young
>    - Gregory R. Lee
>    - Importance of Being Ernest +
>    - Jaime Fernandez
>    - Jakub Wilk +
>    - James Cowgill +
>    - James Sanders
>    - Jean Utke +
>    - Jesse Thoren +
>    - Jim Crist +
>    - Joerg Behrmann +
>    - John Kirkham
>    - Jonathan Helmus
>    - Jonathan L Long
>    - Jonathan Tammo Siebert +
>    - Joseph Fox-Rabinovitz
>    - Joshua Loyal +
>    - Juan Nunez-Iglesias +
>    - Julian Taylor
>    - Kirill Balunov +
>    - Likhith Chitneni +
>    - Lo?c Est?ve
>    - Mads Ohm Larsen
>    - Marein K?nings +
>    - Marten van Kerkwijk
>    - Martin Thoma
>    - Martino Sorbaro +
>    - Marvin Schmidt +
>    - Matthew Brett
>    - Matthias Bussonnier +
>    - Matthias C. M. Troffaes +
>    - Matti Picus
>    - Michael Seifert
>    - Mikhail Pak +
>    - Mortada Mehyar
>    - Nathaniel J. Smith
>    - Nick Papior
>    - Oscar Villellas +
>    - Pauli Virtanen
>    - Pavel Potocek
>    - Pete Peeradej Tanruangporn +
>    - Philipp A +
>    - Ralf Gommers
>    - Robert Kern
>    - Roland Kaufmann +
>    - Ronan Lamy
>    - Sami Salonen +
>    - Sanchez Gonzalez Alvaro
>    - Sebastian Berg
>    - Shota Kawabuchi
>    - Simon Gibbons
>    - Stefan Otte
>    - Stefan Peterson +
>    - Stephan Hoyer
>    - S?ren Fuglede J?rgensen +
>    - Takuya Akiba
>    - Tom Boyd +
>    - Ville Skytt? +
>    - Warren Weckesser
>    - Wendell Smith
>    - Yu Feng
>    - Zixu Zhao +
>    - Z? Vin?cius +
>    - aha66 +
>    - davidjn +
>    - drabach +
>    - drlvk +
>    - jsh9 +
>    - solarjoe +
>    - zengi +
>
> Cheers,
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/34a4b304/attachment.html>

From shoyer at gmail.com  Thu May 11 12:23:20 2017
From: shoyer at gmail.com (Stephan Hoyer)
Date: Thu, 11 May 2017 09:23:20 -0700
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAL1kJvANUVy0WuSs7cJX6hmsCkpEH=hYt-hQ_bVhCUnWNWh69w@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <BY2PR08MB095EEBCC6A9A7FE15381931CBED0@BY2PR08MB095.namprd08.prod.outlook.com>
 <CAL1kJvANUVy0WuSs7cJX6hmsCkpEH=hYt-hQ_bVhCUnWNWh69w@mail.gmail.com>
Message-ID: <CAEQ_Tvf7UXYZ+o7J6ybPySgLBpkL_cnce5MrWyGhPdkdRSJu4w@mail.gmail.com>

Also, as friendly reminder, GitHub is a better place for bug reports than
mailing lists with hundreds of subscribers :).

On Thu, May 11, 2017 at 6:56 AM, Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> Nadav: Can you provide a testcase that fails?
>
> I don't think you're correct - it works just fine when `axis = a.ndims` -
> the issue arises when `axis > a.ndims`, but I'd argue that in that case an
> error is correct behaviour. But still a change, so perhaps needs a release
> note entry
>
> On Thu, 11 May 2017 at 14:25 Nadav Horesh <nadavh at visionsense.com> wrote:
>
>> There is a change to "expand_dims" function, that it is now does not
>> allow axis = a.ndims.
>>
>> This influences matplotlib function get_bending_matrices in
>> triinterpolate.py
>>
>>
>>   Nadav
>> ------------------------------
>> *From:* NumPy-Discussion <numpy-discussion-bounces+nadavh=
>> visionsense.com at python.org> on behalf of Charles R Harris <
>> charlesr.harris at gmail.com>
>> *Sent:* 11 May 2017 04:48:34
>> *To:* numpy-discussion; SciPy-User; SciPy Developers List;
>> python-announce-list at python.org
>> *Subject:* [Numpy-discussion] NumPy v1.13.0rc1 released.
>>
>> Hi All,
>>
>> I'm please to announce the NumPy 1.13.0rc1 release. This release supports
>> Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
>> most ambitious releases in the last several years. Some of the highlights
>> and new functions are
>>
>> *Highlights*
>>
>>    - Operations like ``a + b + c`` will reuse temporaries on some
>>    platforms, resulting in less memory use and faster execution.
>>    - Inplace operations check if inputs overlap outputs and create
>>    temporaries to avoid problems.
>>    - New __array_ufunc__ attribute provides improved ability for classes
>>    to override default ufunc behavior.
>>    -  New np.block function for creating blocked arrays.
>>
>>
>> *New functions*
>>
>>    - New ``np.positive`` ufunc.
>>    - New ``np.divmod`` ufunc provides more efficient divmod.
>>    - New ``np.isnat`` ufunc tests for NaT special values.
>>    - New ``np.heaviside`` ufunc computes the Heaviside function.
>>    - New ``np.isin`` function, improves on ``in1d``.
>>    - New ``np.block`` function for creating blocked arrays.
>>    - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
>>
>> Wheels for the pre-release are available on PyPI. Source tarballs,
>> zipfiles, release notes, and the Changelog are available on github
>> <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.
>>
>> A total of 100 people contributed to this release.  People with a "+" by
>> their
>> names contributed a patch for the first time.
>>
>>    - A. Jesse Jiryu Davis +
>>    - Alessandro Pietro Bardelli +
>>    - Alex Rothberg +
>>    - Alexander Shadchin
>>    - Allan Haldane
>>    - Andres Guzman-Ballen +
>>    - Antoine Pitrou
>>    - Antony Lee
>>    - B R S Recht +
>>    - Baurzhan Muftakhidinov +
>>    - Ben Rowland
>>    - Benda Xu +
>>    - Blake Griffith
>>    - Bradley Wogsland +
>>    - Brandon Carter +
>>    - CJ Carey
>>    - Charles Harris
>>    - Danny Hermes +
>>    - Duke Vijitbenjaronk +
>>    - Egor Klenin +
>>    - Elliott Forney +
>>    - Elliott M Forney +
>>    - Endolith
>>    - Eric Wieser
>>    - Erik M. Bray
>>    - Eugene +
>>    - Evan Limanto +
>>    - Felix Berkenkamp +
>>    - Fran?ois Bissey +
>>    - Frederic Bastien
>>    - Greg Young
>>    - Gregory R. Lee
>>    - Importance of Being Ernest +
>>    - Jaime Fernandez
>>    - Jakub Wilk +
>>    - James Cowgill +
>>    - James Sanders
>>    - Jean Utke +
>>    - Jesse Thoren +
>>    - Jim Crist +
>>    - Joerg Behrmann +
>>    - John Kirkham
>>    - Jonathan Helmus
>>    - Jonathan L Long
>>    - Jonathan Tammo Siebert +
>>    - Joseph Fox-Rabinovitz
>>    - Joshua Loyal +
>>    - Juan Nunez-Iglesias +
>>    - Julian Taylor
>>    - Kirill Balunov +
>>    - Likhith Chitneni +
>>    - Lo?c Est?ve
>>    - Mads Ohm Larsen
>>    - Marein K?nings +
>>    - Marten van Kerkwijk
>>    - Martin Thoma
>>    - Martino Sorbaro +
>>    - Marvin Schmidt +
>>    - Matthew Brett
>>    - Matthias Bussonnier +
>>    - Matthias C. M. Troffaes +
>>    - Matti Picus
>>    - Michael Seifert
>>    - Mikhail Pak +
>>    - Mortada Mehyar
>>    - Nathaniel J. Smith
>>    - Nick Papior
>>    - Oscar Villellas +
>>    - Pauli Virtanen
>>    - Pavel Potocek
>>    - Pete Peeradej Tanruangporn +
>>    - Philipp A +
>>    - Ralf Gommers
>>    - Robert Kern
>>    - Roland Kaufmann +
>>    - Ronan Lamy
>>    - Sami Salonen +
>>    - Sanchez Gonzalez Alvaro
>>    - Sebastian Berg
>>    - Shota Kawabuchi
>>    - Simon Gibbons
>>    - Stefan Otte
>>    - Stefan Peterson +
>>    - Stephan Hoyer
>>    - S?ren Fuglede J?rgensen +
>>    - Takuya Akiba
>>    - Tom Boyd +
>>    - Ville Skytt? +
>>    - Warren Weckesser
>>    - Wendell Smith
>>    - Yu Feng
>>    - Zixu Zhao +
>>    - Z? Vin?cius +
>>    - aha66 +
>>    - davidjn +
>>    - drabach +
>>    - drlvk +
>>    - jsh9 +
>>    - solarjoe +
>>    - zengi +
>>
>> Cheers,
>>
>> Chuck
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/f7dbe256/attachment.html>

From isaac.gerg at gergltd.com  Thu May 11 12:35:23 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 12:35:23 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
Message-ID: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>

I have 2 arrays, a and b which are rec arrays of length 10.  Each array has
5 columns.

I would like to combine all the columns into a single recarray with 10
columns and length 10.

Thanks,
Isaac
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/4a63ddf9/attachment.html>

From isaacpei at gmail.com  Thu May 11 13:17:49 2017
From: isaacpei at gmail.com (Isaac Xin Pei)
Date: Thu, 11 May 2017 17:17:49 +0000
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
Message-ID: <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>

Check Pandas pd.concate ?
On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> I have 2 arrays, a and b which are rec arrays of length 10.  Each array
> has 5 columns.
>
> I would like to combine all the columns into a single recarray with 10
> columns and length 10.
>
> Thanks,
> Isaac
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/356112a9/attachment.html>

From isaac.gerg at gergltd.com  Thu May 11 13:51:22 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 13:51:22 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
Message-ID: <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>

I'd prefer to stay in numpy land if possible.

On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com> wrote:

> Check Pandas pd.concate ?
> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
> wrote:
>
>> I have 2 arrays, a and b which are rec arrays of length 10.  Each array
>> has 5 columns.
>>
>> I would like to combine all the columns into a single recarray with 10
>> columns and length 10.
>>
>> Thanks,
>> Isaac
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/3e45cf6a/attachment-0001.html>

From ben.v.root at gmail.com  Thu May 11 14:02:17 2017
From: ben.v.root at gmail.com (Benjamin Root)
Date: Thu, 11 May 2017 14:02:17 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
Message-ID: <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>

Check in numpy.recfunctions. I know there is merge_arrays() and
stack_arrays(). I forget which one does what.

Cheers!
Ben Root


On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> I'd prefer to stay in numpy land if possible.
>
> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com> wrote:
>
>> Check Pandas pd.concate ?
>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>> wrote:
>>
>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each array
>>> has 5 columns.
>>>
>>> I would like to combine all the columns into a single recarray with 10
>>> columns and length 10.
>>>
>>> Thanks,
>>> Isaac
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/ef6ebcf1/attachment.html>

From isaac.gerg at gergltd.com  Thu May 11 14:28:26 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 14:28:26 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
Message-ID: <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>

newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)

yeilds:

builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
internal function

On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com> wrote:

> Check in numpy.recfunctions. I know there is merge_arrays() and
> stack_arrays(). I forget which one does what.
>
> Cheers!
> Ben Root
>
>
> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
> wrote:
>
>> I'd prefer to stay in numpy land if possible.
>>
>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>> wrote:
>>
>>> Check Pandas pd.concate ?
>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>> wrote:
>>>
>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each array
>>>> has 5 columns.
>>>>
>>>> I would like to combine all the columns into a single recarray with 10
>>>> columns and length 10.
>>>>
>>>> Thanks,
>>>> Isaac
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/f550db51/attachment.html>

From wieser.eric+numpy at gmail.com  Thu May 11 14:31:19 2017
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Thu, 11 May 2017 18:31:19 +0000
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
Message-ID: <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>

If that's the case, can you file an issue on github along with a minimal
example that produces the error, and the full stack trace?

On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>
> yeilds:
>
> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
> internal function
>
> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
> wrote:
>
>> Check in numpy.recfunctions. I know there is merge_arrays() and
>> stack_arrays(). I forget which one does what.
>>
>> Cheers!
>> Ben Root
>>
>>
>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>> wrote:
>>
>>> I'd prefer to stay in numpy land if possible.
>>>
>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>> wrote:
>>>
>>>> Check Pandas pd.concate ?
>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>> wrote:
>>>>
>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>> array has 5 columns.
>>>>>
>>>>> I would like to combine all the columns into a single recarray with 10
>>>>> columns and length 10.
>>>>>
>>>>> Thanks,
>>>>> Isaac
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/8e996408/attachment-0001.html>

From isaac.gerg at gergltd.com  Thu May 11 14:34:50 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 14:34:50 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
Message-ID: <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>

Sure.

On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> If that's the case, can you file an issue on github along with a minimal
> example that produces the error, and the full stack trace?
>
> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>
>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>
>> yeilds:
>>
>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>> internal function
>>
>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>> wrote:
>>
>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>> stack_arrays(). I forget which one does what.
>>>
>>> Cheers!
>>> Ben Root
>>>
>>>
>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>> wrote:
>>>
>>>> I'd prefer to stay in numpy land if possible.
>>>>
>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>> wrote:
>>>>
>>>>> Check Pandas pd.concate ?
>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>>> wrote:
>>>>>
>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>> array has 5 columns.
>>>>>>
>>>>>> I would like to combine all the columns into a single recarray with
>>>>>> 10 columns and length 10.
>>>>>>
>>>>>> Thanks,
>>>>>> Isaac
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/b79d4825/attachment.html>

From isaac.gerg at gergltd.com  Thu May 11 14:52:21 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 14:52:21 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
Message-ID: <CADnN3yUq9+GBXNC1+JrdpRG8DAsrX6OiR66HgHvhNXweBROBzA@mail.gmail.com>

Looking at the code, i think merge and stack take in ndarrays, not recarrays
is that correct?

On Thu, May 11, 2017 at 2:34 PM, Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> Sure.
>
> On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
> wrote:
>
>> If that's the case, can you file an issue on github along with a minimal
>> example that produces the error, and the full stack trace?
>>
>> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>
>>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>>
>>> yeilds:
>>>
>>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>>> internal function
>>>
>>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>>> wrote:
>>>
>>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>>> stack_arrays(). I forget which one does what.
>>>>
>>>> Cheers!
>>>> Ben Root
>>>>
>>>>
>>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>>> wrote:
>>>>
>>>>> I'd prefer to stay in numpy land if possible.
>>>>>
>>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Check Pandas pd.concate ?
>>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>>> array has 5 columns.
>>>>>>>
>>>>>>> I would like to combine all the columns into a single recarray with
>>>>>>> 10 columns and length 10.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Isaac
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/3caefac1/attachment.html>

From m.h.vankerkwijk at gmail.com  Thu May 11 15:51:57 2017
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Thu, 11 May 2017 15:51:57 -0400
Subject: [Numpy-discussion] NumPy v1.13.0rc1 released.
In-Reply-To: <CAEQ_Tvf7UXYZ+o7J6ybPySgLBpkL_cnce5MrWyGhPdkdRSJu4w@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <BY2PR08MB095EEBCC6A9A7FE15381931CBED0@BY2PR08MB095.namprd08.prod.outlook.com>
 <CAL1kJvANUVy0WuSs7cJX6hmsCkpEH=hYt-hQ_bVhCUnWNWh69w@mail.gmail.com>
 <CAEQ_Tvf7UXYZ+o7J6ybPySgLBpkL_cnce5MrWyGhPdkdRSJu4w@mail.gmail.com>
Message-ID: <CAJNV+9tC1fc_VKKr0LYgASgEHkkJ0-qDqWqHQwwEbCB_bhY8Bw@mail.gmail.com>

Hi All,

Do indeed try __array_ufunc__! It should make many things work much
better and possibly faster than was possible with __array_prepare__
and __array_wrap__ (for astropy's Quantity, an ndarray subclass than I
maintain, it gets us a factor of almost 2 in speed for operations
where scaling for units is needed; for array-like classes, it should
enable things that were nearlyimpossible before).

About documentation, I'm not quite sure how to get the documentation
on scipy.docs.org to update (the developer version points points to
1.13dev0 and is last updated in January). Beyond the links Nathaniel
sent, there is also one more specifically aimed at subclasses:

 https://github.com/numpy/numpy/blob/master/numpy/doc/subclassing.py

As noted, the API is provisional, which I hope it all the more reason
to try it out if you think this could be useful -- and do let us know
on github what works and what doesn't [e.g., there is a lively
discussion on whether `ndarray` having its own `__array_ufunc__` for
subclasses to super to is in fact a good idea -- for Quantity, I find
it quite convenient (but then I implemented it...), but perhaps it is
too confusing.  See

https://github.com/numpy/numpy/issues/9079

All the best,

Marten

From derek at astro.physik.uni-goettingen.de  Thu May 11 15:12:16 2017
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Thu, 11 May 2017 21:12:16 +0200
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yUq9+GBXNC1+JrdpRG8DAsrX6OiR66HgHvhNXweBROBzA@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
 <CADnN3yUq9+GBXNC1+JrdpRG8DAsrX6OiR66HgHvhNXweBROBzA@mail.gmail.com>
Message-ID: <3AF0F8E7-F55C-4340-A01E-E1B5AE92AA3B@astro.physik.uni-goettingen.de>

On 11 May 2017, at 8:52 pm, Isaac Gerg <isaac.gerg at gergltd.com> wrote:
> 
> Looking at the code, i think merge and stack take in ndarrays, not recarrays
> is that correct?

It should accept either; however if your a and b are two recarrays with all uniquely named columns
to get the 10-column recarray in your original example you should do

newtable = np.lib.recfunctions.merge_arrays([a, b], flatten=True, asrecarray=True)

otherwise newtable will be a recarray of two columns with 5 nested subcolumns each
(i.e. each column will have a dtype [(?col0?, ?<f8?), (?col1?, ? (?col4, ?<f8?)])

Duplicate column names will raise an error with ?flatten=True?.


				Derek


From wieser.eric+numpy at gmail.com  Thu May 11 17:07:26 2017
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Thu, 11 May 2017 21:07:26 +0000
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
Message-ID: <CAL1kJvA8DYmhR1nfOV4xLrU7t839vq6onDXkKH-deT_t_VFcgA@mail.gmail.com>

Even if you solve your own problem, please do - a SystemError is 100% a
mistake in numpy, and should never be raised from python code, even if you
call a numpy function with the wrong inputs.

Eric

On Thu, 11 May 2017 at 19:35 Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> Sure.
>
> On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
> wrote:
>
>> If that's the case, can you file an issue on github along with a minimal
>> example that produces the error, and the full stack trace?
>>
>> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>
>>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>>
>>> yeilds:
>>>
>>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>>> internal function
>>>
>>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>>> wrote:
>>>
>>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>>> stack_arrays(). I forget which one does what.
>>>>
>>>> Cheers!
>>>> Ben Root
>>>>
>>>>
>>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>>> wrote:
>>>>
>>>>> I'd prefer to stay in numpy land if possible.
>>>>>
>>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Check Pandas pd.concate ?
>>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>>> array has 5 columns.
>>>>>>>
>>>>>>> I would like to combine all the columns into a single recarray with
>>>>>>> 10 columns and length 10.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Isaac
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/4a967195/attachment.html>

From isaac.gerg at gergltd.com  Thu May 11 17:17:08 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Thu, 11 May 2017 17:17:08 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CAL1kJvA8DYmhR1nfOV4xLrU7t839vq6onDXkKH-deT_t_VFcgA@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
 <CAL1kJvA8DYmhR1nfOV4xLrU7t839vq6onDXkKH-deT_t_VFcgA@mail.gmail.com>
Message-ID: <CADnN3yUVjUx3LhXxopJ-5OEXDXSNKSfs3xfvvQJE6DCTufo3ow@mail.gmail.com>

Will do. I'm working on my side to get a solution that works.  It seems
that when i use append_field i dont get a recarray back and when i call
asrecarray=True, i get an error.  once i fix this.  i will replicate the
error and post to github.

On Thu, May 11, 2017 at 5:07 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> Even if you solve your own problem, please do - a SystemError is 100% a
> mistake in numpy, and should never be raised from python code, even if you
> call a numpy function with the wrong inputs.
>
> Eric
>
>
> On Thu, 11 May 2017 at 19:35 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>
>> Sure.
>>
>> On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <wieser.eric+numpy at gmail.com
>> > wrote:
>>
>>> If that's the case, can you file an issue on github along with a minimal
>>> example that produces the error, and the full stack trace?
>>>
>>> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>>
>>>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>>>
>>>> yeilds:
>>>>
>>>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>>>> internal function
>>>>
>>>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>>>> wrote:
>>>>
>>>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>>>> stack_arrays(). I forget which one does what.
>>>>>
>>>>> Cheers!
>>>>> Ben Root
>>>>>
>>>>>
>>>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>>>> wrote:
>>>>>
>>>>>> I'd prefer to stay in numpy land if possible.
>>>>>>
>>>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Check Pandas pd.concate ?
>>>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>>>> array has 5 columns.
>>>>>>>>
>>>>>>>> I would like to combine all the columns into a single recarray with
>>>>>>>> 10 columns and length 10.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Isaac
>>>>>>>> _______________________________________________
>>>>>>>> NumPy-Discussion mailing list
>>>>>>>> NumPy-Discussion at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170511/42ae8991/attachment-0001.html>

From isaac.gerg at gergltd.com  Fri May 12 09:59:19 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Fri, 12 May 2017 09:59:19 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yUVjUx3LhXxopJ-5OEXDXSNKSfs3xfvvQJE6DCTufo3ow@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
 <CAL1kJvA8DYmhR1nfOV4xLrU7t839vq6onDXkKH-deT_t_VFcgA@mail.gmail.com>
 <CADnN3yUVjUx3LhXxopJ-5OEXDXSNKSfs3xfvvQJE6DCTufo3ow@mail.gmail.com>
Message-ID: <CADnN3yW+9sTuYDcdRKrKKcu7v41WA7CADqzufU=uvr5uzchCdw@mail.gmail.com>

I was able to fix my problem. One of the types in the recarray was of type
object when it should have been S27.  Type object was the result of reading
in a matlab generated hdf5 file which had a string for one of the matrix
columns.  I converted this column manually to a S27.

I am reverting my code back to where i had the systemerror to post.

On Thu, May 11, 2017 at 5:17 PM, Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> Will do. I'm working on my side to get a solution that works.  It seems
> that when i use append_field i dont get a recarray back and when i call
> asrecarray=True, i get an error.  once i fix this.  i will replicate the
> error and post to github.
>
> On Thu, May 11, 2017 at 5:07 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
> wrote:
>
>> Even if you solve your own problem, please do - a SystemError is 100% a
>> mistake in numpy, and should never be raised from python code, even if you
>> call a numpy function with the wrong inputs.
>>
>> Eric
>>
>>
>> On Thu, 11 May 2017 at 19:35 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>
>>> Sure.
>>>
>>> On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <
>>> wieser.eric+numpy at gmail.com> wrote:
>>>
>>>> If that's the case, can you file an issue on github along with a
>>>> minimal example that produces the error, and the full stack trace?
>>>>
>>>> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>>>
>>>>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>>>>
>>>>> yeilds:
>>>>>
>>>>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>>>>> internal function
>>>>>
>>>>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>>>>> stack_arrays(). I forget which one does what.
>>>>>>
>>>>>> Cheers!
>>>>>> Ben Root
>>>>>>
>>>>>>
>>>>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'd prefer to stay in numpy land if possible.
>>>>>>>
>>>>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Check Pandas pd.concate ?
>>>>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>>>>> array has 5 columns.
>>>>>>>>>
>>>>>>>>> I would like to combine all the columns into a single recarray
>>>>>>>>> with 10 columns and length 10.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Isaac
>>>>>>>>> _______________________________________________
>>>>>>>>> NumPy-Discussion mailing list
>>>>>>>>> NumPy-Discussion at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> NumPy-Discussion mailing list
>>>>>>>> NumPy-Discussion at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170512/bbf0d0fd/attachment.html>

From jjmo at dtu.dk  Fri May 12 10:28:12 2017
From: jjmo at dtu.dk (=?UTF-8?Q?Jens_J=c3=b8rgen_Mortensen?=)
Date: Fri, 12 May 2017 16:28:12 +0200
Subject: [Numpy-discussion] [SciPy-User] NumPy v1.13.0rc1 released.
In-Reply-To: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
Message-ID: <9df05057-b068-7158-a525-90d7ea89fe2d@dtu.dk>

Den 11-05-2017 kl. 03:48 skrev Charles R Harris:
> Hi All,
>
> I'm please to announce the NumPy 1.13.0rc1 release. This release 
> supports Python 2.7 and 3.4-3.6 and contains many new features. It is 
> one of the most ambitious releases in the last several years. Some of 
> the highlights and new functions are

I found this strange behavior:

(np113) [jensj at ASUS np113]$ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy as np
 >>> np.__version__
'1.13.0rc1'
 >>> s = (27, 27, 27)
 >>> x = np.ones(s, complex)
 >>> y = np.zeros(s)
 >>> y += abs(x * 2.0)**2
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: Cannot cast ufunc add output from dtype('complex128') to 
dtype('float64') with casting rule 'same_kind'

Works OK with s=(3,3,3).

Jens J?rgen

>
> *Highlights*
>
>   * Operations like ``a + b + c`` will reuse temporaries on some
>     platforms, resulting in less memory use and faster execution.
>   * Inplace operations check if inputs overlap outputs and create
>     temporaries to avoid problems.
>   * New __array_ufunc__ attribute provides improved ability for
>     classes to override default ufunc behavior.
>   *  New np.block function for creating blocked arrays.
>
>
> *New functions*
>
>   * New ``np.positive`` ufunc.
>   * New ``np.divmod`` ufunc provides more efficient divmod.
>   * New ``np.isnat`` ufunc tests for NaT special values.
>   * New ``np.heaviside`` ufunc computes the Heaviside function.
>   * New ``np.isin`` function, improves on ``in1d``.
>   * New ``np.block`` function for creating blocked arrays.
>   * New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
>
> Wheels for the pre-release are available on PyPI. Source tarballs, 
> zipfiles, release notes, and the Changelog are available on github 
> <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>.
>
> A total of 100 people contributed to this release.  People with a "+" 
> by their
> names contributed a patch for the first time.
>
>   * A. Jesse Jiryu Davis +
>   * Alessandro Pietro Bardelli +
>   * Alex Rothberg +
>   * Alexander Shadchin
>   * Allan Haldane
>   * Andres Guzman-Ballen +
>   * Antoine Pitrou
>   * Antony Lee
>   * B R S Recht +
>   * Baurzhan Muftakhidinov +
>   * Ben Rowland
>   * Benda Xu +
>   * Blake Griffith
>   * Bradley Wogsland +
>   * Brandon Carter +
>   * CJ Carey
>   * Charles Harris
>   * Danny Hermes +
>   * Duke Vijitbenjaronk +
>   * Egor Klenin +
>   * Elliott Forney +
>   * Elliott M Forney +
>   * Endolith
>   * Eric Wieser
>   * Erik M. Bray
>   * Eugene +
>   * Evan Limanto +
>   * Felix Berkenkamp +
>   * Fran?ois Bissey +
>   * Frederic Bastien
>   * Greg Young
>   * Gregory R. Lee
>   * Importance of Being Ernest +
>   * Jaime Fernandez
>   * Jakub Wilk +
>   * James Cowgill +
>   * James Sanders
>   * Jean Utke +
>   * Jesse Thoren +
>   * Jim Crist +
>   * Joerg Behrmann +
>   * John Kirkham
>   * Jonathan Helmus
>   * Jonathan L Long
>   * Jonathan Tammo Siebert +
>   * Joseph Fox-Rabinovitz
>   * Joshua Loyal +
>   * Juan Nunez-Iglesias +
>   * Julian Taylor
>   * Kirill Balunov +
>   * Likhith Chitneni +
>   * Lo?c Est?ve
>   * Mads Ohm Larsen
>   * Marein K?nings +
>   * Marten van Kerkwijk
>   * Martin Thoma
>   * Martino Sorbaro +
>   * Marvin Schmidt +
>   * Matthew Brett
>   * Matthias Bussonnier +
>   * Matthias C. M. Troffaes +
>   * Matti Picus
>   * Michael Seifert
>   * Mikhail Pak +
>   * Mortada Mehyar
>   * Nathaniel J. Smith
>   * Nick Papior
>   * Oscar Villellas +
>   * Pauli Virtanen
>   * Pavel Potocek
>   * Pete Peeradej Tanruangporn +
>   * Philipp A +
>   * Ralf Gommers
>   * Robert Kern
>   * Roland Kaufmann +
>   * Ronan Lamy
>   * Sami Salonen +
>   * Sanchez Gonzalez Alvaro
>   * Sebastian Berg
>   * Shota Kawabuchi
>   * Simon Gibbons
>   * Stefan Otte
>   * Stefan Peterson +
>   * Stephan Hoyer
>   * S?ren Fuglede J?rgensen +
>   * Takuya Akiba
>   * Tom Boyd +
>   * Ville Skytt? +
>   * Warren Weckesser
>   * Wendell Smith
>   * Yu Feng
>   * Zixu Zhao +
>   * Z? Vin?cius +
>   * aha66 +
>   * davidjn +
>   * drabach +
>   * drlvk +
>   * jsh9 +
>   * solarjoe +
>   * zengi +
>
> Cheers,
>
> Chuck
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at python.org
> https://mail.python.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170512/382bdc52/attachment-0001.html>

From jtaylor.debian at googlemail.com  Fri May 12 11:25:52 2017
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 12 May 2017 17:25:52 +0200
Subject: [Numpy-discussion] [SciPy-User] NumPy v1.13.0rc1 released.
In-Reply-To: <9df05057-b068-7158-a525-90d7ea89fe2d@dtu.dk>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <9df05057-b068-7158-a525-90d7ea89fe2d@dtu.dk>
Message-ID: <b42986a7-bd54-77f8-c90f-c31cf315da79@googlemail.com>

On 12.05.2017 16:28, Jens J?rgen Mortensen wrote:
> Den 11-05-2017 kl. 03:48 skrev Charles R Harris:
>> Hi All,
>>
>> I'm please to announce the NumPy 1.13.0rc1 release. This release
>> supports Python 2.7 and 3.4-3.6 and contains many new features. It is
>> one of the most ambitious releases in the last several years. Some of
>> the highlights and new functions are
> 
> I found this strange behavior:
> 
> (np113) [jensj at ASUS np113]$ python3
> Python 3.5.3 (default, Jan 19 2017, 14:11:04)
> [GCC 6.3.0 20170118] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import numpy as np
>>>> np.__version__
> '1.13.0rc1'
>>>> s = (27, 27, 27)
>>>> x = np.ones(s, complex)
>>>> y = np.zeros(s)
>>>> y += abs(x * 2.0)**2
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: Cannot cast ufunc add output from dtype('complex128') to
> dtype('float64') with casting rule 'same_kind'
> 
> Works OK with s=(3,3,3).
> 
> Jens J?rgen
> 

Thanks, abs() should not elide complex numbers as it changes types.
Something not handled for unary functions.
As a workaround you can use np.abs() which currently does not try to elide.

From sebastian at sipsolutions.net  Fri May 12 11:24:00 2017
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 12 May 2017 17:24:00 +0200
Subject: [Numpy-discussion] [SciPy-User] NumPy v1.13.0rc1 released.
In-Reply-To: <9df05057-b068-7158-a525-90d7ea89fe2d@dtu.dk>
References: <CAB6mnx+78KwqkfUN65RQMefXNBG4myLWKCmGjTQ8GYTg4Qjd5Q@mail.gmail.com>
 <9df05057-b068-7158-a525-90d7ea89fe2d@dtu.dk>
Message-ID: <1494602640.32266.2.camel@sipsolutions.net>

On Fri, 2017-05-12 at 16:28 +0200, Jens J?rgen Mortensen wrote:
> Den 11-05-2017 kl. 03:48 skrev Charles R Harris:
> > Hi All,
> > 
> > I'm please to announce the NumPy 1.13.0rc1 release. This release
> > supports Python 2.7 and 3.4-3.6 and contains many new features. It
> > is one of the most ambitious releases in the last several years.
> > Some of the highlights and new functions are
> ?
> I found this strange behavior:
> 
> (np113) [jensj at ASUS np113]$ python3
> Python 3.5.3 (default, Jan 19 2017, 14:11:04)?
> [GCC 6.3.0 20170118] on linux
> Type "help", "copyright", "credits" or "license" for more
> information.
> >>> import numpy as np
> >>> np.__version__
> '1.13.0rc1'
> >>> s = (27, 27, 27)
> >>> x = np.ones(s, complex)
> >>> y = np.zeros(s)
> >>> y += abs(x * 2.0)**2
> Traceback (most recent call last):
> ? File "<stdin>", line 1, in <module>
> TypeError: Cannot cast ufunc add output from dtype('complex128') to
> dtype('float64') with casting rule 'same_kind'
> 
> Works OK with s=(3,3,3).
> 

I have opened an issue: https://github.com/numpy/numpy/issues/9109

since it is so "odd", I expect it is due to the temporary elision
kicking in when it should not in this case.

- Sebastian


> Jens J?rgen
> 
> > Highlights
> > Operations like ``a + b + c`` will reuse temporaries on some
> > platforms, resulting in less memory use and faster execution.
> > Inplace operations check if inputs overlap outputs and create
> > temporaries to avoid problems.
> > New __array_ufunc__ attribute provides improved ability for classes
> > to override default ufunc behavior.
> > ?New np.block function for creating blocked arrays.
> > 
> > New functions
> > New ``np.positive`` ufunc.
> > New ``np.divmod`` ufunc provides more efficient divmod.
> > New ``np.isnat`` ufunc tests for NaT special values.
> > New ``np.heaviside`` ufunc computes the Heaviside function.
> > New ``np.isin`` function, improves on ``in1d``.
> > New ``np.block`` function for creating blocked arrays.
> > New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.
> > Wheels for the pre-release are available on PyPI. Source tarballs,
> > zipfiles, release notes, and the Changelog are available on github.
> > 
> > A total of 100 people contributed to this release.? People with a
> > "+" by their
> > names contributed a patch for the first time.
> > A. Jesse Jiryu Davis +
> > Alessandro Pietro Bardelli +
> > Alex Rothberg +
> > Alexander Shadchin
> > Allan Haldane
> > Andres Guzman-Ballen +
> > Antoine Pitrou
> > Antony Lee
> > B R S Recht +
> > Baurzhan Muftakhidinov +
> > Ben Rowland
> > Benda Xu +
> > Blake Griffith
> > Bradley Wogsland +
> > Brandon Carter +
> > CJ Carey
> > Charles Harris
> > Danny Hermes +
> > Duke Vijitbenjaronk +
> > Egor Klenin +
> > Elliott Forney +
> > Elliott M Forney +
> > Endolith
> > Eric Wieser
> > Erik M. Bray
> > Eugene +
> > Evan Limanto +
> > Felix Berkenkamp +
> > Fran?ois Bissey +
> > Frederic Bastien
> > Greg Young
> > Gregory R. Lee
> > Importance of Being Ernest +
> > Jaime Fernandez
> > Jakub Wilk +
> > James Cowgill +
> > James Sanders
> > Jean Utke +
> > Jesse Thoren +
> > Jim Crist +
> > Joerg Behrmann +
> > John Kirkham
> > Jonathan Helmus
> > Jonathan L Long
> > Jonathan Tammo Siebert +
> > Joseph Fox-Rabinovitz
> > Joshua Loyal +
> > Juan Nunez-Iglesias +
> > Julian Taylor
> > Kirill Balunov +
> > Likhith Chitneni +
> > Lo?c Est?ve
> > Mads Ohm Larsen
> > Marein K?nings +
> > Marten van Kerkwijk
> > Martin Thoma
> > Martino Sorbaro +
> > Marvin Schmidt +
> > Matthew Brett
> > Matthias Bussonnier +
> > Matthias C. M. Troffaes +
> > Matti Picus
> > Michael Seifert
> > Mikhail Pak +
> > Mortada Mehyar
> > Nathaniel J. Smith
> > Nick Papior
> > Oscar Villellas +
> > Pauli Virtanen
> > Pavel Potocek
> > Pete Peeradej Tanruangporn +
> > Philipp A +
> > Ralf Gommers
> > Robert Kern
> > Roland Kaufmann +
> > Ronan Lamy
> > Sami Salonen +
> > Sanchez Gonzalez Alvaro
> > Sebastian Berg
> > Shota Kawabuchi
> > Simon Gibbons
> > Stefan Otte
> > Stefan Peterson +
> > Stephan Hoyer
> > S?ren Fuglede J?rgensen +
> > Takuya Akiba
> > Tom Boyd +
> > Ville Skytt? +
> > Warren Weckesser
> > Wendell Smith
> > Yu Feng
> > Zixu Zhao +
> > Z? Vin?cius +
> > aha66 +
> > davidjn +
> > drabach +
> > drlvk +
> > jsh9 +
> > solarjoe +
> > zengi +
> > Cheers,
> > 
> > Chuck
> > 
> > 
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at python.org
> > https://mail.python.org/mailman/listinfo/scipy-user
> ?
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170512/26fe6bbc/attachment.sig>

From isaac.gerg at gergltd.com  Fri May 12 16:03:58 2017
From: isaac.gerg at gergltd.com (Isaac Gerg)
Date: Fri, 12 May 2017 16:03:58 -0400
Subject: [Numpy-discussion] How to concatenate 2 recarrays into a single
 recarray
In-Reply-To: <CADnN3yW+9sTuYDcdRKrKKcu7v41WA7CADqzufU=uvr5uzchCdw@mail.gmail.com>
References: <CADnN3yVVKHT=CuPU2ij7XdA9dLQyXcrY2KO9JqTyKxom=-Kf-w@mail.gmail.com>
 <CAGWjKHh9+6-mbhJ=Ukm6ub=tPU8AqRqNbBFiYQa8G0Nw8JRcgw@mail.gmail.com>
 <CADnN3yU-9HyHx6_DV7HmvZtkMyjTiC0bzG4Hz11SyLR6zWn_gQ@mail.gmail.com>
 <CANNq6FmO3LHNKX7Hgw-24yfSK+VQ+_1v-eXdGgDe3yTY5qZTaQ@mail.gmail.com>
 <CADnN3yVwmPWGQ-EyuyDWFJi0kbg4kk+frNH-v4mz8eM_2HWt9Q@mail.gmail.com>
 <CAL1kJvD1hR8TmW0L_wYA5_zwQRBpOP2d73mgx4kyxgq8RqD9=g@mail.gmail.com>
 <CADnN3yUCLRPnCX6eOLXm3fBbQEStmAqQQDXRHQh4Ve8ewhGy0w@mail.gmail.com>
 <CAL1kJvA8DYmhR1nfOV4xLrU7t839vq6onDXkKH-deT_t_VFcgA@mail.gmail.com>
 <CADnN3yUVjUx3LhXxopJ-5OEXDXSNKSfs3xfvvQJE6DCTufo3ow@mail.gmail.com>
 <CADnN3yW+9sTuYDcdRKrKKcu7v41WA7CADqzufU=uvr5uzchCdw@mail.gmail.com>
Message-ID: <CADnN3yU9hZJ4n6bk8qxeASrqmn0c55q4A=US94SmGcpHg_9AMA@mail.gmail.com>

It looks like some of the problem I was seeing has to do with reading in
hdf5 files from matlab that contain strings for column types.  when they
are read in using h5py the type is reported as 'O' which is object.  The
problem is I can't reproduce the problem in small script because I can't
get hdf5 to create these files with 'O' type; I have to go through matlab.

On Fri, May 12, 2017 at 9:59 AM, Isaac Gerg <isaac.gerg at gergltd.com> wrote:

> I was able to fix my problem. One of the types in the recarray was of type
> object when it should have been S27.  Type object was the result of reading
> in a matlab generated hdf5 file which had a string for one of the matrix
> columns.  I converted this column manually to a S27.
>
> I am reverting my code back to where i had the systemerror to post.
>
> On Thu, May 11, 2017 at 5:17 PM, Isaac Gerg <isaac.gerg at gergltd.com>
> wrote:
>
>> Will do. I'm working on my side to get a solution that works.  It seems
>> that when i use append_field i dont get a recarray back and when i call
>> asrecarray=True, i get an error.  once i fix this.  i will replicate the
>> error and post to github.
>>
>> On Thu, May 11, 2017 at 5:07 PM, Eric Wieser <wieser.eric+numpy at gmail.com
>> > wrote:
>>
>>> Even if you solve your own problem, please do - a SystemError is 100% a
>>> mistake in numpy, and should never be raised from python code, even if you
>>> call a numpy function with the wrong inputs.
>>>
>>> Eric
>>>
>>>
>>> On Thu, 11 May 2017 at 19:35 Isaac Gerg <isaac.gerg at gergltd.com> wrote:
>>>
>>>> Sure.
>>>>
>>>> On Thu, May 11, 2017 at 2:31 PM, Eric Wieser <
>>>> wieser.eric+numpy at gmail.com> wrote:
>>>>
>>>>> If that's the case, can you file an issue on github along with a
>>>>> minimal example that produces the error, and the full stack trace?
>>>>>
>>>>> On Thu, 11 May 2017 at 19:29 Isaac Gerg <isaac.gerg at gergltd.com>
>>>>> wrote:
>>>>>
>>>>>> newtable = np.lib.recfunctions.merge_arrays([a, b], asrecarray=True)
>>>>>>
>>>>>> yeilds:
>>>>>>
>>>>>> builtins.SystemError: ..\Objects\dictobject.c:2054: bad argument to
>>>>>> internal function
>>>>>>
>>>>>> On Thu, May 11, 2017 at 2:02 PM, Benjamin Root <ben.v.root at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Check in numpy.recfunctions. I know there is merge_arrays() and
>>>>>>> stack_arrays(). I forget which one does what.
>>>>>>>
>>>>>>> Cheers!
>>>>>>> Ben Root
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 11, 2017 at 1:51 PM, Isaac Gerg <isaac.gerg at gergltd.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'd prefer to stay in numpy land if possible.
>>>>>>>>
>>>>>>>> On Thu, May 11, 2017 at 1:17 PM, Isaac Xin Pei <isaacpei at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Check Pandas pd.concate ?
>>>>>>>>> On Thu, May 11, 2017 at 12:45 PM Isaac Gerg <
>>>>>>>>> isaac.gerg at gergltd.com> wrote:
>>>>>>>>>
>>>>>>>>>> I have 2 arrays, a and b which are rec arrays of length 10.  Each
>>>>>>>>>> array has 5 columns.
>>>>>>>>>>
>>>>>>>>>> I would like to combine all the columns into a single recarray
>>>>>>>>>> with 10 columns and length 10.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Isaac
>>>>>>>>>> _______________________________________________
>>>>>>>>>> NumPy-Discussion mailing list
>>>>>>>>>> NumPy-Discussion at python.org
>>>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> NumPy-Discussion mailing list
>>>>>>>>> NumPy-Discussion at python.org
>>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> NumPy-Discussion mailing list
>>>>>>>> NumPy-Discussion at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at python.org
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170512/02fc4426/attachment-0001.html>

From njs at pobox.com  Sun May 14 01:45:53 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 13 May 2017 22:45:53 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on NumPy
Message-ID: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>

Hi all,

As some of you know, I've been working for... quite some time now to
try to secure funding for NumPy. So I'm excited that I can now
officially announce that BIDS [1] is planning to hire several folks
specifically to work on NumPy. These will full time positions at UC
Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
and the general goal will be to work on some of the major priorities
we identified at the last dev meeting: more flexible dtypes, better
interoperation with other array libraries, paying down technical debt,
and so forth. Though I'm sure the details will change as we start to
dig into things and engage with the community.

More details soon; universities move slowly, so nothing's going to
happen immediately. But this is definitely happening and I wanted to
get something out publicly before the conference season starts ? so if
you're someone who might be interested in coming to work with me and
the other awesome folks at BIDS, then this is a heads-up: drop me a
line and we can chat! I'll be at PyCon next week if anyone happens to
be there. And feel free to spread the word.

-n

[1] http://bids.berkeley.edu/

-- 
Nathaniel J. Smith -- https://vorpus.org

From chris.barker at noaa.gov  Sun May 14 17:11:40 2017
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sun, 14 May 2017 14:11:40 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
Message-ID: <-383731338024949696@unknownmsgid>

Awesome! This is really great news.

Does this mean is several person-years of funding secured?

-CHB

> On May 13, 2017, at 10:47 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
> Hi all,
>
> As some of you know, I've been working for... quite some time now to
> try to secure funding for NumPy. So I'm excited that I can now
> officially announce that BIDS [1] is planning to hire several folks
> specifically to work on NumPy. These will full time positions at UC
> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
> and the general goal will be to work on some of the major priorities
> we identified at the last dev meeting: more flexible dtypes, better
> interoperation with other array libraries, paying down technical debt,
> and so forth. Though I'm sure the details will change as we start to
> dig into things and engage with the community.
>
> More details soon; universities move slowly, so nothing's going to
> happen immediately. But this is definitely happening and I wanted to
> get something out publicly before the conference season starts ? so if
> you're someone who might be interested in coming to work with me and
> the other awesome folks at BIDS, then this is a heads-up: drop me a
> line and we can chat! I'll be at PyCon next week if anyone happens to
> be there. And feel free to spread the word.
>
> -n
>
> [1] http://bids.berkeley.edu/
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion

From charlesr.harris at gmail.com  Sun May 14 17:56:35 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 14 May 2017 15:56:35 -0600
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
Message-ID: <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>

On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Hi all,
>
> As some of you know, I've been working for... quite some time now to
> try to secure funding for NumPy. So I'm excited that I can now
> officially announce that BIDS [1] is planning to hire several folks
> specifically to work on NumPy. These will full time positions at UC
> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
> and the general goal will be to work on some of the major priorities
> we identified at the last dev meeting: more flexible dtypes, better
> interoperation with other array libraries, paying down technical debt,
> and so forth. Though I'm sure the details will change as we start to
> dig into things and engage with the community.
>
> More details soon; universities move slowly, so nothing's going to
> happen immediately. But this is definitely happening and I wanted to
> get something out publicly before the conference season starts ? so if
> you're someone who might be interested in coming to work with me and
> the other awesome folks at BIDS, then this is a heads-up: drop me a
> line and we can chat! I'll be at PyCon next week if anyone happens to
> be there. And feel free to spread the word.
>

Excellent news. Do you have any sort of timeline in mind?

It will be interesting to see what changes this leads to, both in the code
and in the project sociology.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170514/7eee8790/attachment.html>

From njs at pobox.com  Sun May 14 18:37:59 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 14 May 2017 15:37:59 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
Message-ID: <CAPJVwBkgfLf5_qU6UcQAQYpEO0GL6xKmnxY-3BZiHgv+Ljv2sQ@mail.gmail.com>

On Sun, May 14, 2017 at 2:56 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Hi all,
>>
>> As some of you know, I've been working for... quite some time now to
>> try to secure funding for NumPy. So I'm excited that I can now
>> officially announce that BIDS [1] is planning to hire several folks
>> specifically to work on NumPy. These will full time positions at UC
>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>> and the general goal will be to work on some of the major priorities
>> we identified at the last dev meeting: more flexible dtypes, better
>> interoperation with other array libraries, paying down technical debt,
>> and so forth. Though I'm sure the details will change as we start to
>> dig into things and engage with the community.
>>
>> More details soon; universities move slowly, so nothing's going to
>> happen immediately. But this is definitely happening and I wanted to
>> get something out publicly before the conference season starts ? so if
>> you're someone who might be interested in coming to work with me and
>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>> line and we can chat! I'll be at PyCon next week if anyone happens to
>> be there. And feel free to spread the word.
>
>
> Excellent news. Do you have any sort of timeline in mind?

The exact timeline's going to be determined in large part by
university+funder logistics. I thought it was going to happen last
year, so at this point I'm just going with the flow :-). The process
for hiring staff definitely takes a few months at a minimum; with
postdocs there's a little more flexibility.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Sun May 14 18:40:51 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 14 May 2017 15:40:51 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <-383731338024949696@unknownmsgid>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <-383731338024949696@unknownmsgid>
Message-ID: <CAPJVwBkzjiTtUJAdLrR8YZ8JtMR4mNuqpL9+eYp=c84j+BGagQ@mail.gmail.com>

On Sun, May 14, 2017 at 2:11 PM, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
> Awesome! This is really great news.
>
> Does this mean is several person-years of funding secured?

Yes ? hoping to give more details there soon. (There's nothing dire
and secretive, it's just the logistics of getting an announcement
approved by funder communication people didn't work with getting
something out by PyCon, so this is the slightly confusing compromise.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From matthew.brett at gmail.com  Mon May 15 04:43:07 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 15 May 2017 09:43:07 +0100
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
Message-ID: <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>

Hi,

On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Hi all,
>>
>> As some of you know, I've been working for... quite some time now to
>> try to secure funding for NumPy. So I'm excited that I can now
>> officially announce that BIDS [1] is planning to hire several folks
>> specifically to work on NumPy. These will full time positions at UC
>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>> and the general goal will be to work on some of the major priorities
>> we identified at the last dev meeting: more flexible dtypes, better
>> interoperation with other array libraries, paying down technical debt,
>> and so forth. Though I'm sure the details will change as we start to
>> dig into things and engage with the community.
>>
>> More details soon; universities move slowly, so nothing's going to
>> happen immediately. But this is definitely happening and I wanted to
>> get something out publicly before the conference season starts ? so if
>> you're someone who might be interested in coming to work with me and
>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>> line and we can chat! I'll be at PyCon next week if anyone happens to
>> be there. And feel free to spread the word.
>
>
> Excellent news. Do you have any sort of timeline in mind?
>
> It will be interesting to see what changes this leads to, both in the code
> and in the project sociology.

I was thinking the same thing - if this does come about, it would
likely have a big impact on practical governance.  It could also mean
that more important development conversations happen off-list.   It
seems to me it would be good to plan for this consciously.

Cheers,

Matthew

From numpy at mspacek.mm.st  Mon May 15 10:47:14 2017
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Mon, 15 May 2017 16:47:14 +0200
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <CAEQ_TvfXnL5K=K9iu=tmS3qFnJd6sfFUOEB+_sT=XpPEaqu7KA@mail.gmail.com>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
 <CAEQ_TvfXnL5K=K9iu=tmS3qFnJd6sfFUOEB+_sT=XpPEaqu7KA@mail.gmail.com>
Message-ID: <2f725443-114f-a287-c388-14a14fed2fd7@mspacek.mm.st>

On 2017-05-09 07:39 PM, Stephan Hoyer wrote:
> On Tue, May 9, 2017 at 9:46 AM, Martin Spacek <numpy at mspacek.mm.st
> <mailto:numpy at mspacek.mm.st>> wrote:
>
>     Looking at my own habits and uses, it seems to me that finding the indices
>     of matching values of one array in another is a more common use case than
>     finding insertion indices of one array into another sorted array. So, I
>     propose that np.search(), or something like it, could be even more useful
>     than np.searchsorted().
>
>
> The current version of this PR only returns the indices of the /first/ match
> (rather than all matches), which is an important detail. I would strongly
> consider including that detail in the name (e.g., by calling this "find_first"
> rather than "search"), because my naive expectation for a method called "search"
> is to find all matches.
>
> In any case, I agree that this functionality would be welcome. Getting the
> details right for a high performance solution is tricky, and there is strong
> evidence of interest given the 200+ upvotes on this StackOverflow question:
> http://stackoverflow.com/questions/432112/is-there-a-numpy-function-to-return-the-first-index-of-something-in-an-array
>

Good point about it only finding the first hit. However, `np.find_first` sounds 
a bit awkward to me. I've updated the PR to have a `which` kwarg that specifies 
whether it should return the first or last hit for values in `v` that have 
multiple hits in `a`.

https://github.com/numpy/numpy/pull/9055

Not sure if I already mentioned this somewhere, but we might also consider 
naming this `np.index` due to its analogous behaviour to the Python list method 
`.index()`.

Martin

From ben.v.root at gmail.com  Mon May 15 11:31:50 2017
From: ben.v.root at gmail.com (Benjamin Root)
Date: Mon, 15 May 2017 11:31:50 -0400
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
Message-ID: <CANNq6F=vVRfQ_ckU=zWDGT5r6cjrMDLpWqD7cdQCLM8LeccUmg@mail.gmail.com>

Great news, Nathaniel! It was a huge boost to matplotlib a couple of years
ago when we got an FTE, even if it was just for a few months. While that
effort didn't directly produce any new features, we were able to overhaul
some very old parts of the codebase. Probably why the effort was so
successful was that, 1) Michael had a clear idea of what needed work and
how to achieve it and 2) the components impacted were mostly not
user-facing.

With respect to off-list conversations, one thing that the matplotlib devs
have done is set up a weekly Google Hangouts session. A summary of that
meeting is then posted to the mailing list. A practice like that (posting
summaries of regular meetings) might be sufficient to feed off-line
discussions back to the greater community.


Cheers!
Ben Root


On Mon, May 15, 2017 at 4:43 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi,
>
> On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> Hi all,
> >>
> >> As some of you know, I've been working for... quite some time now to
> >> try to secure funding for NumPy. So I'm excited that I can now
> >> officially announce that BIDS [1] is planning to hire several folks
> >> specifically to work on NumPy. These will full time positions at UC
> >> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
> >> and the general goal will be to work on some of the major priorities
> >> we identified at the last dev meeting: more flexible dtypes, better
> >> interoperation with other array libraries, paying down technical debt,
> >> and so forth. Though I'm sure the details will change as we start to
> >> dig into things and engage with the community.
> >>
> >> More details soon; universities move slowly, so nothing's going to
> >> happen immediately. But this is definitely happening and I wanted to
> >> get something out publicly before the conference season starts ? so if
> >> you're someone who might be interested in coming to work with me and
> >> the other awesome folks at BIDS, then this is a heads-up: drop me a
> >> line and we can chat! I'll be at PyCon next week if anyone happens to
> >> be there. And feel free to spread the word.
> >
> >
> > Excellent news. Do you have any sort of timeline in mind?
> >
> > It will be interesting to see what changes this leads to, both in the
> code
> > and in the project sociology.
>
> I was thinking the same thing - if this does come about, it would
> likely have a big impact on practical governance.  It could also mean
> that more important development conversations happen off-list.   It
> seems to me it would be good to plan for this consciously.
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170515/2d144862/attachment.html>

From njs at pobox.com  Mon May 15 16:59:28 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 15 May 2017 13:59:28 -0700
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
Message-ID: <CAPJVwBk2edx0Lg4DMbq7WkEpexC-GzCQQM04ePgRoqFavk2fSA@mail.gmail.com>

On May 9, 2017 9:47 AM, "Martin Spacek" <numpy at mspacek.mm.st> wrote:

Hello,

I've opened up a pull request to add a function called np.search(), or
something like it, to complement np.searchsorted():

https://github.com/numpy/numpy/pull/9055

There's also this issue I opened before starting the PR:

https://github.com/numpy/numpy/issues/9052

Proposed API changes require discussion on the list, so here I am!

This proposed function (and perhaps array method?) does the same as
np.searchsorted(a, v), but doesn't require `a` to be sorted, and explicitly
checks if all the values in `v` are a subset of those in `a`. If not, it
currently raises an error, but that could be controlled via a kwarg.

As I mentioned in the PR, I often find myself abusing np.searchsorted() by
not explicitly checking these assumptions. The temptation to use it is
great, because it's such a fast and convenient function, and most of the
time that I use it, the assumptions are indeed valid. Explicitly checking
those assumptions each and every time before I use np.searchsorted() is
tedious, and easy to forget to do. I wouldn't be surprised if many others
abuse np.searchsorted() in the same way.


It's worth noting though that the "sorted" part is a critical part of what
makes it fast. If we're looking for k needles in an n-item haystack, then:

If the haystack is already sorted and we know it, using searchsorted does
it in k*log2(n) comparisons. (Could be reduced to average case O(k log log
n) for simple scalars by using interpolation search, but I don't think
searchsorted is that clever atm.)

If the haystack is not sorted, then sorting it and then using searchsorted
requires a total of O(n log n) + k*log2(n) comparisons.

And if the haystack is not sorted, then doing linear search to find the
first item like list.index does requires on average 0.5*k*n comparisons.

This analysis ignores memory effects, which are important -- linear memory
access is faster than random access, and searchsorted is all about making
memory access maximally unpredictable. But even so, I think
sorting-then-searching will be reasonably competitive pretty much from the
start, and for moderately large k and n values the difference between (n +
k)*log(n) and n*k is huge.

Another issue is that sorting requires an O(n)-sized temporary buffer
(assuming you can't mutate the haystack in place). But if your haystack is
a large enough fraction of memory that you can't afford is buffer, then
it's likely large enough that you can't afford linear searching either...


Looking at my own habits and uses, it seems to me that finding the indices
of matching values of one array in another is a more common use case than
finding insertion indices of one array into another sorted array. So, I
propose that np.search(), or something like it, could be even more useful
than np.searchsorted().


My main concern here would be creating a trap for the unwary, where people
use search() naively because it's so nice and convenient, and then
eventually get surprised by a nasty quadratic slowdown. There's a whole
blog about these traps :-) https://accidentallyquadratic.tumblr.com/

Otoh there are also huge number of numpy use cases where it doesn't matter
if some calculation is 1000x slower than it should be, as long as it works
and is discoverable...

So it sounds like one obvious thing would be to have a version of
searchsorted that checks for matches (maybe side="exact"? Though that's not
easy to find...). That's clearly useful, and orthogonal to the
linear/binary search issue, so we shouldn't make it a reason people are
tempted to choose the inferior algorithm.

...ok, how's this for a suggestion. Give np.search a strategy= kwarg, with
options "linear", "searchsorted", and "auto". Linear does the obvious
thing, searchsorted generates a sorter array using argsort (unless the user
provided one) and then calls searchsorted, and auto picks one of them
depending on whether a sorter array was provided and how large the arrays
are. The default is auto. In all cases it looks for exact matches.

I guess by default "not found" should be signaled with an exception, and
then there should be some option to have it return a sentinel value
instead? The problem is that since we're returning integers then there's no
sentinel value that's necessarily an invalid index (e.g. indexing will
happily accept -1).

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170515/8a5ef7d0/attachment.html>

From shoyer at gmail.com  Mon May 15 19:25:32 2017
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 15 May 2017 23:25:32 +0000
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <CAPJVwBk2edx0Lg4DMbq7WkEpexC-GzCQQM04ePgRoqFavk2fSA@mail.gmail.com>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
 <CAPJVwBk2edx0Lg4DMbq7WkEpexC-GzCQQM04ePgRoqFavk2fSA@mail.gmail.com>
Message-ID: <CAEQ_TveGdovd_yuQMKuxpQpkSzQAQyXy+mycOgqyHJ0yPCPD5A@mail.gmail.com>

I like the idea of a strategy keyword argument. strategy='auto' leaves the
door open for future improvements, e.g., if we ever add hash tables to
numpy.

For the algorithm, I think we actually want to sort the needles array as
well in most (all?) cases.

If haystack is also sorted, advancing thorough both arrays at once brings
down the cost of the actual search itself down to O(n+k). (Possibly this is
worth exposing as np.searchbothsorted or something similar?)

If we don't sort haystack, we can still use binary search on the needles to
bring the search cost down to O(n log k).
On Mon, May 15, 2017 at 5:00 PM Nathaniel Smith <njs at pobox.com> wrote:

> On May 9, 2017 9:47 AM, "Martin Spacek" <numpy at mspacek.mm.st> wrote:
>
> Hello,
>
> I've opened up a pull request to add a function called np.search(), or
> something like it, to complement np.searchsorted():
>
> https://github.com/numpy/numpy/pull/9055
>
> There's also this issue I opened before starting the PR:
>
> https://github.com/numpy/numpy/issues/9052
>
> Proposed API changes require discussion on the list, so here I am!
>
> This proposed function (and perhaps array method?) does the same as
> np.searchsorted(a, v), but doesn't require `a` to be sorted, and explicitly
> checks if all the values in `v` are a subset of those in `a`. If not, it
> currently raises an error, but that could be controlled via a kwarg.
>
> As I mentioned in the PR, I often find myself abusing np.searchsorted() by
> not explicitly checking these assumptions. The temptation to use it is
> great, because it's such a fast and convenient function, and most of the
> time that I use it, the assumptions are indeed valid. Explicitly checking
> those assumptions each and every time before I use np.searchsorted() is
> tedious, and easy to forget to do. I wouldn't be surprised if many others
> abuse np.searchsorted() in the same way.
>
>
> It's worth noting though that the "sorted" part is a critical part of what
> makes it fast. If we're looking for k needles in an n-item haystack, then:
>
> If the haystack is already sorted and we know it, using searchsorted does
> it in k*log2(n) comparisons. (Could be reduced to average case O(k log log
> n) for simple scalars by using interpolation search, but I don't think
> searchsorted is that clever atm.)
>
> If the haystack is not sorted, then sorting it and then using searchsorted
> requires a total of O(n log n) + k*log2(n) comparisons.
>
> And if the haystack is not sorted, then doing linear search to find the
> first item like list.index does requires on average 0.5*k*n comparisons.
>
> This analysis ignores memory effects, which are important -- linear memory
> access is faster than random access, and searchsorted is all about making
> memory access maximally unpredictable. But even so, I think
> sorting-then-searching will be reasonably competitive pretty much from the
> start, and for moderately large k and n values the difference between (n +
> k)*log(n) and n*k is huge.
>
> Another issue is that sorting requires an O(n)-sized temporary buffer
> (assuming you can't mutate the haystack in place). But if your haystack is
> a large enough fraction of memory that you can't afford is buffer, then
> it's likely large enough that you can't afford linear searching either...
>
>
> Looking at my own habits and uses, it seems to me that finding the indices
> of matching values of one array in another is a more common use case than
> finding insertion indices of one array into another sorted array. So, I
> propose that np.search(), or something like it, could be even more useful
> than np.searchsorted().
>
>
> My main concern here would be creating a trap for the unwary, where people
> use search() naively because it's so nice and convenient, and then
> eventually get surprised by a nasty quadratic slowdown. There's a whole
> blog about these traps :-) https://accidentallyquadratic.tumblr.com/
>
> Otoh there are also huge number of numpy use cases where it doesn't matter
> if some calculation is 1000x slower than it should be, as long as it works
> and is discoverable...
>
> So it sounds like one obvious thing would be to have a version of
> searchsorted that checks for matches (maybe side="exact"? Though that's not
> easy to find...). That's clearly useful, and orthogonal to the
> linear/binary search issue, so we shouldn't make it a reason people are
> tempted to choose the inferior algorithm.
>
> ...ok, how's this for a suggestion. Give np.search a strategy= kwarg, with
> options "linear", "searchsorted", and "auto". Linear does the obvious
> thing, searchsorted generates a sorter array using argsort (unless the user
> provided one) and then calls searchsorted, and auto picks one of them
> depending on whether a sorter array was provided and how large the arrays
> are. The default is auto. In all cases it looks for exact matches.
>
> I guess by default "not found" should be signaled with an exception, and
> then there should be some option to have it return a sentinel value
> instead? The problem is that since we're returning integers then there's no
> sentinel value that's necessarily an invalid index (e.g. indexing will
> happily accept -1).
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170515/4bbc73c5/attachment-0001.html>

From p.e.creasey.00 at googlemail.com  Mon May 15 22:02:35 2017
From: p.e.creasey.00 at googlemail.com (Peter Creasey)
Date: Mon, 15 May 2017 19:02:35 -0700
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
Message-ID: <CACUQHuL6dBbpkunUc=YiT=wbZj9yE8kT889S0N+nY2D9LgGNNw@mail.gmail.com>

> From: Stephan Hoyer <shoyer at gmail.com>
>
> I like the idea of a strategy keyword argument. strategy='auto' leaves the
> door open for future improvements, e.g., if we ever add hash tables to
> numpy.
>
> For the algorithm, I think we actually want to sort the needles array as
> well in most (all?) cases.
>
> If haystack is also sorted, advancing thorough both arrays at once brings
> down the cost of the actual search itself down to O(n+k). (Possibly this is
> worth exposing as np.searchbothsorted or something similar?)

I actually suggest reducing the scope of the problem to

>>> search_unique(haystack, needles)

which finds the index* in a haystack (of unique values) of each needle
and can make the assumption (e.g. as Stephen Hoyer points out) that if
len(needles)<<len(haystack), then you can just sort the needles and
iterate through the haystack in O(H log(N)) steps (otherwise you sort
the haystack and iterate through the needles instead).

Whilst you could make this function work in the more general
non-unique case (i.e. using stable sorts and finding the first
occurrence) I notice this often an accident and users would be better
served with an exception. I also prefer raising an exception if a
needle is unfound (perhaps with an unfound_index=None kwarg).

Finally I am a little wary of strategy=?hashtable? (though I love
hash-tables!) since choosing the one hash that is great for everyone?s
data does not seem an easy problem.

Peter

* ?index_? sounds more attractive than ?search_?, but we?re stuck with
the python connotation it would be the first occurrence and the numpy
connotation of that you are giving rather than receiving indices.

From jaime.frio at gmail.com  Tue May 16 05:31:52 2017
From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=)
Date: Tue, 16 May 2017 11:31:52 +0200
Subject: [Numpy-discussion] Proposal: np.search() to complement
 np.searchsorted()
In-Reply-To: <CAEQ_TveGdovd_yuQMKuxpQpkSzQAQyXy+mycOgqyHJ0yPCPD5A@mail.gmail.com>
References: <2d6b979e-24c1-a4d4-5645-4a61958fb36a@mspacek.mm.st>
 <CAPJVwBk2edx0Lg4DMbq7WkEpexC-GzCQQM04ePgRoqFavk2fSA@mail.gmail.com>
 <CAEQ_TveGdovd_yuQMKuxpQpkSzQAQyXy+mycOgqyHJ0yPCPD5A@mail.gmail.com>
Message-ID: <CAPOWHW=cJ46Sr+vCoXcRHBo8NVTob3zdtn5zoXFMY6ZVM51ytQ@mail.gmail.com>

On Tue, May 16, 2017 at 1:25 AM, Stephan Hoyer <shoyer at gmail.com> wrote:

> I like the idea of a strategy keyword argument. strategy='auto' leaves the
> door open for future improvements, e.g., if we ever add hash tables to
> numpy.
>
> For the algorithm, I think we actually want to sort the needles array as
> well in most (all?) cases.
>
> If haystack is also sorted, advancing thorough both arrays at once brings
> down the cost of the actual search itself down to O(n+k). (Possibly this is
> worth exposing as np.searchbothsorted or something similar?)
>
> If we don't sort haystack, we can still use binary search on the needles
> to bring the search cost down to O(n log k).
>

I have several old branches implementing most of this:

   - https://github.com/jaimefrio/numpy/tree/sorted-searchsorted This one
   adds a sorted_keys= kwarg to searchsorted that assumes the keys are also
   sorted and does the linear scan of both.
   - https://github.com/jaimefrio/numpy/tree/mergesorted Adds an internal
   mergesorted function that merges two sorted arrays into a new one, with
   several handlings of repeated items, originally intended to speed-up
   intersect1d and friends in arraysetops.
   - https://github.com/jaimefrio/numpy/tree/hash-unique Implements a hash
   table for numpy arrays, largely based on the implementation of Python
   dicts, and uses it for unique.

Some of the speed-ups are considerable, but I guess I found the extra
complexity didn't justify sending any of those as a PR.

But if we really want to make that search thing happen, then maybe it's
time to bring them back from the dead.

Jaime



> On Mon, May 15, 2017 at 5:00 PM Nathaniel Smith <njs at pobox.com> wrote:
>
>> On May 9, 2017 9:47 AM, "Martin Spacek" <numpy at mspacek.mm.st> wrote:
>>
>> Hello,
>>
>> I've opened up a pull request to add a function called np.search(), or
>> something like it, to complement np.searchsorted():
>>
>> https://github.com/numpy/numpy/pull/9055
>>
>> There's also this issue I opened before starting the PR:
>>
>> https://github.com/numpy/numpy/issues/9052
>>
>> Proposed API changes require discussion on the list, so here I am!
>>
>> This proposed function (and perhaps array method?) does the same as
>> np.searchsorted(a, v), but doesn't require `a` to be sorted, and explicitly
>> checks if all the values in `v` are a subset of those in `a`. If not, it
>> currently raises an error, but that could be controlled via a kwarg.
>>
>> As I mentioned in the PR, I often find myself abusing np.searchsorted()
>> by not explicitly checking these assumptions. The temptation to use it is
>> great, because it's such a fast and convenient function, and most of the
>> time that I use it, the assumptions are indeed valid. Explicitly checking
>> those assumptions each and every time before I use np.searchsorted() is
>> tedious, and easy to forget to do. I wouldn't be surprised if many others
>> abuse np.searchsorted() in the same way.
>>
>>
>> It's worth noting though that the "sorted" part is a critical part of
>> what makes it fast. If we're looking for k needles in an n-item haystack,
>> then:
>>
>> If the haystack is already sorted and we know it, using searchsorted does
>> it in k*log2(n) comparisons. (Could be reduced to average case O(k log log
>> n) for simple scalars by using interpolation search, but I don't think
>> searchsorted is that clever atm.)
>>
>> If the haystack is not sorted, then sorting it and then using
>> searchsorted requires a total of O(n log n) + k*log2(n) comparisons.
>>
>> And if the haystack is not sorted, then doing linear search to find the
>> first item like list.index does requires on average 0.5*k*n comparisons.
>>
>> This analysis ignores memory effects, which are important -- linear
>> memory access is faster than random access, and searchsorted is all about
>> making memory access maximally unpredictable. But even so, I think
>> sorting-then-searching will be reasonably competitive pretty much from the
>> start, and for moderately large k and n values the difference between (n +
>> k)*log(n) and n*k is huge.
>>
>> Another issue is that sorting requires an O(n)-sized temporary buffer
>> (assuming you can't mutate the haystack in place). But if your haystack is
>> a large enough fraction of memory that you can't afford is buffer, then
>> it's likely large enough that you can't afford linear searching either...
>>
>>
>> Looking at my own habits and uses, it seems to me that finding the
>> indices of matching values of one array in another is a more common use
>> case than finding insertion indices of one array into another sorted array.
>> So, I propose that np.search(), or something like it, could be even more
>> useful than np.searchsorted().
>>
>>
>> My main concern here would be creating a trap for the unwary, where
>> people use search() naively because it's so nice and convenient, and then
>> eventually get surprised by a nasty quadratic slowdown. There's a whole
>> blog about these traps :-) https://accidentallyquadratic.tumblr.com/
>>
>> Otoh there are also huge number of numpy use cases where it doesn't
>> matter if some calculation is 1000x slower than it should be, as long as it
>> works and is discoverable...
>>
>> So it sounds like one obvious thing would be to have a version of
>> searchsorted that checks for matches (maybe side="exact"? Though that's not
>> easy to find...). That's clearly useful, and orthogonal to the
>> linear/binary search issue, so we shouldn't make it a reason people are
>> tempted to choose the inferior algorithm.
>>
>> ...ok, how's this for a suggestion. Give np.search a strategy= kwarg,
>> with options "linear", "searchsorted", and "auto". Linear does the obvious
>> thing, searchsorted generates a sorter array using argsort (unless the user
>> provided one) and then calls searchsorted, and auto picks one of them
>> depending on whether a sorter array was provided and how large the arrays
>> are. The default is auto. In all cases it looks for exact matches.
>>
>> I guess by default "not found" should be signaled with an exception, and
>> then there should be some option to have it return a sentinel value
>> instead? The problem is that since we're returning integers then there's no
>> sentinel value that's necessarily an invalid index (e.g. indexing will
>> happily accept -1).
>>
>> -n
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>


-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170516/b8b7d154/attachment-0001.html>

From NissimD at elspec-ltd.com  Wed May 17 12:50:40 2017
From: NissimD at elspec-ltd.com (Nissim Derdiger)
Date: Wed, 17 May 2017 16:50:40 +0000
Subject: [Numpy-discussion] Compare NumPy arrays with threshold and return
 the differences
Message-ID: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>

Hi,

In my script, I need to compare big NumPy arrays (2D or 3D), and return a list of all cells with difference bigger than a defined threshold.
The compare itself can be done easily done with "allclose" function, like that:
Threshold = 0.1
if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
    Print('Same')

But this compare does not return which cells are not the same.

The easiest (yet naive) way to know which cells are not the same is to use a simple for loops code like this one:
def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
   if not Arr1.shape == Arr2.shape:
       return ['Arrays size not the same']
   Dimensions = Arr1.shape
   Diff = []
   for i in range(Dimensions [0]):
       for j in range(Dimensions [1]):
           if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold, equal_nan=True):
               Diff.append(',' + str(i) + ',' + str(j) + ',' + str(Arr1[i,j]) + ','
               + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
       return Diff

(and same for 3D arrays - with 1 more for loop)
This way is very slow when the Arrays are big and full of none-equal cells.

Is there a fast straight forward way in case they are not the same - to get a list of the uneven cells? maybe some built-in function in the NumPy itself?
Thanks!
Nissim


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/a8bfd324/attachment.html>

From pmhobson at gmail.com  Wed May 17 13:13:46 2017
From: pmhobson at gmail.com (Paul Hobson)
Date: Wed, 17 May 2017 10:13:46 -0700
Subject: [Numpy-discussion] Compare NumPy arrays with threshold and
 return the differences
In-Reply-To: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
References: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
Message-ID: <CADT3MEABot==+z_iL7qkzim0rDM+0hN4kP4W-veKeoqEW2pDrA@mail.gmail.com>

I would do something like:

diff_is_large = (array1 - array2) > threshold
index_at_large_diff = numpy.nonzero(diff_is_large)
array1[index_at_large_diff].tolist()


On Wed, May 17, 2017 at 9:50 AM, Nissim Derdiger <NissimD at elspec-ltd.com>
wrote:

> Hi,
> In my script, I need to compare big NumPy arrays (2D or 3D), and return a
> list of all cells with difference bigger than a defined threshold.
> The compare itself can be done easily done with "allclose" function, like
> that:
> Threshold = 0.1
> if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
>     Print('Same')
> But this compare does not return *which* cells are not the same.
>
> The easiest (yet naive) way to know which cells are not the same is to use
> a simple for loops code like this one:
> def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
>    if not Arr1.shape == Arr2.shape:
>        return ['Arrays size not the same']
>    Dimensions = Arr1.shape
>    Diff = []
>    for i in range(Dimensions [0]):
>        for j in range(Dimensions [1]):
>            if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold,
> equal_nan=True):
>                Diff.append(',' + str(i) + ',' + str(j) + ',' +
> str(Arr1[i,j]) + ','
>                + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
>        return Diff
> (and same for 3D arrays - with 1 more for loop)
> This way is very slow when the Arrays are big and full of none-equal cells.
>
> Is there a fast straight forward way in case they are not the same - to
> get a list of the uneven cells? maybe some built-in function in the NumPy
> itself?
> Thanks!
> Nissim
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/6183339c/attachment.html>

From robert.kern at gmail.com  Wed May 17 13:16:09 2017
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 May 2017 10:16:09 -0700
Subject: [Numpy-discussion] Compare NumPy arrays with threshold and
 return the differences
In-Reply-To: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
References: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
Message-ID: <CAF6FJisn3Oj18HOOP-DJGOi7rTwr-1U4npef+wCd=ENnMkMFmw@mail.gmail.com>

On Wed, May 17, 2017 at 9:50 AM, Nissim Derdiger <NissimD at elspec-ltd.com>
wrote:

> Hi,
> In my script, I need to compare big NumPy arrays (2D or 3D), and return a
> list of all cells with difference bigger than a defined threshold.
> The compare itself can be done easily done with "allclose" function, like
> that:
> Threshold = 0.1
> if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
>     Print('Same')
> But this compare does not return *which* cells are not the same.
>
> The easiest (yet naive) way to know which cells are not the same is to use
> a simple for loops code like this one:
> def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
>    if not Arr1.shape == Arr2.shape:
>        return ['Arrays size not the same']
>    Dimensions = Arr1.shape
>    Diff = []
>    for i in range(Dimensions [0]):
>        for j in range(Dimensions [1]):
>            if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold,
> equal_nan=True):
>                Diff.append(',' + str(i) + ',' + str(j) + ',' +
> str(Arr1[i,j]) + ','
>                + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
>        return Diff
> (and same for 3D arrays - with 1 more for loop)
> This way is very slow when the Arrays are big and full of none-equal cells.
>
> Is there a fast straight forward way in case they are not the same - to
> get a list of the uneven cells? maybe some built-in function in the NumPy
> itself?
>

Use `close_mask = np.isclose(Arr1, Arr2, Threshold, equal_nan=True)` to
return a boolean mask the same shape as the arrays which is True where the
elements are close and False where they are not. You can invert it to get a
boolean mask which is True where they are "far" with respect to the
threshold: `far_mask = ~close_mask`. Then you can use `i_idx, j_idx =
np.nonzero(far_mask)` to get arrays of the `i` and `j` indices where the
values are far. For example:

for i, j in zip(i_idx, j_idx):
    print("{0}, {1}, {2}, {3}, {4}, Fail".format(i, j, Arr1[i, j], Arr2[i,
j], Threshold))

-- 
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/3d57f695/attachment-0001.html>

From daniele at grinta.net  Wed May 17 13:35:34 2017
From: daniele at grinta.net (Daniele Nicolodi)
Date: Wed, 17 May 2017 11:35:34 -0600
Subject: [Numpy-discussion] Compare NumPy arrays with threshold and
 return the differences
In-Reply-To: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
References: <9EFE3345170EF24DB67C61C1B05EEEDB4073F384@EX10.Elspec.local>
Message-ID: <812493a5-45cc-2382-3296-4a596738bde3@grinta.net>

On 5/17/17 10:50 AM, Nissim Derdiger wrote:
> Hi,
> In my script, I need to compare big NumPy arrays (2D or 3D), and return
> a list of all cells with difference bigger than a defined threshold.
> The compare itself can be done easily done with "allclose" function,
> like that:
> Threshold = 0.1
> if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
>     Print('Same')
> But this compare does not return *_which_* cells are not the same.
>  
> The easiest (yet naive) way to know which cells are not the same is to
> use a simple for loops code like this one:
> def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
>    if not Arr1.shape == Arr2.shape:
>        return ['Arrays size not the same']

I think you have been exposed to too much Matlab :-) Why the [] around
the string? The pythonic way to react to unexpected conditions is to
raise an exception:

         raise ValueError('arrays size not the same')

>    Dimensions = Arr1.shape 
>    Diff = []
>    for i in range(Dimensions [0]):
>        for j in range(Dimensions [1]):
>            if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold,
> equal_nan=True):
>                Diff.append(',' + str(i) + ',' + str(j) + ',' +
> str(Arr1[i,j]) + ','
>                + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')

Here you are also doing something very unusual. Why do you concatenate
all those strings? It would be more efficient to return the indexes of
the array elements matching the conditions and print them out in a
second step.

>        return Diff
> (and same for 3D arrays - with 1 more for loop)
> This way is very slow when the Arrays are big and full of none-equal cells.
>  
> Is there a fast straight forward way in case they are not the same - to
> get a list of the uneven cells? maybe some built-in function in the
> NumPy itself?

a = np.random.randn(100, 100)
b = np.random.randn(100, 100)

ids = np.nonzero(np.abs(a - b) > threshold)

gives you a tuple of the indexes of the array elements pairs satisfying
your condition.  If you want to print them:

matcha = a[ids]
matchb = b[ids]

idt = np.vstack(ids).T

for i, ai, bi in zip(ids, matcha, matchb):
    c = ','.join(str(x) for x in i)
    print('{},{},{},{},Fail'.format(c, ai, bi,threshold))

works for 2D and 3D (on nD) arrays.

However, if you have many elements matching your condition this is going
to be slow and not very useful to look at. Maybe you can think about a
different way to visualize this result.

Cheers,
Dan


From NissimD at elspec-ltd.com  Thu May 18 08:07:07 2017
From: NissimD at elspec-ltd.com (Nissim Derdiger)
Date: Thu, 18 May 2017 12:07:07 +0000
Subject: [Numpy-discussion] Compare NumPy arrays with threshold
Message-ID: <9EFE3345170EF24DB67C61C1B05EEEDB40740429@EX10.Elspec.local>

Hi again,
Thanks for the responses to my question!
Roberts answer worked very well for me, except for 1 small issue:

This line:
close_mask = np.isclose(MatA, MatB, Threshold, equal_nan=True)
returns each difference twice - once j in compare to I and once for I in compare to j

for example:

for this input:
MatA = [[10,20,30],[40,50,60]]
MatB = [[10,30,30],[40,50,160]]

My old code will return:
0,1,20,30
1,3,60,160
You code returns:
0,1,20,30
1,3,60,160
0,1,30,20
1,3,160,60


I can simply cut "close_mask" to half so I'll have only 1 iteration, but that does not seems to be efficient..
any ideas?



Also, what should I change to support 3D arrays as well?


Thanks again,
Nissim.




-----Original Message-----
From: NumPy-Discussion [mailto:numpy-discussion-bounces+nissimd=elspec-ltd.com at python.org] On Behalf Of numpy-discussion-request at python.org
Sent: Wednesday, May 17, 2017 8:17 PM
To: numpy-discussion at python.org
Subject: NumPy-Discussion Digest, Vol 128, Issue 18

Send NumPy-Discussion mailing list submissions to
        numpy-discussion at python.org<mailto:numpy-discussion at python.org>

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/numpy-discussion
or, via email, send a message with subject or body 'help' to
        numpy-discussion-request at python.org<mailto:numpy-discussion-request at python.org>

You can reach the person managing the list at
        numpy-discussion-owner at python.org<mailto:numpy-discussion-owner at python.org>

When replying, please edit your Subject line so it is more specific than "Re: Contents of NumPy-Discussion digest..."


Today's Topics:

   1. Compare NumPy arrays with threshold and return the
      differences (Nissim Derdiger)
   2. Re: Compare NumPy arrays with threshold and return the
      differences (Paul Hobson)
   3. Re: Compare NumPy arrays with threshold and return the
      differences (Robert Kern)


----------------------------------------------------------------------

Message: 1
Date: Wed, 17 May 2017 16:50:40 +0000
From: Nissim Derdiger <NissimD at elspec-ltd.com<mailto:NissimD at elspec-ltd.com>>
To: "numpy-discussion at python.org<mailto:numpy-discussion at python.org>" <numpy-discussion at python.org<mailto:numpy-discussion at python.org>>
Subject: [Numpy-discussion] Compare NumPy arrays with threshold and
        return the differences
Message-ID:
        <9EFE3345170EF24DB67C61C1B05EEEDB4073F384 at EX10.Elspec.local<mailto:9EFE3345170EF24DB67C61C1B05EEEDB4073F384 at EX10.Elspec.local>>
Content-Type: text/plain; charset="us-ascii"

Hi,

In my script, I need to compare big NumPy arrays (2D or 3D), and return a list of all cells with difference bigger than a defined threshold.
The compare itself can be done easily done with "allclose" function, like that:
Threshold = 0.1
if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
    Print('Same')

But this compare does not return which cells are not the same.

The easiest (yet naive) way to know which cells are not the same is to use a simple for loops code like this one:
def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
   if not Arr1.shape == Arr2.shape:
       return ['Arrays size not the same']
   Dimensions = Arr1.shape
   Diff = []
   for i in range(Dimensions [0]):
       for j in range(Dimensions [1]):
           if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold, equal_nan=True):
               Diff.append(',' + str(i) + ',' + str(j) + ',' + str(Arr1[i,j]) + ','
               + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
       return Diff

(and same for 3D arrays - with 1 more for loop) This way is very slow when the Arrays are big and full of none-equal cells.

Is there a fast straight forward way in case they are not the same - to get a list of the uneven cells? maybe some built-in function in the NumPy itself?
Thanks!
Nissim


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/a8bfd324/attachment-0001.html>

------------------------------

Message: 2
Date: Wed, 17 May 2017 10:13:46 -0700
From: Paul Hobson <pmhobson at gmail.com<mailto:pmhobson at gmail.com>>
To: Discussion of Numerical Python <numpy-discussion at python.org<mailto:numpy-discussion at python.org>>
Subject: Re: [Numpy-discussion] Compare NumPy arrays with threshold
        and return the differences
Message-ID:
        <CADT3MEABot==+z_iL7qkzim0rDM+0hN4kP4W-veKeoqEW2pDrA at mail.gmail.com<mailto:CADT3MEABot==+z_iL7qkzim0rDM+0hN4kP4W-veKeoqEW2pDrA at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"

I would do something like:

diff_is_large = (array1 - array2) > threshold index_at_large_diff = numpy.nonzero(diff_is_large)
array1[index_at_large_diff].tolist()


On Wed, May 17, 2017 at 9:50 AM, Nissim Derdiger <NissimD at elspec-ltd.com<mailto:NissimD at elspec-ltd.com>>
wrote:

> Hi,
> In my script, I need to compare big NumPy arrays (2D or 3D), and
> return a list of all cells with difference bigger than a defined threshold.
> The compare itself can be done easily done with "allclose" function,
> like
> that:
> Threshold = 0.1
> if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
>     Print('Same')
> But this compare does not return *which* cells are not the same.
>
> The easiest (yet naive) way to know which cells are not the same is to
> use a simple for loops code like this one:
> def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
>    if not Arr1.shape == Arr2.shape:
>        return ['Arrays size not the same']
>    Dimensions = Arr1.shape
>    Diff = []
>    for i in range(Dimensions [0]):
>        for j in range(Dimensions [1]):
>            if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold,
> equal_nan=True):
>                Diff.append(',' + str(i) + ',' + str(j) + ',' +
> str(Arr1[i,j]) + ','
>                + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
>        return Diff
> (and same for 3D arrays - with 1 more for loop) This way is very slow
> when the Arrays are big and full of none-equal cells.
>
> Is there a fast straight forward way in case they are not the same -
> to get a list of the uneven cells? maybe some built-in function in the
> NumPy itself?
> Thanks!
> Nissim
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org<mailto:NumPy-Discussion at python.org>
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/6183339c/attachment-0001.html>

------------------------------

Message: 3
Date: Wed, 17 May 2017 10:16:09 -0700
From: Robert Kern <robert.kern at gmail.com<mailto:robert.kern at gmail.com>>
To: Discussion of Numerical Python <numpy-discussion at python.org<mailto:numpy-discussion at python.org>>
Subject: Re: [Numpy-discussion] Compare NumPy arrays with threshold
        and return the differences
Message-ID:
        <CAF6FJisn3Oj18HOOP-DJGOi7rTwr-1U4npef+wCd=ENnMkMFmw at mail.gmail.com<mailto:CAF6FJisn3Oj18HOOP-DJGOi7rTwr-1U4npef+wCd=ENnMkMFmw at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"

On Wed, May 17, 2017 at 9:50 AM, Nissim Derdiger <NissimD at elspec-ltd.com<mailto:NissimD at elspec-ltd.com>>
wrote:

> Hi,
> In my script, I need to compare big NumPy arrays (2D or 3D), and
> return a list of all cells with difference bigger than a defined threshold.
> The compare itself can be done easily done with "allclose" function,
> like
> that:
> Threshold = 0.1
> if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)):
>     Print('Same')
> But this compare does not return *which* cells are not the same.
>
> The easiest (yet naive) way to know which cells are not the same is to
> use a simple for loops code like this one:
> def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold):
>    if not Arr1.shape == Arr2.shape:
>        return ['Arrays size not the same']
>    Dimensions = Arr1.shape
>    Diff = []
>    for i in range(Dimensions [0]):
>        for j in range(Dimensions [1]):
>            if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold,
> equal_nan=True):
>                Diff.append(',' + str(i) + ',' + str(j) + ',' +
> str(Arr1[i,j]) + ','
>                + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n')
>        return Diff
> (and same for 3D arrays - with 1 more for loop) This way is very slow
> when the Arrays are big and full of none-equal cells.
>
> Is there a fast straight forward way in case they are not the same -
> to get a list of the uneven cells? maybe some built-in function in the
> NumPy itself?
>

Use `close_mask = np.isclose(Arr1, Arr2, Threshold, equal_nan=True)` to return a boolean mask the same shape as the arrays which is True where the elements are close and False where they are not. You can invert it to get a boolean mask which is True where they are "far" with respect to the
threshold: `far_mask = ~close_mask`. Then you can use `i_idx, j_idx = np.nonzero(far_mask)` to get arrays of the `i` and `j` indices where the values are far. For example:

for i, j in zip(i_idx, j_idx):
    print("{0}, {1}, {2}, {3}, {4}, Fail".format(i, j, Arr1[i, j], Arr2[i, j], Threshold))

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170517/3d57f695/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at python.org<mailto:NumPy-Discussion at python.org>
https://mail.python.org/mailman/listinfo/numpy-discussion


------------------------------

End of NumPy-Discussion Digest, Vol 128, Issue 18
*************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170518/d9808b4c/attachment-0001.html>

From marc.barbry at mailoo.org  Thu May 18 09:04:43 2017
From: marc.barbry at mailoo.org (marc)
Date: Thu, 18 May 2017 15:04:43 +0200
Subject: [Numpy-discussion] failed to add routine to the core module
Message-ID: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>

Dear Numpy developers,

I'm trying to add a routine to calculate the sum of a product of two 
arrays (a dot product). But that would not increase the memory (from 
what I saw np.dot is increasing the memory while it should not be 
necessary). The idea is to avoid the use of the temporary array in the 
calculation of the variance ( numpy/numpy/core/_methods.py line 112).

The routine that I want to implement look like this in python,

|arr = np.random.rand(100000)|
|mean = arr.mean()|
|var = 0.0|
|for ai in arr: var += (ai-mean)**2|

I would like to implement it in the umath module. As a first step, I 
tried to reproduce the divmod function of umath, but I did not manage to 
do it, you can find my fork here 
<https://github.com/mbarbry/numpy/tree/looking_around> (the branch with 
the changes is call looking_around). During compilation I get the 
following error,

|gcc: numpy/core/src/multiarray/number.c
In file included from numpy/core/src/multiarray/number.c:17:0: 
numpy/core/src/multiarray/number.c: In function ?array_sum_multiply?:
numpy/core/src/private/binop_override.h:176:39: error: ?PyNumberMethods 
{aka struct <anonymous>}? has no member named ?nb_sum_multiply? 
(void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
                             ^
numpy/core/src/private/binop_override.h:180:13: note: in expansion of 
macro ?BINOP_IS_FORWARD? if (BINOP_IS_FORWARD(m1, m2, slot_expr, 
test_func) && \
         ^
numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro 
?BINOP_GIVE_UP_IF_NEEDED? BINOP_GIVE_UP_IF_NEEDED(m1, m2, 
nb_sum_multiply, array_sum_multiply);|

Sorry if my question seems basic, but I'm new in Numpy development.
Any help?

Thank you in advance,

Marc Barbry

PS: I opened an issues as well on the github repository
https://github.com/numpy/numpy/issues/9130
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170518/991ee7b5/attachment.html>

From m.h.vankerkwijk at gmail.com  Thu May 18 10:32:21 2017
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Thu, 18 May 2017 10:32:21 -0400
Subject: [Numpy-discussion] failed to add routine to the core module
In-Reply-To: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>
References: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>
Message-ID: <CAJNV+9unRN00=ZnGjz7_G8jRu3tVbj-pEhAFcjCZvU_gc2f2sQ@mail.gmail.com>

Hi Marc,

ufuncs are quite tricky to compile. Part of your problem is that, I
think, you started a bit too high up: `divmod` is also a binary
operation, so that part you do not need at all. It may be an idea to
start instead with a PR that implemented a new ufunc, e.g.,
https://github.com/numpy/numpy/pull/8795, so that you can see what is
involved.

All the best,

Marten



On Thu, May 18, 2017 at 9:04 AM, marc <marc.barbry at mailoo.org> wrote:
> Dear Numpy developers,
>
> I'm trying to add a routine to calculate the sum of a product of two arrays
> (a dot product). But that would not increase the memory (from what I saw
> np.dot is increasing the memory while it should not be necessary). The idea
> is to avoid the use of the temporary array in the calculation of the
> variance ( numpy/numpy/core/_methods.py line 112).
>
> The routine that I want to implement look like this in python,
>
> arr = np.random.rand(100000)
> mean = arr.mean()
> var = 0.0
> for ai in arr: var += (ai-mean)**2
>
> I would like to implement it in the umath module. As a first step, I tried
> to reproduce the divmod function of umath, but I did not manage to do it,
> you can find my fork here (the branch with the changes is call
> looking_around). During compilation I get the following error,
>
> gcc: numpy/core/src/multiarray/number.c
> In file included from numpy/core/src/multiarray/number.c:17:0:
> numpy/core/src/multiarray/number.c: In function ?array_sum_multiply?:
> numpy/core/src/private/binop_override.h:176:39: error: ?PyNumberMethods {aka
> struct <anonymous>}? has no member named ?nb_sum_multiply?
> (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
>                             ^
> numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro
> ?BINOP_IS_FORWARD? if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
>         ^
> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
> ?BINOP_GIVE_UP_IF_NEEDED? BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply,
> array_sum_multiply);
>
> Sorry if my question seems basic, but I'm new in Numpy development.
> Any help?
>
> Thank you in advance,
>
> Marc Barbry
>
> PS: I opened an issues as well on the github repository
> https://github.com/numpy/numpy/issues/9130
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

From marc.barbry at mailoo.org  Thu May 18 11:19:31 2017
From: marc.barbry at mailoo.org (marc)
Date: Thu, 18 May 2017 17:19:31 +0200
Subject: [Numpy-discussion] failed to add routine to the core module
In-Reply-To: <CAJNV+9unRN00=ZnGjz7_G8jRu3tVbj-pEhAFcjCZvU_gc2f2sQ@mail.gmail.com>
References: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>
 <CAJNV+9unRN00=ZnGjz7_G8jRu3tVbj-pEhAFcjCZvU_gc2f2sQ@mail.gmail.com>
Message-ID: <9502fbcc-9dff-1ac9-f377-db9e2cee5845@mailoo.org>

Hello Marten,

Thank you for your help, effectively, the example that you propose is 
much easier to imitate, I can now continue further.

Thanks,
Marc

On 05/18/2017 04:32 PM, Marten van Kerkwijk wrote:
> Hi Marc,
>
> ufuncs are quite tricky to compile. Part of your problem is that, I
> think, you started a bit too high up: `divmod` is also a binary
> operation, so that part you do not need at all. It may be an idea to
> start instead with a PR that implemented a new ufunc, e.g.,
> https://github.com/numpy/numpy/pull/8795, so that you can see what is
> involved.
>
> All the best,
>
> Marten
>
>
>
> On Thu, May 18, 2017 at 9:04 AM, marc <marc.barbry at mailoo.org> wrote:
>> Dear Numpy developers,
>>
>> I'm trying to add a routine to calculate the sum of a product of two arrays
>> (a dot product). But that would not increase the memory (from what I saw
>> np.dot is increasing the memory while it should not be necessary). The idea
>> is to avoid the use of the temporary array in the calculation of the
>> variance ( numpy/numpy/core/_methods.py line 112).
>>
>> The routine that I want to implement look like this in python,
>>
>> arr = np.random.rand(100000)
>> mean = arr.mean()
>> var = 0.0
>> for ai in arr: var += (ai-mean)**2
>>
>> I would like to implement it in the umath module. As a first step, I tried
>> to reproduce the divmod function of umath, but I did not manage to do it,
>> you can find my fork here (the branch with the changes is call
>> looking_around). During compilation I get the following error,
>>
>> gcc: numpy/core/src/multiarray/number.c
>> In file included from numpy/core/src/multiarray/number.c:17:0:
>> numpy/core/src/multiarray/number.c: In function ?array_sum_multiply?:
>> numpy/core/src/private/binop_override.h:176:39: error: ?PyNumberMethods {aka
>> struct <anonymous>}? has no member named ?nb_sum_multiply?
>> (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
>>                              ^
>> numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro
>> ?BINOP_IS_FORWARD? if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
>>          ^
>> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
>> ?BINOP_GIVE_UP_IF_NEEDED? BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply,
>> array_sum_multiply);
>>
>> Sorry if my question seems basic, but I'm new in Numpy development.
>> Any help?
>>
>> Thank you in advance,
>>
>> Marc Barbry
>>
>> PS: I opened an issues as well on the github repository
>> https://github.com/numpy/numpy/issues/9130
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion


From sebastian at sipsolutions.net  Thu May 18 11:48:14 2017
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 18 May 2017 17:48:14 +0200
Subject: [Numpy-discussion] failed to add routine to the core module
In-Reply-To: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>
References: <8bb0ba6c-2d39-25fc-fae0-2472dadd3a5a@mailoo.org>
Message-ID: <1495122494.29226.1.camel@sipsolutions.net>

On Thu, 2017-05-18 at 15:04 +0200, marc wrote:
> Dear Numpy developers,
> I'm trying to add a routine to calculate the sum of a product of two
> arrays (a dot product). But that would not increase the memory (from
> what I saw np.dot is increasing the memory while it should not be
> necessary). The idea is to avoid the use of the temporary array in
> the calculation of the variance ( numpy/numpy/core/_methods.py line
> 112).

np.dot should only increase memory in some cases (such as non-
contiguous arrays) and be much faster in most cases (unless e.g. you do
not have a BLAS compatible type). You might also want to check out
np.einsum, which is pretty slick and can handle these kind of
operations as well. Note that `np.dot` calls into BLAS so that it is in
general much faster then np.einsum.

- Sebastian

> The routine that I want to implement look like this in python,
> arr = np.random.rand(100000)
> mean = arr.mean()
> var = 0.0
> for ai in arr: var += (ai-mean)**2
> I would like to implement it in the umath module. As a first step, I
> tried to reproduce the divmod function of umath, but I did not manage
> to do it, you can find my fork here (the branch with the changes is
> call looking_around). During compilation I get the following error,
> gcc: numpy/core/src/multiarray/number.c?
> In file included from numpy/core/src/multiarray/number.c:17:0:
> numpy/core/src/multiarray/number.c: In function
> ?array_sum_multiply?:?
> numpy/core/src/private/binop_override.h:176:39: error:
> ?PyNumberMethods {aka struct <anonymous>}? has no member named
> ?nb_sum_multiply? (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) !=
> (void*)(test_func))?
> ??? ??? ??? ??? ??? ??? ??? ^?
> numpy/core/src/private/binop_override.h:180:13: note: in expansion of
> macro ?BINOP_IS_FORWARD? if (BINOP_IS_FORWARD(m1, m2, slot_expr,
> test_func) && \
> ??? ??? ^?
> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
> ?BINOP_GIVE_UP_IF_NEEDED? BINOP_GIVE_UP_IF_NEEDED(m1, m2,
> nb_sum_multiply, array_sum_multiply);
> Sorry if my question seems basic, but I'm new in Numpy development.
> Any help?
> Thank you in advance,
> Marc Barbry
> 
> PS: I opened an issues as well on the github repository
> https://github.com/numpy/numpy/issues/9130?
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170518/3aef5332/attachment-0001.sig>

From charlesr.harris at gmail.com  Thu May 18 17:42:23 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 18 May 2017 15:42:23 -0600
Subject: [Numpy-discussion] NumPy 1.13.0rc2 released
Message-ID: <CAB6mnxLY6oLKQBAC3H0rib6vWR29oZsVQdPKqsqtzDwQAy_=9w@mail.gmail.com>

Hi All,

I'm pleased to announce the NumPy 1.13.0rc2 release. This release supports
Python 2.7 and 3.4-3.6 and contains many new features. It is one of the
most ambitious releases in the last several years. Some of the highlights
and new functions are

*Highlights*

   - Operations like ``a + b + c`` will reuse temporaries on some
   platforms, resulting in less memory use and faster execution.
   - Inplace operations check if inputs overlap outputs and create
   temporaries to avoid problems.
   - New __array_ufunc__ attribute provides improved ability for classes to
   override default ufunc behavior.
   - New np.block function for creating blocked arrays.


*New functions*

   - New ``np.positive`` ufunc.
   - New ``np.divmod`` ufunc provides more efficient divmod.
   - New ``np.isnat`` ufunc tests for NaT special values.
   - New ``np.heaviside`` ufunc computes the Heaviside function.
   - New ``np.isin`` function, improves on ``in1d``.
   - New ``np.block`` function for creating blocked arrays.
   - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API.

Wheels for the pre-release are available on PyPI. Source tarballs,
zipfiles, release notes, and the changelog are available on github
<https://github.com/numpy/numpy/releases/tag/v1.13.0rc2>.

A total of 102 people contributed to this release.  People with a "+" by
their
names contributed a patch for the first time.

   - A. Jesse Jiryu Davis +
   - Alessandro Pietro Bardelli +
   - Alex Rothberg +
   - Alexander Shadchin
   - Allan Haldane
   - Andres Guzman-Ballen +
   - Antoine Pitrou
   - Antony Lee
   - B R S Recht +
   - Baurzhan Muftakhidinov +
   - Ben Rowland
   - Benda Xu +
   - Blake Griffith
   - Bradley Wogsland +
   - Brandon Carter +
   - CJ Carey
   - Charles Harris
   - Christoph Gohlke
   - Danny Hermes +
   - David Hagen +
   - David Nicholson +
   - Duke Vijitbenjaronk +
   - Egor Klenin +
   - Elliott Forney +
   - Elliott M Forney +
   - Endolith
   - Eric Wieser
   - Erik M. Bray
   - Eugene +
   - Evan Limanto +
   - Felix Berkenkamp +
   - Fran?ois Bissey +
   - Frederic Bastien
   - Greg Young
   - Gregory R. Lee
   - Importance of Being Ernest +
   - Jaime Fernandez
   - Jakub Wilk +
   - James Cowgill +
   - James Sanders
   - Jean Utke +
   - Jesse Thoren +
   - Jim Crist +
   - Joerg Behrmann +
   - John Kirkham
   - Jonathan Helmus
   - Jonathan L Long
   - Jonathan Tammo Siebert +
   - Joseph Fox-Rabinovitz
   - Joshua Loyal +
   - Juan Nunez-Iglesias +
   - Julian Taylor
   - Kirill Balunov +
   - Likhith Chitneni +
   - Lo?c Est?ve
   - Mads Ohm Larsen
   - Marein K?nings +
   - Marten van Kerkwijk
   - Martin Thoma
   - Martino Sorbaro +
   - Marvin Schmidt +
   - Matthew Brett
   - Matthias Bussonnier +
   - Matthias C. M. Troffaes +
   - Matti Picus
   - Michael Seifert
   - Mikhail Pak +
   - Mortada Mehyar
   - Nathaniel J. Smith
   - Nick Papior
   - Oscar Villellas +
   - Pauli Virtanen
   - Pavel Potocek
   - Pete Peeradej Tanruangporn +
   - Philipp A +
   - Ralf Gommers
   - Robert Kern
   - Roland Kaufmann +
   - Ronan Lamy
   - Sami Salonen +
   - Sanchez Gonzalez Alvaro
   - Sebastian Berg
   - Shota Kawabuchi
   - Simon Gibbons
   - Stefan Otte
   - Stefan Peterson +
   - Stephan Hoyer
   - S?ren Fuglede J?rgensen +
   - Takuya Akiba
   - Tom Boyd +
   - Ville Skytt? +
   - Warren Weckesser
   - Wendell Smith
   - Yu Feng
   - Zixu Zhao +
   - Z? Vin?cius +
   - aha66 +
   - drabach +
   - drlvk +
   - jsh9 +
   - solarjoe +
   - zengi +

Cheers,

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170518/fb20dad7/attachment.html>

From robert.kern at gmail.com  Thu May 18 20:34:10 2017
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 May 2017 17:34:10 -0700
Subject: [Numpy-discussion] Compare NumPy arrays with threshold
In-Reply-To: <9EFE3345170EF24DB67C61C1B05EEEDB40740429@EX10.Elspec.local>
References: <9EFE3345170EF24DB67C61C1B05EEEDB40740429@EX10.Elspec.local>
Message-ID: <CAF6FJitObTv2hThYvZ6JsGyeddvfO-TyD5skc4uKtsCzG_s_2g@mail.gmail.com>

On Thu, May 18, 2017 at 5:07 AM, Nissim Derdiger <NissimD at elspec-ltd.com>
wrote:
>
> Hi again,
> Thanks for the responses to my question!
> Roberts answer worked very well for me, except for 1 small issue:
>
> This line:
> close_mask = np.isclose(MatA, MatB, Threshold, equal_nan=True)
> returns each difference twice ? once j in compare to I and once for I in
compare to j

No, it returns a boolean array the same size as MatA and MatB. It literally
can't contain "each difference twice". Maybe there is something else in
your code that is producing the doubling that you see, possibly in the
printing of the results.

I'm not seeing the behavior that you speak of. Please post your complete
code that produced the doubled output that you see.

import numpy as np

MatA = np.array([[10,20,30],[40,50,60]])
MatB = np.array([[10,30,30],[40,50,160]])
Threshold = 1.0

# Note the `atol=` here. I missed it before.
close_mask = np.isclose(MatA, MatB, atol=Threshold, equal_nan=True)
far_mask = ~close_mask
i_idx, j_idx = np.nonzero(far_mask)
for i, j in zip(i_idx, j_idx):
    print("{0}, {1}, {2}, {3}, {4}, Fail".format(i, j, MatA[i, j], MatB[i,
j], Threshold))


I get the following output:

$ python isclose.py
0, 1, 20, 30, 1.0, Fail
1, 2, 60, 160, 1.0, Fail

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170518/807fdd4c/attachment.html>

From cimrman3 at ntc.zcu.cz  Fri May 19 04:52:37 2017
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 19 May 2017 10:52:37 +0200
Subject: [Numpy-discussion] ANN: SfePy 2017.2
Message-ID: <dd27e819-98db-d1bc-c1fd-cb86bcc29133@ntc.zcu.cz>

I am pleased to announce release 2017.2 of SfePy.

Description
-----------

SfePy (simple finite elements in Python) is a software for solving systems of
coupled partial differential equations by the finite element method or by the
isogeometric analysis (limited support). It is distributed under the new BSD
license.

Home page: http://sfepy.org
Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/
Git (source) repository, issue tracker: https://github.com/sfepy/sfepy

Highlights of this release
--------------------------

- simplified and unified implementation of some homogenized coefficients
- support for saving custom structured data to HDF5 files
- new tutorial on preparing meshes using FreeCAD/OpenSCAD and Gmsh

For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1
(rather long and technical).

Cheers,
Robert Cimrman

---

Contributors to this release in alphabetical order:

Robert Cimrman
Jan Heczko
Lubos Kejzlar
Vladimir Lukes
Matyas Novak

From frank at horow.net  Fri May 19 12:13:12 2017
From: frank at horow.net (Frank Horowitz)
Date: Fri, 19 May 2017 12:13:12 -0400
Subject: [Numpy-discussion] accumulating version of numpy.put?
Message-ID: <50351B80-11B2-4014-A91F-0E22E0B384AE@horow.net>

Hi All,

In the numpy library, are there any equivalences of numpy.put to additively store the values rather than simply overwriting?

Alternatively, would there be significant speed penalties to code of the form

a.flat[ind] += v 

Thanks in advance for any help you might be able to provide!

Frank Horowitz


From wieser.eric+numpy at gmail.com  Fri May 19 12:24:01 2017
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Fri, 19 May 2017 16:24:01 +0000
Subject: [Numpy-discussion] accumulating version of numpy.put?
In-Reply-To: <50351B80-11B2-4014-A91F-0E22E0B384AE@horow.net>
References: <50351B80-11B2-4014-A91F-0E22E0B384AE@horow.net>
Message-ID: <CAL1kJvD-h+Nj-2W09vwaLqS2Ckj08WFRGaRNtgtEEHoNTwcWQA@mail.gmail.com>

Frank,

I think you?re looking for np.add.at(a, ind, v), documented here
<https://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.at.html>

Eric
?

On Fri, 19 May 2017 at 17:13 Frank Horowitz <frank at horow.net> wrote:

> Hi All,
>
> In the numpy library, are there any equivalences of numpy.put to
> additively store the values rather than simply overwriting?
>
> Alternatively, would there be significant speed penalties to code of the
> form
>
> a.flat[ind] += v
>
> Thanks in advance for any help you might be able to provide!
>
> Frank Horowitz
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170519/e1a39c37/attachment.html>

From njs at pobox.com  Fri May 19 16:48:16 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 19 May 2017 13:48:16 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAPJVwBkzjiTtUJAdLrR8YZ8JtMR4mNuqpL9+eYp=c84j+BGagQ@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <-383731338024949696@unknownmsgid>
 <CAPJVwBkzjiTtUJAdLrR8YZ8JtMR4mNuqpL9+eYp=c84j+BGagQ@mail.gmail.com>
Message-ID: <CAPJVwBkjDPBLZxqpcE_RW0zSzGh+2KT9BxVL=HbsLrvCRPMBHg@mail.gmail.com>

Okay, a few more details :-)

The initial funding here is a grant from the Gordon and Betty Moore
Foundation to UCB with me as PI, in the amount of $645,020. There's
also another thing in the pipeline that might supplement that, but
it'll be
~6 months yet before we know for sure. So keep your fingers crossed I guess.

Here's some text from the proposal (the references to "this year" may
give some sense of how long this has taken...):
  https://docs.google.com/document/d/1xHjQqc8V8zJk7WSCyw9NPCpMYZ2Urh0cmFm2vDd14ZE/edit

-n

On Sun, May 14, 2017 at 3:40 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Sun, May 14, 2017 at 2:11 PM, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
>> Awesome! This is really great news.
>>
>> Does this mean is several person-years of funding secured?
>
> Yes ? hoping to give more details there soon. (There's nothing dire
> and secretive, it's just the logistics of getting an announcement
> approved by funder communication people didn't work with getting
> something out by PyCon, so this is the slightly confusing compromise.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Fri May 19 17:06:52 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 19 May 2017 14:06:52 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
Message-ID: <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>

On Mon, May 15, 2017 at 1:43 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> Hi all,
>>>
>>> As some of you know, I've been working for... quite some time now to
>>> try to secure funding for NumPy. So I'm excited that I can now
>>> officially announce that BIDS [1] is planning to hire several folks
>>> specifically to work on NumPy. These will full time positions at UC
>>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>>> and the general goal will be to work on some of the major priorities
>>> we identified at the last dev meeting: more flexible dtypes, better
>>> interoperation with other array libraries, paying down technical debt,
>>> and so forth. Though I'm sure the details will change as we start to
>>> dig into things and engage with the community.
>>>
>>> More details soon; universities move slowly, so nothing's going to
>>> happen immediately. But this is definitely happening and I wanted to
>>> get something out publicly before the conference season starts ? so if
>>> you're someone who might be interested in coming to work with me and
>>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>>> line and we can chat! I'll be at PyCon next week if anyone happens to
>>> be there. And feel free to spread the word.
>>
>>
>> Excellent news. Do you have any sort of timeline in mind?
>>
>> It will be interesting to see what changes this leads to, both in the code
>> and in the project sociology.
>
> I was thinking the same thing - if this does come about, it would
> likely have a big impact on practical governance.  It could also mean
> that more important development conversations happen off-list.   It
> seems to me it would be good to plan for this consciously.

Yeah, definitely. Being able to handle changes like this was one of
the major motivations for all the governance discussions we started a
few years ago, and it's something we'll need to keep an eye on going
forward. To state it explicitly though: the idea is to fund folks so
that they can contribute to numpy within our existing process of open
community review, and preserving and growing that community is very
much one of the grant's goals; no-one should get special privileges
because of where their paycheck is coming from. If at some point you
(or anyone) feel like we're deviating from that please speak up.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From matthew.brett at gmail.com  Sat May 20 06:34:45 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 20 May 2017 11:34:45 +0100
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
Message-ID: <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>

Hi,

On Fri, May 19, 2017 at 10:06 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, May 15, 2017 at 1:43 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Sun, May 14, 2017 at 10:56 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>>
>>> On Sat, May 13, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> As some of you know, I've been working for... quite some time now to
>>>> try to secure funding for NumPy. So I'm excited that I can now
>>>> officially announce that BIDS [1] is planning to hire several folks
>>>> specifically to work on NumPy. These will full time positions at UC
>>>> Berkeley, postdoc or staff, with probably 2 year (initial) contracts,
>>>> and the general goal will be to work on some of the major priorities
>>>> we identified at the last dev meeting: more flexible dtypes, better
>>>> interoperation with other array libraries, paying down technical debt,
>>>> and so forth. Though I'm sure the details will change as we start to
>>>> dig into things and engage with the community.
>>>>
>>>> More details soon; universities move slowly, so nothing's going to
>>>> happen immediately. But this is definitely happening and I wanted to
>>>> get something out publicly before the conference season starts ? so if
>>>> you're someone who might be interested in coming to work with me and
>>>> the other awesome folks at BIDS, then this is a heads-up: drop me a
>>>> line and we can chat! I'll be at PyCon next week if anyone happens to
>>>> be there. And feel free to spread the word.
>>>
>>>
>>> Excellent news. Do you have any sort of timeline in mind?
>>>
>>> It will be interesting to see what changes this leads to, both in the code
>>> and in the project sociology.
>>
>> I was thinking the same thing - if this does come about, it would
>> likely have a big impact on practical governance.  It could also mean
>> that more important development conversations happen off-list.   It
>> seems to me it would be good to plan for this consciously.
>
> Yeah, definitely. Being able to handle changes like this was one of
> the major motivations for all the governance discussions we started a
> few years ago, and it's something we'll need to keep an eye on going
> forward. To state it explicitly though: the idea is to fund folks so
> that they can contribute to numpy within our existing process of open
> community review, and preserving and growing that community is very
> much one of the grant's goals; no-one should get special privileges
> because of where their paycheck is coming from. If at some point you
> (or anyone) feel like we're deviating from that please speak up.

I think Chuck's term 'sociology' is a good one; although it's good to
have the governance document, I don't think it covers all the issues
that your initiative brings up.  At the moment, the people doing the
most work on numpy (by commits at least) are Chuck, Eric and Julian.
As far as I know, we don't have any full-time developers.   When
you've got up and running it sounds like you'll have at least two
full-time developers.  I guess you'll be their manager, and that they
will be physically housed in the BIDS.   So, this will represent a big
shift in practical influence, from a more or less completely
distributed pool of developer hours, to something much closer to an
institution-owned project.   Of course, we can hope that this doesn't
have any negative consequences in terms of project dynamics, but it
seems to me that it would be sensible to think of the risks, and plan
for them, rather than waiting for the expected problems to arise, when
they may be too late to fix.

We had this kind of discussion with Travis a while ago, and he pushed
back about conflicts of interests for people in the BIDS.  At the
time, that wasn't a serious issue, because you were the only BIDS
member actively committing to numpy.  That changes when it's you, and
two full-time developers.

Cheers,

Matthew

From charlesr.harris at gmail.com  Sun May 21 10:19:42 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 21 May 2017 08:19:42 -0600
Subject: [Numpy-discussion] SciPy 2017 attendence
Message-ID: <CAB6mnx+jw8w86GpZ6BaZAknCttjvtxfpX1zdvEa9GZHjud+k_A@mail.gmail.com>

Hi All,

Just curious as to who may be attending SciPy 2017.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170521/d4999879/attachment.html>

From m.h.vankerkwijk at gmail.com  Sun May 21 14:27:52 2017
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Sun, 21 May 2017 14:27:52 -0400
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
Message-ID: <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>

Hi All,

First, it will be great to have more people developing! On avoiding
potential conflicts: I'm not overly worried, in part because of my
experience with astropy (for which NASA support developers at STScI
and CXC). One possible solution for trying to avoid them would be to
adapt the typical academic rule for conflicts of interests to PRs,
that non-trivial ones cannot be merged by someone who has a conflict
of interest with the author, i.e., it cannot be a superviser, someone
from the same institute, etc.

But probably the larger issue is how to communicate larger ideas about
where to go, ideas which might form "over coffee" at BIDS. We have
NEPs and this mailing list, but neither seem particularly effective,
in part because it simply is hard to take enough time to really think
through the more complicated ideas that would be the whole purpose of
a NEP. I don't know that this has a solution beyond trying to ensure
that one always involves people who should be interested in it.

All the best,

Marten

From matthew.brett at gmail.com  Mon May 22 07:12:49 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 22 May 2017 12:12:49 +0100
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
Message-ID: <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>

Hi,

On Sun, May 21, 2017 at 7:27 PM, Marten van Kerkwijk
<m.h.vankerkwijk at gmail.com> wrote:
> Hi All,
>
> First, it will be great to have more people developing! On avoiding
> potential conflicts: I'm not overly worried, in part because of my
> experience with astropy (for which NASA support developers at STScI
> and CXC). One possible solution for trying to avoid them would be to
> adapt the typical academic rule for conflicts of interests to PRs,
> that non-trivial ones cannot be merged by someone who has a conflict
> of interest with the author, i.e., it cannot be a superviser, someone
> from the same institute, etc.

There's surely a spectrum from "I'm sure it's going to be fine, let's
just see what happens" to detailed documentation of procedure and
management.  In this case I'm arguing for something fairly well to the
right of "I'm sure it's going to be fine" - it seems to me that we
could get 80% of the way to a reassuring blueprint with a relatively
small amount of effort.

Cheers,

Matthew

From m.h.vankerkwijk at gmail.com  Mon May 22 11:52:01 2017
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Mon, 22 May 2017 11:52:01 -0400
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
Message-ID: <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>

Hi Matthew,

> it seems to me that we could get 80% of the way to a reassuring blueprint with a relatively small amount of effort.

My sentence "adapt the typical academic rule for conflicts of
interests to PRs, that non-trivial ones cannot be merged by someone
who has a conflict of interest with the author, i.e., it cannot be a
superviser, someone from the same institute, etc." was meant as a
suggestion for part of this blueprint!

I'll readily admit, though, that since I'm not overly worried, I
haven't even looked at the policies that are in place, nor do I intend
to contribute much beyond this e-mail. Indeed, it may be that the old
adage "every initiative is punishable" holds here... would you, or one
of the others who feels it is important to have a blueprint, be
willing to provide a concrete text for discussion?

All the best,

Marten

From shoyer at gmail.com  Mon May 22 12:29:52 2017
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 22 May 2017 12:29:52 -0400
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
 <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
Message-ID: <CAEQ_TvfqYfXdhQFPOdMPX6v7QkjLqrKpPo_U+RAjTEiGsYM1Ww@mail.gmail.com>

On Mon, May 22, 2017 at 11:52 AM, Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> My sentence "adapt the typical academic rule for conflicts of
> interests to PRs, that non-trivial ones cannot be merged by someone
> who has a conflict of interest with the author, i.e., it cannot be a
> superviser, someone from the same institute, etc." was meant as a
> suggestion for part of this blueprint!
>

This sounds like a good rule of thumb to me. As a practical matter, asking
someone outside to approve changes is a good way to ensure that decisions
are not short-circuited by offline discussions. But remember that per our
governance procedures, we already require consensus for decision making. So
I don't think we need an actual change here.

I'll readily admit, though, that since I'm not overly worried, I
> haven't even looked at the policies that are in place, nor do I intend
> to contribute much beyond this e-mail.


I am also not worried about this, really not at all. NumPy already has
governance procedures and a steering committee for handling exactly these
sorts of concerns, should they arise (which I also consider extremely
unlikely in the case of BIDS and their non-profit funder).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170522/88bdf6fc/attachment.html>

From matthew.brett at gmail.com  Mon May 22 12:35:21 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 22 May 2017 17:35:21 +0100
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
 <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
Message-ID: <CAH6Pt5qsXt7arY8skBpQCh+kZAVDK2hm1sGW7HQGXSzSvAU=Bw@mail.gmail.com>

Hi,

On Mon, May 22, 2017 at 4:52 PM, Marten van Kerkwijk
<m.h.vankerkwijk at gmail.com> wrote:
> Hi Matthew,
>
>> it seems to me that we could get 80% of the way to a reassuring blueprint with a relatively small amount of effort.
>
> My sentence "adapt the typical academic rule for conflicts of
> interests to PRs, that non-trivial ones cannot be merged by someone
> who has a conflict of interest with the author, i.e., it cannot be a
> superviser, someone from the same institute, etc." was meant as a
> suggestion for part of this blueprint!
>
> I'll readily admit, though, that since I'm not overly worried, I
> haven't even looked at the policies that are in place, nor do I intend
> to contribute much beyond this e-mail. Indeed, it may be that the old
> adage "every initiative is punishable" holds here...

I understand what you're saying, but I think a more helpful way of
thinking of it, is putting the groundwork in place for the most
fruitful possible collaboration.

> would you, or one
> of the others who feels it is important to have a blueprint, be
> willing to provide a concrete text for discussion?

It doesn't make sense for me to do that, I'm #13 for commits in the
last year.  I'm just one of the many people who completely depend on
numpy.  Also, taking a little time to think these things through seems
like a small investment with the potential for significant gain, in
terms of improving communication and mitigating risk.

So, I think my suggestion is that it would be a good idea for
Nathaniel and the current steering committee to talk through how this
is going to play out, how the work will be selected and directed, and
so on.

Cheers,

Matthew

From sebastian at sipsolutions.net  Mon May 22 13:04:06 2017
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 22 May 2017 19:04:06 +0200
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAH6Pt5qsXt7arY8skBpQCh+kZAVDK2hm1sGW7HQGXSzSvAU=Bw@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
 <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
 <CAH6Pt5qsXt7arY8skBpQCh+kZAVDK2hm1sGW7HQGXSzSvAU=Bw@mail.gmail.com>
Message-ID: <1495472646.6514.1.camel@sipsolutions.net>

On Mon, 2017-05-22 at 17:35 +0100, Matthew Brett wrote:
> Hi,
> 
> On Mon, May 22, 2017 at 4:52 PM, Marten van Kerkwijk
> <m.h.vankerkwijk at gmail.com> wrote:
> > Hi Matthew,
> > 
> > > it seems to me that we could get 80% of the way to a reassuring
> > > blueprint with a relatively small amount of effort.
> > 
> > My sentence "adapt the typical academic rule for conflicts of
> > interests to PRs, that non-trivial ones cannot be merged by someone
> > who has a conflict of interest with the author, i.e., it cannot be
> > a
> > superviser, someone from the same institute, etc." was meant as a
> > suggestion for part of this blueprint!
> > 
> > I'll readily admit, though, that since I'm not overly worried, I
> > haven't even looked at the policies that are in place, nor do I
> > intend
> > to contribute much beyond this e-mail. Indeed, it may be that the
> > old
> > adage "every initiative is punishable" holds here...
> 
> I understand what you're saying, but I think a more helpful way of
> thinking of it, is putting the groundwork in place for the most
> fruitful possible collaboration.
> 
> > would you, or one
> > of the others who feels it is important to have a blueprint, be
> > willing to provide a concrete text for discussion?
> 
> It doesn't make sense for me to do that, I'm #13 for commits in the
> last year.??I'm just one of the many people who completely depend on
> numpy.??Also, taking a little time to think these things through
> seems
> like a small investment with the potential for significant gain, in
> terms of improving communication and mitigating risk.
> 
> So, I think my suggestion is that it would be a good idea for
> Nathaniel and the current steering committee to talk through how this
> is going to play out, how the work will be selected and directed, and
> so on.
> 

Frankly, I would suggest to wait for now and ask whoever is going to
get the job to work out how they think it should be handled. And then
we complain if we expect more/better ;).
For now I only would say that I will expect more community type of work
then we now often manage to do. And things such as meticulously
sticking to writing NEPs.
So the only thing I can see that might be good is putting "community
work" or something like it specifically as part of the job description,
and thats up to Nathaniel probably.

Some things like not merging large changes by two people sittings in
the same office should be obvious (and even if it happens, we can
revert). But its nothing much new there I think.

- Sebastian


> Cheers,
> 
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170522/c356f9bb/attachment.sig>

From njs at pobox.com  Wed May 24 23:11:12 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 24 May 2017 20:11:12 -0700
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <1495472646.6514.1.camel@sipsolutions.net>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
 <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
 <CAH6Pt5qsXt7arY8skBpQCh+kZAVDK2hm1sGW7HQGXSzSvAU=Bw@mail.gmail.com>
 <1495472646.6514.1.camel@sipsolutions.net>
Message-ID: <CAPJVwBkoPHcO7thGb9ave6fRNcSmh3Br2-omHNUNu0Ke+LtkSQ@mail.gmail.com>

On Mon, May 22, 2017 at 10:04 AM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Mon, 2017-05-22 at 17:35 +0100, Matthew Brett wrote:
>> Hi,
>>
>> On Mon, May 22, 2017 at 4:52 PM, Marten van Kerkwijk
>> <m.h.vankerkwijk at gmail.com> wrote:
>> > Hi Matthew,
>> >
>> > > it seems to me that we could get 80% of the way to a reassuring
>> > > blueprint with a relatively small amount of effort.
>> >
>> > My sentence "adapt the typical academic rule for conflicts of
>> > interests to PRs, that non-trivial ones cannot be merged by someone
>> > who has a conflict of interest with the author, i.e., it cannot be
>> > a
>> > superviser, someone from the same institute, etc." was meant as a
>> > suggestion for part of this blueprint!
>> >
>> > I'll readily admit, though, that since I'm not overly worried, I
>> > haven't even looked at the policies that are in place, nor do I
>> > intend
>> > to contribute much beyond this e-mail. Indeed, it may be that the
>> > old
>> > adage "every initiative is punishable" holds here...
>>
>> I understand what you're saying, but I think a more helpful way of
>> thinking of it, is putting the groundwork in place for the most
>> fruitful possible collaboration.
>>
>> > would you, or one
>> > of the others who feels it is important to have a blueprint, be
>> > willing to provide a concrete text for discussion?
>>
>> It doesn't make sense for me to do that, I'm #13 for commits in the
>> last year.  I'm just one of the many people who completely depend on
>> numpy.  Also, taking a little time to think these things through
>> seems
>> like a small investment with the potential for significant gain, in
>> terms of improving communication and mitigating risk.
>>
>> So, I think my suggestion is that it would be a good idea for
>> Nathaniel and the current steering committee to talk through how this
>> is going to play out, how the work will be selected and directed, and
>> so on.
>>
>
> Frankly, I would suggest to wait for now and ask whoever is going to
> get the job to work out how they think it should be handled. And then
> we complain if we expect more/better ;).

This is roughly where I am as well. Certainly this is an important
issue, but we've already done a lot of groundwork in the abstract ?
the dev meeting, formalizing the governance document, and so forth
(and recall that "let's get to a point where we can apply for grants"
was explicitly one of the goals in those discussions). I think at this
point the most productive thing to do is wait until we have a more
concrete picture of who/what/when will be happening, so we can make a
concrete plan.

> For now I only would say that I will expect more community type of work
> then we now often manage to do. And things such as meticulously
> sticking to writing NEPs.
> So the only thing I can see that might be good is putting "community
> work" or something like it specifically as part of the job description,

Definitely.

> and thats up to Nathaniel probably.
>
> Some things like not merging large changes by two people sittings in
> the same office should be obvious (and even if it happens, we can
> revert). But its nothing much new there I think.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From Permafacture at gmail.com  Wed May 24 23:54:16 2017
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Wed, 24 May 2017 22:54:16 -0500
Subject: [Numpy-discussion] UC Berkeley hiring developers to work on
 NumPy
In-Reply-To: <CAPJVwBkoPHcO7thGb9ave6fRNcSmh3Br2-omHNUNu0Ke+LtkSQ@mail.gmail.com>
References: <CAPJVwBnZ30XPMMaA_7dv10u+rJsXQ5ybKU8pgdomqqvXtqSd9g@mail.gmail.com>
 <CAB6mnxJb4MW=Tk82ga6SwuLWqL6fesSD-gyNqaEZvRNwKtovVg@mail.gmail.com>
 <CAH6Pt5pVGCWsktrdSW-3pCaMjoMrHhjzbXzqu-BJzSpDa4pXMg@mail.gmail.com>
 <CAPJVwBmUOJG12uE8nAgopHB305RqU0kaQJZwu+pXTiVSJyB3pQ@mail.gmail.com>
 <CAH6Pt5qfyNg9_AjXM2DT=R+oPxfMCNV+2xpVnbrWW6DKgAkKcQ@mail.gmail.com>
 <CAJNV+9sv5TeL_5tk4DZGmyWbt1A0edy0_fBWCBjwBgN0Q+wEqg@mail.gmail.com>
 <CAH6Pt5qQy65D5wMnGq9aJN0q0tNU2o+5Rhz05-zoZJaJ3FRfEQ@mail.gmail.com>
 <CAJNV+9uNdCsrNVKwDj4LckEYBG5-zGNsnYoJ0JW6O57=kOWExw@mail.gmail.com>
 <CAH6Pt5qsXt7arY8skBpQCh+kZAVDK2hm1sGW7HQGXSzSvAU=Bw@mail.gmail.com>
 <1495472646.6514.1.camel@sipsolutions.net>
 <CAPJVwBkoPHcO7thGb9ave6fRNcSmh3Br2-omHNUNu0Ke+LtkSQ@mail.gmail.com>
Message-ID: <CAHedU1Ss-T_1sdeAwZGt8cae-FtyYXZYgRJFwLZ4SQWa_0apwQ@mail.gmail.com>

+1 for a conflict of interest policy.  A member of another group reviewing
and pulling a change is a reasonable expectation. Explicit is better than
implicit.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170524/4e088ab7/attachment.html>

From charlesr.harris at gmail.com  Sun May 28 16:53:33 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 28 May 2017 14:53:33 -0600
Subject: [Numpy-discussion] Future of ufuncs
Message-ID: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>

Hi All,

This post is to open a discussion of the future of ufuncs. There are two
contradictory ideas that have floated about regarding ufuncs evolution. One
is to generalize ufuncs to operate on buffers, essentially separating them
from their current entanglement with ndarrays. The other is to accept that
they are fundamentally part of the ndarray universe and move them into the
multiarray module, thus avoiding the odd overloading of functions in the
multiarray module. The first has been a long time proposal that I once
thought sounded good, but I've come to prefer the second. That change of
mind was driven by the resulting code simplification and the removal of a
dependence on a Python feature, buffers, that we cannot easily modify to
adapt to changing needs and new dtypes. Because I'd like to move the
ufuncs, if we decide to move them, sometime after NumPy 1.14 is released,
now seems a good time to decide the issue.

Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170528/099ec248/attachment.html>

From sebastian at sipsolutions.net  Mon May 29 05:40:39 2017
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 29 May 2017 11:40:39 +0200
Subject: [Numpy-discussion] Future of ufuncs
In-Reply-To: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>
References: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>
Message-ID: <1496050839.11285.1.camel@sipsolutions.net>

On Sun, 2017-05-28 at 14:53 -0600, Charles R Harris wrote:
> Hi All,
> This post is to open a discussion of the future of ufuncs. There are
> two contradictory ideas that have floated about regarding ufuncs
> evolution. One is to generalize ufuncs to operate on buffers,
> essentially separating them from their current entanglement with
> ndarrays. The other is to accept that they are fundamentally part of
> the ndarray universe and move them into the multiarray module, thus
> avoiding the odd overloading of functions in the multiarray module.
> The first has been a long time proposal that I once thought sounded
> good, but I've come to prefer the second. That change of mind was
> driven by the resulting code simplification and the removal of a
> dependence on a Python feature, buffers, that we cannot easily modify
> to adapt to changing needs and new dtypes. Because I'd like to move
> the ufuncs, if we decide to move them, sometime after NumPy 1.14 is
> released, now seems a good time to decide the issue.
> Thoughts?


I did not think about it much. But I agree that the dtypes are probably
the biggest issue, also I am not sure anymore if there is much of a
gain on having ufuncs work on buffers in any case?

The dtype thing goes a bit back to ideas like the datashape things and
trying to make the dtypes somewhat separate from numpy? Though I doubt
I would want to make that an explicit goal.

I wonder how much of the C-loops and type resolving we could/should
expose? What I mean is that ufuncs are:

 * type resolving (somewhat ufunc specific)
 * outer loops (normal, reduce, etc.) using nditer (buffering)
 * inner 1d loops

It is a bit more complicating, but just wondering if it might make
sense to try and expose the individual ufunc things (type resolving and
1d loop) but not all the outer loop nditer setup which is ndarray
specific in any case (honestly, I am not sure it is entirely possible
it is already exposed).

- Sebastian


> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170529/779e7347/attachment.sig>

From m.h.vankerkwijk at gmail.com  Mon May 29 14:32:17 2017
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Mon, 29 May 2017 14:32:17 -0400
Subject: [Numpy-discussion] Future of ufuncs
In-Reply-To: <1496050839.11285.1.camel@sipsolutions.net>
References: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>
 <1496050839.11285.1.camel@sipsolutions.net>
Message-ID: <CAJNV+9sVKdB=_0UTA3w2PRLp+bQWQyu7TNRN0eMHk8QPmKwm2Q@mail.gmail.com>

Hi Chuck,

Like Sebastian, I wonder a little about what level you are talking
about. Presumably, it is the actual implementation of the ufunc? I.e.,
this is not about the upper logic that decides which `__array_ufunc__`
to call, etc.

If so, I agree with you that it would seem to make most sense to move
the implementation to `multiarray`; the current structure certainly is
a major hurdle to understanding how things work!

Indeed, I guess in terms of my earlier suggestion to make much of a
ufunc happen in `ndarray.__array_ufunc__`, one could seem the type
resolution and iteration happening there. If one were to expose the
inner loops, anyone working with buffers could then use the ufuncs by
defining their own __array_ufunc__.

All the best,

Marten

From charlesr.harris at gmail.com  Mon May 29 16:51:02 2017
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 29 May 2017 14:51:02 -0600
Subject: [Numpy-discussion] Future of ufuncs
In-Reply-To: <CAJNV+9sVKdB=_0UTA3w2PRLp+bQWQyu7TNRN0eMHk8QPmKwm2Q@mail.gmail.com>
References: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>
 <1496050839.11285.1.camel@sipsolutions.net>
 <CAJNV+9sVKdB=_0UTA3w2PRLp+bQWQyu7TNRN0eMHk8QPmKwm2Q@mail.gmail.com>
Message-ID: <CAB6mnx+_0r+j2TTkZ=8-KSPXLRwV+2t2EL9ZJCyMPaPS=6Buow@mail.gmail.com>

On Mon, May 29, 2017 at 12:32 PM, Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Hi Chuck,
>
> Like Sebastian, I wonder a little about what level you are talking
> about. Presumably, it is the actual implementation of the ufunc? I.e.,
> this is not about the upper logic that decides which `__array_ufunc__`
> to call, etc.
>
> If so, I agree with you that it would seem to make most sense to move
> the implementation to `multiarray`; the current structure certainly is
> a major hurdle to understanding how things work!
>
> Indeed, I guess in terms of my earlier suggestion to make much of a
> ufunc happen in `ndarray.__array_ufunc__`, one could seem the type
> resolution and iteration happening there. If one were to expose the
> inner loops, anyone working with buffers could then use the ufuncs by
> defining their own __array_ufunc__.
>

The idea of separating ufuncs from ndarray was put forward many years ago,
maybe five or six. What I seek here is a record that we have given up on
that ambition, so do not need to take it into consideration in the future.
In particular, we can feel free to couple ufuncs even more tightly with
ndarray.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20170529/6965406e/attachment.html>

From njs at pobox.com  Mon May 29 18:34:47 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 May 2017 15:34:47 -0700
Subject: [Numpy-discussion] Future of ufuncs
In-Reply-To: <CAB6mnx+_0r+j2TTkZ=8-KSPXLRwV+2t2EL9ZJCyMPaPS=6Buow@mail.gmail.com>
References: <CAB6mnxLNL40bRDxwvXT3z=_FK6bHg8Ask4qj4H7fUa66bQov_g@mail.gmail.com>
 <1496050839.11285.1.camel@sipsolutions.net>
 <CAJNV+9sVKdB=_0UTA3w2PRLp+bQWQyu7TNRN0eMHk8QPmKwm2Q@mail.gmail.com>
 <CAB6mnx+_0r+j2TTkZ=8-KSPXLRwV+2t2EL9ZJCyMPaPS=6Buow@mail.gmail.com>
Message-ID: <CAPJVwB=K5_uSQzsSWjgMEC5KaPbuET+Z15C53c3KRV1qz_zoTw@mail.gmail.com>

On Mon, May 29, 2017 at 1:51 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, May 29, 2017 at 12:32 PM, Marten van Kerkwijk
> <m.h.vankerkwijk at gmail.com> wrote:
>>
>> Hi Chuck,
>>
>> Like Sebastian, I wonder a little about what level you are talking
>> about. Presumably, it is the actual implementation of the ufunc? I.e.,
>> this is not about the upper logic that decides which `__array_ufunc__`
>> to call, etc.
>>
>> If so, I agree with you that it would seem to make most sense to move
>> the implementation to `multiarray`; the current structure certainly is
>> a major hurdle to understanding how things work!
>>
>> Indeed, I guess in terms of my earlier suggestion to make much of a
>> ufunc happen in `ndarray.__array_ufunc__`, one could seem the type
>> resolution and iteration happening there. If one were to expose the
>> inner loops, anyone working with buffers could then use the ufuncs by
>> defining their own __array_ufunc__.
>
>
> The idea of separating ufuncs from ndarray was put forward many years ago,
> maybe five or six. What I seek here is a record that we have given up on
> that ambition, so do not need to take it into consideration in the future.
> In particular, we can feel free to couple ufuncs even more tightly with
> ndarray.

I think we do want to separate ufuncs from ndarray semantically: it
should be possible to use ufuncs on sparse arrays, dask arrays, etc.
etc.

But I don't think that altering ufuncs to work directly on
buffer/memoryview objects, or shipping them as a separate package from
the rest of numpy, is a useful step towards this goal.

Right now, handling buffers/memoryviews is easy: one can trivially
convert between them and ndarray without making any copies. I don't
know of any interesting problems that are blocked because ufuncs work
on ndarrays instead of buffer/memoryview objects. The interesting
problems are where there's a fundamentally different storage strategy
involved, like sparse/dask/... arrays.

And similarly, I don't see what problems are solved by splitting them
out for building or distribution.

OTOH, trying to accomplish either of these things definitely has a
cost in terms of churn, complexity, double the workload for
release-management, etc. Even the current split between the multiarray
and umath modules causes problems all the time. It's mostly boring
problems like having little utility functions that are needed in both
places but awkward to share, or problems caused by the complicated
machinery needed to let them interact properly (set_numeric_ops and
all that) ? this doesn't seem like stuff that's adding any value.

Plus, there's a major problem that buffers/memoryviews don't have any
way to represent all the dtypes we currently support (e.g. datetime64)
and don't have any way to add new ones, and the only way to fix this
would be to write a PEP, shepherding patches through python-dev,
waiting for the next python major release and then dropping support
for all older Python releases. None of this is going to happen soon;
probably we should plan on the assumption that it will never happen.
So I don't see how this could work at all.

So my vote is for merging the multiarray and umath code bases
together, and then taking advantage of the resulting flexibility to
refactor the internals to provide cleanly separated interfaces at the
API level.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org