From charlesr.harris at gmail.com Wed Jan 1 13:05:01 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Jan 2020 11:05:01 -0700 Subject: [Numpy-discussion] NumPy 1.17.5 Released Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.17.5 has been released. This release fixes bugs reported against the 1.17.4 release. The supported Python versions are 3.5-3.7. This is the last planned release that supports Python 3.5. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . Downstream developers building this release should use Cython >= 0.29.14 and, if using OpenBLAS, OpenBLAS >= v0.3.7. It is recommended that developers interested in the new random bit generators upgrade to the NumPy 1.18.x series, as it has updated documentation and many small improvements. *Highlights* - The ``np.testing.utils`` functions have been updated from 1.19.0-dev0. This improves the function documentation and error messages as well extending the ``assert_array_compare`` function to additional types. *Contributors* A total of 6 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Eric Wieser - Ilhan Polat - Matti Picus - Michael Hudson-Doyle - Ralf Gommers *Pull requests merged* A total of 8 pull requests were merged for this release. - `#14593 `__: MAINT: backport Cython API cleanup to 1.17.x, remove docs - `#14937 `__: BUG: fix integer size confusion in handling array's ndmin argument - `#14939 `__: BUILD: remove SSE2 flag from numpy.random builds - `#14993 `__: MAINT: Added Python3.8 branch to dll lib discovery - `#15038 `__: BUG: Fix refcounting in ufunc object loops - `#15067 `__: BUG: Exceptions tracebacks are dropped - `#15175 `__: ENH: Backport improvements to testing functions. - `#15213 `__: REL: Prepare for the NumPy 1.17.5 release. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jan 2 18:51:46 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Jan 2020 16:51:46 -0700 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. Message-ID: Hi All, Just want to propose cleaning up the Python 2.7 compatibility code for NumPy 1.19. Any objections? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Jan 2 19:00:38 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 02 Jan 2020 18:00:38 -0600 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. In-Reply-To: References: Message-ID: <84fb66063b8a4992a50d1d9e4b3ab804311a2c95.camel@sipsolutions.net> On Thu, 2020-01-02 at 16:51 -0700, Charles R Harris wrote: > Hi All, > > Just want to propose cleaning up the Python 2.7 compatibility code > for NumPy 1.19. Any objections? > Not from me, sounds like a good plan, lets get it over with :). - Sebastian > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From millman at berkeley.edu Thu Jan 2 19:19:00 2020 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 2 Jan 2020 16:19:00 -0800 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. In-Reply-To: References: Message-ID: Hi Chuck, I just finished doing that for NetworkX and would be happy to start a PR for NumPy tomorrow (assuming there are no objections). Best regards, Jarrod On Thu, Jan 2, 2020 at 3:53 PM Charles R Harris wrote: > > Hi All, > > Just want to propose cleaning up the Python 2.7 compatibility code for NumPy 1.19. Any objections? > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Thu Jan 2 19:52:26 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Jan 2020 17:52:26 -0700 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. In-Reply-To: References: Message-ID: On Thu, Jan 2, 2020 at 5:20 PM Jarrod Millman wrote: > Hi Chuck, > > I just finished doing that for NetworkX and would be happy to start a > PR for NumPy tomorrow (assuming there are no objections). > > Best regards, > Jarrod > Hi Jarrod, There are four current PRs making a start on the task, they need review and coordination. If you could help with that it would be a good start. They are tagged with `Py3K`. There are also several related issues with that tag. I think we can also remove Python <= 3.4 stuff, removing 3.5 can probably wait until NumPy 1.20. See https://github.com/numpy/numpy/labels/27%20-%20Py3K for the list of tagged PRs/issues. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Jan 2 20:10:09 2020 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 2 Jan 2020 17:10:09 -0800 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. In-Reply-To: References: Message-ID: Sounds good! Jarrod On Thu, Jan 2, 2020 at 4:53 PM Charles R Harris wrote: > > > > On Thu, Jan 2, 2020 at 5:20 PM Jarrod Millman wrote: >> >> Hi Chuck, >> >> I just finished doing that for NetworkX and would be happy to start a >> PR for NumPy tomorrow (assuming there are no objections). >> >> Best regards, >> Jarrod > > > Hi Jarrod, > > There are four current PRs making a start on the task, they need review and coordination. If you could help with that it would be a good start. They are tagged with `Py3K`. There are also several related issues with that tag. I think we can also remove Python <= 3.4 stuff, removing 3.5 can probably wait until NumPy 1.20. See https://github.com/numpy/numpy/labels/27%20-%20Py3K for the list of tagged PRs/issues. > > > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From ralf.gommers at gmail.com Fri Jan 3 02:30:43 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 3 Jan 2020 08:30:43 +0100 Subject: [Numpy-discussion] Cleaning up Python 2.7 code. In-Reply-To: <84fb66063b8a4992a50d1d9e4b3ab804311a2c95.camel@sipsolutions.net> References: <84fb66063b8a4992a50d1d9e4b3ab804311a2c95.camel@sipsolutions.net> Message-ID: On Fri, Jan 3, 2020 at 1:01 AM Sebastian Berg wrote: > On Thu, 2020-01-02 at 16:51 -0700, Charles R Harris wrote: > > Hi All, > > > > Just want to propose cleaning up the Python 2.7 compatibility code > > for NumPy 1.19. Any objections? > > > > Not from me, sounds like a good plan, lets get it over with :). > +1 Ralf > - Sebastian > > > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Fri Jan 3 07:11:44 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 3 Jan 2020 07:11:44 -0500 Subject: [Numpy-discussion] Deprecate numpy.dual? Message-ID: In response to some work on improving the documentation of `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard suggested that the documentation of the module `numpy.dual` should also be improved. When I mentioned this suggestion in the community meeting on December 11, it was suggested that we should probably deprecate `numpy.dual`. I think some current NumPy developers (myself included at the time the topic came up) are unfamiliar with the history and purpose of this module, so I spent some time reading code and github issues and wrote up some notes. These notes are available at https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md If you are not familiar with `numpy.dual`, you might find those notes useful. Now that I know a bit more about `numpy.dual`, I'm not sure it should be deprecated. It provides a hook for other libraries to selectively replace the use of the exposed functions in internal NumPy code, so if a library has a better version of, say, `linalg.eigh`, it can configure `numpy.dual` to use its version. Then, for example, NumPy multivariate normal distribution code could benefit from the use of that library's version of `eigh`. The NumPy documentation of `numpy.dual` refers specifically to SciPy, but it could be used by any library. Does anyone know if any other libraries use `register_func` to put their functions into the `numpy.dual` namespace? SciPy currently registers some functions, but there is an open issue in which it is proposed that SciPy no longer register its functions with `numpy.dual`: https://github.com/scipy/scipy/issues/10441 This email is to start the discussion of the future of `numpy.dual`. Some of the options: 1. Quietly continue the status quo. 2. Deprecate `numpy.dual`. 3. Spend time improving the documentation of this feature, and perhaps even expand the functions that are supported. What do you think? For those who were involved in the creation of `numpy.dual`: is it working out like you expected? If not, is it worthwhile maintaining it? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 3 11:29:47 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 03 Jan 2020 10:29:47 -0600 Subject: [Numpy-discussion] Deprecate numpy.dual? In-Reply-To: References: Message-ID: <0884ad9e787128070118cc602bc8d39adf24a5b0.camel@sipsolutions.net> On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote: > In response to some work on improving the documentation of > `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard > suggested that the documentation of the module `numpy.dual` should > also be improved. When I mentioned this suggestion in the community > meeting on December 11, it was suggested that we should probably > deprecate `numpy.dual`. > > I think some current NumPy developers (myself included at the time > the topic came up) are unfamiliar with the history and purpose of > this module, so I spent some time reading code and github issues and > wrote up some notes. These notes are available at > > > https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md > > If you are not familiar with `numpy.dual`, you might find those notes > useful. > > Now that I know a bit more about `numpy.dual`, I'm not sure it should > be deprecated. It provides a hook for other libraries to selectively > replace the use of the exposed functions in internal NumPy code, so > if a library has a better version of, say, `linalg.eigh`, it can > configure `numpy.dual` to use its version. Then, for example, NumPy > multivariate normal distribution code could benefit from the use of > that library's version of `eigh`. That is in principle true, but I do not think we use `dual` at all internally right now in numpy, and I doubt there is more than a hand full uses out there. Dual is an override mechanism for functionality on ndarrays implemented also by numpy. In either case, I still tend towards deprecation. It seems to have issues and the main use case probably was to improve the situation when NumPy was compiled without an optimized BLAS/LAPACK. That probably was a common problem at some point, but I am not sure it is still an issue. Overriding functionality with faster implementations is of course a valid use-case and maybe `dual` is not a bad solution to the problem [0]. But I think we should discuss this more generally with other options. IMO deprecating this practically unused thing now does not mean we cannot do something similar in the future. - Sebastian [0] It has its own namespace, so is opt-in for the end user. You can only support a single backend at a time, although I am not sure that matters too much. If overrides provide a function to override, it is explicit to the end user as to what gets executed as well. > The NumPy documentation of `numpy.dual` refers specifically to SciPy, > but it could be used by any library. Does anyone know if any other > libraries use `register_func` to put their functions into the > `numpy.dual` namespace? > > SciPy currently registers some functions, but there is an open issue > in which it is proposed that SciPy no longer register its functions > with `numpy.dual`: > > https://github.com/scipy/scipy/issues/10441 > > This email is to start the discussion of the future of `numpy.dual`. > Some of the options: > > 1. Quietly continue the status quo. > 2. Deprecate `numpy.dual`. > 3. Spend time improving the documentation of this feature, and > perhaps even expand the functions that are supported. > > What do you think? For those who were involved in the creation of > `numpy.dual`: is it working out like you expected? If not, is it > worthwhile maintaining it? > > Warren > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From warren.weckesser at gmail.com Fri Jan 3 11:53:58 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 3 Jan 2020 11:53:58 -0500 Subject: [Numpy-discussion] Deprecate numpy.dual? In-Reply-To: <0884ad9e787128070118cc602bc8d39adf24a5b0.camel@sipsolutions.net> References: <0884ad9e787128070118cc602bc8d39adf24a5b0.camel@sipsolutions.net> Message-ID: On 1/3/20, Sebastian Berg wrote: > On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote: >> In response to some work on improving the documentation of >> `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard >> suggested that the documentation of the module `numpy.dual` should >> also be improved. When I mentioned this suggestion in the community >> meeting on December 11, it was suggested that we should probably >> deprecate `numpy.dual`. >> >> I think some current NumPy developers (myself included at the time >> the topic came up) are unfamiliar with the history and purpose of >> this module, so I spent some time reading code and github issues and >> wrote up some notes. These notes are available at >> >> >> https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md >> >> If you are not familiar with `numpy.dual`, you might find those notes >> useful. >> >> Now that I know a bit more about `numpy.dual`, I'm not sure it should >> be deprecated. It provides a hook for other libraries to selectively >> replace the use of the exposed functions in internal NumPy code, so >> if a library has a better version of, say, `linalg.eigh`, it can >> configure `numpy.dual` to use its version. Then, for example, NumPy >> multivariate normal distribution code could benefit from the use of >> that library's version of `eigh`. > > That is in principle true, but I do not think we use `dual` at all > internally right now in numpy, and I doubt there is more than a hand > full uses out there. In the notes, I listed the internal uses of `numpy.dual` within numpy that I found: 1. In the code that generates random variates from the multivariate normal distribution, one of `svd`, `eigh` or `cholesky` are used from `numpy.dual`. 2. In `matrixlib/defmatrix.py`, the `.I` property of the `matrix` class uses either `inv` or `pinv` from `numpy.dual` to compute its value. 3. The window function `numpy.kaiser` uses `numpy.dual.i0`. > > Dual is an override mechanism for functionality on ndarrays implemented > also by numpy. > > In either case, I still tend towards deprecation. It seems to have > issues and the main use case probably was to improve the situation when > NumPy was compiled without an optimized BLAS/LAPACK. That probably was > a common problem at some point, but I am not sure it is still an issue. > > Overriding functionality with faster implementations is of course a > valid use-case and maybe `dual` is not a bad solution to the problem > [0]. But I think we should discuss this more generally with other > options. IMO deprecating this practically unused thing now does not > mean we cannot do something similar in the future. It probably makes sense to have the general discussion before deprecating `numpy.dual`--there is a (slim?) chance that `numpy.dual` will turn out to be the best option. Warren > > - Sebastian > > > [0] It has its own namespace, so is opt-in for the end user. You can > only support a single backend at a time, although I am not sure that > matters too much. If overrides provide a function to override, it is > explicit to the end user as to what gets executed as well. > > >> The NumPy documentation of `numpy.dual` refers specifically to SciPy, >> but it could be used by any library. Does anyone know if any other >> libraries use `register_func` to put their functions into the >> `numpy.dual` namespace? >> >> SciPy currently registers some functions, but there is an open issue >> in which it is proposed that SciPy no longer register its functions >> with `numpy.dual`: >> >> https://github.com/scipy/scipy/issues/10441 >> >> This email is to start the discussion of the future of `numpy.dual`. >> Some of the options: >> >> 1. Quietly continue the status quo. >> 2. Deprecate `numpy.dual`. >> 3. Spend time improving the documentation of this feature, and >> perhaps even expand the functions that are supported. >> >> What do you think? For those who were involved in the creation of >> `numpy.dual`: is it working out like you expected? If not, is it >> worthwhile maintaining it? >> >> Warren >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Fri Jan 3 13:13:14 2020 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 3 Jan 2020 13:13:14 -0500 Subject: [Numpy-discussion] Deprecate numpy.dual? In-Reply-To: References: <0884ad9e787128070118cc602bc8d39adf24a5b0.camel@sipsolutions.net> Message-ID: On Fri, Jan 3, 2020 at 11:55 AM Warren Weckesser wrote: > In the notes, I listed the internal uses of `numpy.dual` within numpy > that I found: > > 1. In the code that generates random variates from the multivariate normal > distribution, one of `svd`, `eigh` or `cholesky` are used from > `numpy.dual`. > This was always a bad idea (originally mine, FWIW). The variability in arithmetic introduced by using possibly-different LAPACK versions depending on whether or not scipy is installed (and how it was built) is not worth the very minor speed advantages. The covariance matrices are typically small. And where they are large (e.g. Gaussian processes), you should probably be doing your own decompositions instead of using `multivariate_normal()`. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From przemyslaw.simajchel at gmail.com Fri Jan 3 18:58:59 2020 From: przemyslaw.simajchel at gmail.com (=?UTF-8?Q?Przemys=C5=82aw_Simajchel?=) Date: Sat, 4 Jan 2020 00:58:59 +0100 Subject: [Numpy-discussion] Providing functions to convert axes of object arrays to and from lists Message-ID: Hi, A feature to implement functions converting last axes of object arrays to and from lists has been requested in the following issue: https://github.com/numpy/numpy/issues/14478 I've implemented this feature, and the code can be viewed in the following pull request: https://github.com/numpy/numpy/pull/15091 The original issue asked for functions operating on just the last axis of this array, but I decided to allow to choose a specific axis for conversion from array of objects to array of lists, because this allows for the functions to be somewhat inverses of eachother. The linked pull request provides an example showing where the inverse property wouldn't hold with only last axis conversions. I would like to hear your opinions about this, and if it's agreed upon,also discuss the choice of the default value of the axis parameter. At the moment it's the first axis, but I assume that this being the last axis would also make sense. Also regarding a bit more cosmetic issues, the names of the functions and their place in the project (currently lib/shape_base.py) should probably be discussed, as their current names are long and clunky, yet I haven't come up with anything prettier, that would also be this expressive. Please note that this is my first contribution to the project and so I'd also appreciate a little help regarding the failing tests on the pull request, locally everything passed so I'm not sure what's wrong. Cheers, Przemys?aw Simajchel -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbmcleod at gmail.com Sun Jan 5 14:09:26 2020 From: robbmcleod at gmail.com (Robert McLeod) Date: Sun, 5 Jan 2020 11:09:26 -0800 Subject: [Numpy-discussion] Announcing Numexpr 2.7.1 Message-ID: Hi everyone, This is a version bump to add support for Python 3.8 and NumPy 1.18. We are also removing support for Python 3.4. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.7.0 to 2.7.1 ---------------------------- - Python 3.8 support has been added. - Python 3.4 support is discontinued. - The tests are now compatible with NumPy 1.18. - `site.cfg.example` was updated to use the `libraries` tag instead of `mkl_libs`, which is recommended for newer version of NumPy. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcleod at gmail.com robert.mcleod at hitachi-hhtc.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jan 6 18:20:30 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Jan 2020 16:20:30 -0700 Subject: [Numpy-discussion] NumPy 1.18.1 released Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.18.1 has been released. This release contains fixes for bugs reported against NumPy 1.18.0. Two bugs in particular that caused widespread problems downstream were: - The cython random extension test was not using a temporary directory for building, resulting in a permission violation. Fixed. - Numpy distutils was appending -std=c99 to all C compiler runs, leading to changed behavior and compile problems downstream. That flag is now only applied when building numpy C code. The Python versions supported in this release are 3.5-3.8. Downstream developers should use Cython >= 0.29.14 for Python 3.8 support and OpenBLAS >= 3.7 to avoid errors on the Skylake architecture. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Matti Picus - Maxwell Aladago - Pauli Virtanen - Ralf Gommers - Tyler Reddy - Warren Weckesser *Pull requests merged* A total of 13 pull requests were merged for this release. - `#15158 `__: MAINT: Update pavement.py for towncrier. - `#15159 `__: DOC: add moved modules to 1.18 release note - `#15161 `__: MAINT, DOC: Minor backports and updates for 1.18.x - `#15176 `__: TST: Add assert_array_equal test for big integer arrays - `#15184 `__: BUG: use tmp dir and check version for cython test. - `#15220 `__: BUG: distutils: fix msvc+gfortran openblas handling corner case - `#15221 `__: BUG: remove -std=c99 for c++ compilation - `#15222 `__: MAINT: unskip test on win32 - `#15223 `__: TST: add BLAS ILP64 run in Travis & Azure - `#15245 `__: MAINT: only add --std=c99 where needed - `#15246 `__: BUG: lib: Fix handling of integer arrays by gradient. - `#15247 `__: MAINT: Do not use private Python function in testing - `#15250 `__: REL: Prepare for the NumPy 1.18.1 release. Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Mon Jan 6 23:29:47 2020 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 6 Jan 2020 20:29:47 -0800 Subject: [Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules Message-ID: I am pleased to present a new NumPy Enhancement Proposal for discussion: "NEP-37: A dispatch protocol for NumPy-like modules." Feedback would be very welcome! The full text follows. The rendered proposal can also be found online at https://numpy.org/neps/nep-0037-array-module.html Best, Stephan Hoyer =================================================== NEP 37 ? A dispatch protocol for NumPy-like modules =================================================== :Author: Stephan Hoyer :Author: Hameer Abbasi :Author: Sebastian Berg :Status: Draft :Type: Standards Track :Created: 2019-12-29 Abstract -------- NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most use-cases for ``__array_function__``. The protocol requires explicit adoption by both users and library authors, which ensures backwards compatibility, and is also significantly simpler than ``__array_function__``, both of which we expect will make it easier to adopt. Why ``__array_function__`` hasn't been enough --------------------------------------------- There are two broad ways in which NEP-18 has fallen short of its goals: 1. **Maintainability concerns**. `__array_function__` has significant implications for libraries that use it: - Projects like `PyTorch `_, `JAX `_ and even `scipy.sparse `_ have been reluctant to implement `__array_function__` in part because they are concerned about **breaking existing code**: users expect NumPy functions like ``np.concatenate`` to return NumPy arrays. This is a fundamental limitation of the ``__array_function__`` design, which we chose to allow overriding the existing ``numpy`` namespace. - ``__array_function__`` currently requires an "all or nothing" approach to implementing NumPy's API. There is no good pathway for **incremental adoption**, which is particularly problematic for established projects for which adopting ``__array_function__`` would result in breaking changes. - It is no longer possible to use **aliases to NumPy functions** within modules that support overrides. For example, both CuPy and JAX set ``result_type = np.result_type``. - Implementing **fall-back mechanisms** for unimplemented NumPy functions by using NumPy's implementation is hard to get right (but see the `version from dask `_), because ``__array_function__`` does not present a consistent interface. Converting all arguments of array type requires recursing into generic arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - **Array creation** routines (e.g., ``np.arange`` and those in ``np.random``) need some other mechanism for indicating what type of arrays to create. `NEP 36 `_ proposed adding optional ``like=`` arguments to functions without existing array arguments. However, we still lack any mechanism to override methods on objects, such as those needed by ``np.random.RandomState``. - **Array conversion** can't reuse the existing coercion functions like ``np.asarray``, because ``np.asarray`` sometimes means "convert to an exact ``np.ndarray``" and other times means "convert to something _like_ a NumPy array." This led to the `NEP 30 `_ proposal for a separate ``np.duckarray`` function, but this still does not resolve how to cast one duck array into a type matching another duck array. ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- We propose a new user-facing mechanism for dispatching to a duck-array implementation, ``numpy.get_array_module``. ``get_array_module`` performs the same type resolution as ``__array_function__`` and returns a module with an API promised to match the standard interface of ``numpy`` that can implement operations on all provided array types. The protocol itself is both simpler and more powerful than ``__array_function__``, because it doesn't need to worry about actually implementing functions. We believe it resolves most of the maintainability and functionality limitations of ``__array_function__``. The new protocol is opt-in, explicit and with local control; see :ref:`appendix-design-choices` for discussion on the importance of these design features. The array module contract ========================= Modules returned by ``get_array_module``/``__array_module__`` should make a best effort to implement NumPy's core functionality on new array types(s). Unimplemented functionality should simply be omitted (e.g., accessing an unimplemented function should raise ``AttributeError``). In the future, we anticipate codifying a protocol for requesting restricted subsets of ``numpy``; see :ref:`requesting-restricted-subsets` for more details. How to use ``get_array_module`` =============================== Code that wants to support generic duck arrays should explicitly call ``get_array_module`` to determine an appropriate array module from which to call functions, rather than using the ``numpy`` namespace directly. For example: .. code:: python # calls the appropriate version of np.something for x and y module = np.get_array_module(x, y) module.something(x, y) Both array creation and array conversion are supported, because dispatching is handled by ``get_array_module`` rather than via the types of function arguments. For example, to use random number generation functions or methods, we can simply pull out the appropriate submodule: .. code:: python def duckarray_add_random(array): module = np.get_array_module(array) noise = module.random.randn(*array.shape) return array + noise We can also write the duck-array ``stack`` function from `NEP 30 `_, without the need for a new ``np.duckarray`` function: .. code:: python def duckarray_stack(arrays): module = np.get_array_module(*arrays) arrays = [module.asarray(arr) for arr in arrays] shapes = {arr.shape for arr in arrays} if len(shapes) != 1: raise ValueError('all input arrays must have the same shape') expanded_arrays = [arr[module.newaxis, ...] for arr in arrays] return module.concatenate(expanded_arrays, axis=0) By default, ``get_array_module`` will return the ``numpy`` module if no arguments are arrays. This fall-back can be explicitly controlled by providing the ``module`` keyword-only argument. It is also possible to indicate that an exception should be raised instead of returning a default array module by setting ``module=None``. How to implement ``__array_module__`` ===================================== Libraries implementing a duck array type that want to support ``get_array_module`` need to implement the corresponding protocol, ``__array_module__``. This new protocol is based on Python's dispatch protocol for arithmetic, and is essentially a simpler version of ``__array_function__``. Only one argument is passed into ``__array_module__``, a Python collection of unique array types passed into ``get_array_module``, i.e., all arguments with an ``__array_module__`` attribute. The special method should either return an namespace with an API matching ``numpy``, or ``NotImplemented``, indicating that it does not know how to handle the operation: .. code:: python class MyArray: def __array_module__(self, types): if not all(issubclass(t, MyArray) for t in types): return NotImplemented return my_array_module Returning custom objects from ``__array_module__`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``my_array_module`` will typically, but need not always, be a Python module. Returning a custom objects (e.g., with functions implemented via ``__getattr__``) may be useful for some advanced use cases. For example, custom objects could allow for partial implementations of duck array modules that fall-back to NumPy (although this is not recommended in general because such fall-back behavior can be error prone): .. code:: python class MyArray: def __array_module__(self, types): if all(issubclass(t, MyArray) for t in types): return ArrayModule() else: return NotImplemented class ArrayModule: def __getattr__(self, name): import base_module return getattr(base_module, name, getattr(numpy, name)) Subclassing from ``numpy.ndarray`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the same guidance about well-defined type casting hierarchies from NEP-18 still applies. ``numpy.ndarray`` itself contains a matching implementation of ``__array_module__``, which is convenient for subclasses: .. code:: python class ndarray: def __array_module__(self, types): if all(issubclass(t, ndarray) for t in types): return numpy else: return NotImplemented NumPy's internal machinery ========================== The type resolution rules of ``get_array_module`` follow the same model as Python and NumPy's existing dispatch protocols: subclasses are called before super-classes, and otherwise left to right. ``__array_module__`` is guaranteed to be called only a single time on each unique type. The actual implementation of `get_array_module` will be in C, but should be equivalent to this Python code: .. code:: python def get_array_module(*arrays, default=numpy): implementing_arrays, types = _implementing_arrays_and_types(arrays) if not implementing_arrays and default is not None: return default for array in implementing_arrays: module = array.__array_module__(types) if module is not NotImplemented: return module raise TypeError("no common array module found") def _implementing_arrays_and_types(relevant_arrays): types = [] implementing_arrays = [] for array in relevant_arrays: t = type(array) if t not in types and hasattr(t, '__array_module__'): types.append(t) # Subclasses before superclasses, otherwise left to right index = len(implementing_arrays) for i, old_array in enumerate(implementing_arrays): if issubclass(t, type(old_array)): index = i break implementing_arrays.insert(index, array) return implementing_arrays, types Relationship with ``__array_ufunc__`` and ``__array_function__`` ---------------------------------------------------------------- These older protocols have distinct use-cases and should remain =============================================================== ``__array_module__`` is intended to resolve limitations of ``__array_function__``, so it is natural to consider whether it could entirely replace ``__array_function__``. This would offer dual benefits: (1) simplifying the user-story about how to override NumPy and (2) removing the slowdown associated with checking for dispatch when calling every NumPy function. However, ``__array_module__`` and ``__array_function__`` are pretty different from a user perspective: it requires explicit calls to ``get_array_function``, rather than simply reusing original ``numpy`` functions. This is probably fine for *libraries* that rely on duck-arrays, but may be frustratingly verbose for interactive use. Some of the dispatching use-cases for ``__array_ufunc__`` are also solved by ``__array_module__``, but not all of them. For example, it is still useful to be able to define non-NumPy ufuncs (e.g., from Numba or SciPy) in a generic way on non-NumPy arrays (e.g., with dask.array). Given their existing adoption and distinct use cases, we don't think it makes sense to remove or deprecate ``__array_function__`` and ``__array_ufunc__`` at this time. Mixin classes to implement ``__array_function__`` and ``__array_ufunc__`` ========================================================================= Despite the user-facing differences, ``__array_module__`` and a module implementing NumPy's API still contain sufficient functionality needed to implement dispatching with the existing duck array protocols. For example, the following mixin classes would provide sensible defaults for these special methods in terms of ``get_array_module`` and ``__array_module__``: .. code:: python class ArrayUfuncFromModuleMixin: def __array_ufunc__(self, ufunc, method, *inputs, **kwargs): arrays = inputs + kwargs.get('out', ()) try: array_module = np.get_array_module(*arrays) except TypeError: return NotImplemented try: # Note this may have false positive matches, if ufunc.__name__ # matches the name of a ufunc defined by NumPy. Unfortunately # there is no way to determine in which module a ufunc was # defined. new_ufunc = getattr(array_module, ufunc.__name__) except AttributeError: return NotImplemented try: callable = getattr(new_ufunc, method) except AttributeError: return NotImplemented return callable(*inputs, **kwargs) class ArrayFunctionFromModuleMixin: def __array_function__(self, func, types, args, kwargs): array_module = self.__array_module__(types) if array_module is NotImplemented: return NotImplemented # Traverse submodules to find the appropriate function modules = func.__module__.split('.') assert modules[0] == 'numpy' for submodule in modules[1:]: module = getattr(module, submodule, None) new_func = getattr(module, func.__name__, None) if new_func is None: return NotImplemented return new_func(*args, **kwargs) To make it easier to write duck arrays, we could also add these mixin classes into ``numpy.lib.mixins`` (but the examples above may suffice). Alternatives considered ----------------------- Naming ====== We like the name ``__array_module__`` because it mirrors the existing ``__array_function__`` and ``__array_ufunc__`` protocols. Another reasonable choice could be ``__array_namespace__``. It is less clear what the NumPy function that calls this protocol should be called (``get_array_module`` in this proposal). Some possible alternatives: ``array_module``, ``common_array_module``, ``resolve_array_module``, ``get_namespace``, ``get_numpy``, ``get_numpylike_module``, ``get_duck_array_module``. .. _requesting-restricted-subsets: Requesting restricted subsets of NumPy's API ============================================ Over time, NumPy has accumulated a very large API surface, with over 600 attributes in the top level ``numpy`` module alone. It is unlikely that any duck array library could or would want to implement all of these functions and classes, because the frequently used subset of NumPy is much smaller. We think it would be useful exercise to define "minimal" subset(s) of NumPy's API, omitting rarely used or non-recommended functionality. For example, minimal NumPy might include ``stack``, but not the other stacking functions ``column_stack``, ``dstack``, ``hstack`` and ``vstack``. This could clearly indicate to duck array authors and users want functionality is core and what functionality they can skip. Support for requesting a restricted subset of NumPy's API would be a natural feature to include in ``get_array_function`` and ``__array_module__``, e.g., .. code:: python # array_module is only guaranteed to contain "minimal" NumPy array_module = np.get_array_module(*arrays, request='minimal') To facilitate testing with NumPy and use with any valid duck array library, NumPy itself would return restricted versions of the ``numpy`` module when ``get_array_module`` is called only on NumPy arrays. Omitted functions would simply not exist. Unfortunately, we have not yet figured out what these restricted subsets should be, so it doesn't make sense to do this yet. When/if we do, we could either add new keyword arguments to ``get_array_module`` or add new top level functions, e.g., ``get_minimal_array_module``. We would also need to add either a new protocol patterned off of ``__array_module__`` (e.g., ``__array_module_minimal__``), or could add an optional second argument to ``__array_module__`` (catching errors with ``try``/``except``). A new namespace for implicit dispatch ===================================== Instead of supporting overrides in the main `numpy` namespace with ``__array_function__``, we could create a new opt-in namespace, e.g., ``numpy.api``, with versions of NumPy functions that support dispatching. These overrides would need new opt-in protocols, e.g., ``__array_function_api__`` patterned off of ``__array_function__``. This would resolve the biggest limitations of ``__array_function__`` by being opt-in and would also allow for unambiguously overriding functions like ``asarray``, because ``np.api.asarray`` would always mean "convert an array-like object." But it wouldn't solve all the dispatching needs met by ``__array_module__``, and would leave us with supporting a considerably more complex protocol both for array users and implementors. We could potentially implement such a new namespace *via* the ``__array_module__`` protocol. Certainly some users would find this convenient, because it is slightly less boilerplate. But this would leave users with a confusing choice: when should they use `get_array_module` vs. `np.api.something`. Also, we would have to add and maintain a whole new module, which is considerably more expensive than merely adding a function. Dispatching on both types and arrays instead of only types ========================================================== Instead of supporting dispatch only via unique array types, we could also support dispatch via array objects, e.g., by passing an ``arrays`` argument as part of the ``__array_module__`` protocol. This could potentially be useful for dispatch for arrays with metadata, such provided by Dask and Pint, but would impose costs in terms of type safety and complexity. For example, a library that supports arrays on both CPUs and GPUs might decide on which device to create a new arrays from functions like ``ones`` based on input arguments: .. code:: python class Array: def __array_module__(self, types, arrays): useful_arrays = tuple(a in arrays if isinstance(a, Array)) if not useful_arrays: return NotImplemented prefer_gpu = any(a.prefer_gpu for a in useful_arrays) return ArrayModule(prefer_gpu) class ArrayModule: def __init__(self, prefer_gpu): self.prefer_gpu = prefer_gpu def __getattr__(self, name): import base_module base_func = getattr(base_module, name) return functools.partial(base_func, prefer_gpu=self.prefer_gpu) This might be useful, but it's not clear if we really need it. Pint seems to get along OK without any explicit array creation routines (favoring multiplication by units, e.g., ``np.ones(5) * ureg.m``), and for the most part Dask is also OK with existing ``__array_function__`` style overides (e.g., favoring ``np.ones_like`` over ``np.ones``). Choosing whether to place an array on the CPU or GPU could be solved by `making array creation lazy `_. .. _appendix-design-choices: Appendix: design choices for API overrides ------------------------------------------ There is a large range of possible design choices for overriding NumPy's API. Here we discuss three major axes of the design decision that guided our design for ``__array_module__``. Opt-in vs. opt-out for users ============================ The ``__array_ufunc__`` and ``__array_function__`` protocols provide a mechanism for overriding NumPy functions *within NumPy's existing namespace*. This means that users need to explicitly opt-out if they do not want any overridden behavior, e.g., by casting arrays with ``np.asarray()``. In theory, this approach lowers the barrier for adopting these protocols in user code and libraries, because code that uses the standard NumPy namespace is automatically compatible. But in practice, this hasn't worked out. For example, most well-maintained libraries that use NumPy follow the best practice of casting all inputs with ``np.asarray()``, which they would have to explicitly relax to use ``__array_function__``. Our experience has been that making a library compatible with a new duck array type typically requires at least a small amount of work to accommodate differences in the data model and operations that can be implemented efficiently. These opt-out approaches also considerably complicate backwards compatibility for libraries that adopt these protocols, because by opting in as a library they also opt-in their users, whether they expect it or not. For winning over libraries that have been unable to adopt ``__array_function__``, an opt-in approach seems like a must. Explicit vs. implicit choice of implementation ============================================== Both ``__array_ufunc__`` and ``__array_function__`` have implicit control over dispatching: the dispatched functions are determined via the appropriate protocols in every function call. This generalizes well to handling many different types of objects, as evidenced by its use for implementing arithmetic operators in Python, but it has two downsides: 1. *Speed*: it imposes additional overhead in every function call, because each function call needs to inspect each of its arguments for overrides. This is why arithmetic on builtin Python numbers is slow. 2. *Readability*: it is not longer immediately evident to readers of code what happens when a function is called, because the function's implementation could be overridden by any of its arguments. In contrast, importing a new library (e.g., ``import dask.array as da``) with an API matching NumPy is entirely explicit. There is no overhead from dispatch or ambiguity about which implementation is being used. Explicit and implicit choice of implementations are not mutually exclusive options. Indeed, most implementations of NumPy API overrides via ``__array_function__`` that we are familiar with (namely, dask, CuPy and sparse, but not Pint) also include an explicit way to use their version of NumPy's API by importing a module directly (``dask.array``, ``cupy`` or ``sparse``, respectively). Local vs. non-local vs. global control ====================================== The final design axis is how users control the choice of API: - **Local control**, as exemplified by multiple dispatch and Python protocols for arithmetic, determines which implementation to use either by checking types or calling methods on the direct arguments of a function. - **Non-local control** such as `np.errstate `_ overrides behavior with global-state via function decorators or context-managers. Control is determined hierarchically, via the inner-most context. - **Global control** provides a mechanism for users to set default behavior, either via function calls or configuration files. For example, matplotlib allows setting a global choice of plotting backend. Local control is generally considered a best practice for API design, because control flow is entirely explicit, which makes it the easiest to understand. Non-local and global control are occasionally used, but generally either due to ignorance or a lack of better alternatives. In the case of duck typing for NumPy's public API, we think non-local or global control would be mistakes, mostly because they **don't compose well**. If one library sets/needs one set of overrides and then internally calls a routine that expects another set of overrides, the resulting behavior may be very surprising. Higher order functions are especially problematic, because the context in which functions are evaluated may not be the context in which they are defined. One class of override use cases where we think non-local and global control are appropriate is for choosing a backend system that is guaranteed to have an entirely consistent interface, such as a faster alternative implementation of ``numpy.fft`` on NumPy arrays. However, these are out of scope for the current proposal, which is focused on duck arrays. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jan 7 14:39:48 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 07 Jan 2020 13:39:48 -0600 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, Jan. 8 Message-ID: <884388ddb87113af846dd106cffa8d6869228ad6.camel@sipsolutions.net> Hi all, After the holiday break, there will be a NumPy Community meeting Wednesday January 8 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Tue Jan 7 16:11:31 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 7 Jan 2020 22:11:31 +0100 Subject: [Numpy-discussion] Deprecate numpy.dual? In-Reply-To: References: <0884ad9e787128070118cc602bc8d39adf24a5b0.camel@sipsolutions.net> Message-ID: On Fri, Jan 3, 2020 at 5:54 PM Warren Weckesser wrote: > On 1/3/20, Sebastian Berg wrote: > > On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote: > >> In response to some work on improving the documentation of > >> `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard > >> suggested that the documentation of the module `numpy.dual` should > >> also be improved. When I mentioned this suggestion in the community > >> meeting on December 11, it was suggested that we should probably > >> deprecate `numpy.dual`. > >> > >> I think some current NumPy developers (myself included at the time > >> the topic came up) are unfamiliar with the history and purpose of > >> this module, so I spent some time reading code and github issues and > >> wrote up some notes. These notes are available at > >> > >> > >> > https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md > >> > >> If you are not familiar with `numpy.dual`, you might find those notes > >> useful. > >> > >> Now that I know a bit more about `numpy.dual`, I'm not sure it should > >> be deprecated. It provides a hook for other libraries to selectively > >> replace the use of the exposed functions in internal NumPy code, so > >> if a library has a better version of, say, `linalg.eigh`, it can > >> configure `numpy.dual` to use its version. Then, for example, NumPy > >> multivariate normal distribution code could benefit from the use of > >> that library's version of `eigh`. > > > > That is in principle true, but I do not think we use `dual` at all > > internally right now in numpy, and I doubt there is more than a hand > > full uses out there. > > In the notes, I listed the internal uses of `numpy.dual` within numpy > that I found: > > 1. In the code that generates random variates from the multivariate normal > distribution, one of `svd`, `eigh` or `cholesky` are used from > `numpy.dual`. > Agree with Robert's comment on this: not a good idea. 2. In `matrixlib/defmatrix.py`, the `.I` property of the `matrix` class > uses either `inv` or `pinv` from `numpy.dual` to compute its value. > Easy to get rid of, and not very interesting. 3. The window function `numpy.kaiser` uses `numpy.dual.i0`. > SciPy doesn't even register its special.i0. We should just get rid of all this. > > > > > > Dual is an override mechanism for functionality on ndarrays implemented > > also by numpy. > > > > In either case, I still tend towards deprecation. It seems to have > > issues and the main use case probably was to improve the situation when > > NumPy was compiled without an optimized BLAS/LAPACK. That probably was > > a common problem at some point, but I am not sure it is still an issue. > That's not an issue anymore, it's been years since that was popping up occasionally. > > > Overriding functionality with faster implementations is of course a > > valid use-case and maybe `dual` is not a bad solution to the problem > > [0]. But I think we should discuss this more generally with other > > options. IMO deprecating this practically unused thing now does not > > mean we cannot do something similar in the future. > +1 > It probably makes sense to have the general discussion before > deprecating `numpy.dual`--there is a (slim?) chance that `numpy.dual` > will turn out to be the best option. > I think it's *very* safe to say that's not the case. Let's just be pragmatic and not hedge on slim options. Cheers, Ralf > > Warren > > > > > > - Sebastian > > > > > > [0] It has its own namespace, so is opt-in for the end user. You can > > only support a single backend at a time, although I am not sure that > > matters too much. If overrides provide a function to override, it is > > explicit to the end user as to what gets executed as well. > > > > > >> The NumPy documentation of `numpy.dual` refers specifically to SciPy, > >> but it could be used by any library. Does anyone know if any other > >> libraries use `register_func` to put their functions into the > >> `numpy.dual` namespace? > >> > >> SciPy currently registers some functions, but there is an open issue > >> in which it is proposed that SciPy no longer register its functions > >> with `numpy.dual`: > >> > >> https://github.com/scipy/scipy/issues/10441 > >> > >> This email is to start the discussion of the future of `numpy.dual`. > >> Some of the options: > >> > >> 1. Quietly continue the status quo. > >> 2. Deprecate `numpy.dual`. > >> 3. Spend time improving the documentation of this feature, and > >> perhaps even expand the functions that are supported. > >> > >> What do you think? For those who were involved in the creation of > >> `numpy.dual`: is it working out like you expected? If not, is it > >> worthwhile maintaining it? > >> > >> Warren > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Jan 8 08:22:35 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 8 Jan 2020 08:22:35 -0500 Subject: [Numpy-discussion] Is the build-system section of pyproject.toml a maintained list of the build dependencies? Message-ID: I'm doing some work on the travis-ci scripts, and I'd like to remove some redundant calls of 'pip install'. The scripts should get the build dependencies from a configuration file instead of having hard-coded pip commands. Is pyproject.toml the appropriate file to use for this? (Note: we also have test_requirements.txt, but as the name says, those are the dependencies for running the tests, not for building numpy.) Warren From warren.weckesser at gmail.com Wed Jan 8 11:37:14 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 8 Jan 2020 11:37:14 -0500 Subject: [Numpy-discussion] Is the build-system section of pyproject.toml a maintained list of the build dependencies? In-Reply-To: References: Message-ID: On 1/8/20, Warren Weckesser wrote: > I'm doing some work on the travis-ci scripts, and I'd like to remove > some redundant calls of 'pip install'. The scripts should get the > build dependencies from a configuration file instead of having > hard-coded pip commands. Is pyproject.toml the appropriate file to > use for this? (Note: we also have test_requirements.txt, but as the > name says, those are the dependencies for running the tests, not for > building numpy.) > Updating my question: `pyproject.toml` lists numpy's build dependencies in the `build_system` section of the file: [build-system] # Minimum requirements for the build system to execute. requires = [ "setuptools", "wheel", "Cython>=0.29.14", # Note: keep in sync with tools/cythonize.py ] So the file serves the equivalent purpose of a `requirements.txt` file. Is there an option to pip that would allow something like pip install -r pyproject.toml (with some other flag or option as needed) to install the build requirements found in pyproject.toml? In https://github.com/numpy/numpy/pull/15275, I wrote a few lines of Python to get the dependencies from pyproject.toml, but it seems like that shouldn't be necessary. Warren > Warren > From kevin.k.sheppard at gmail.com Wed Jan 8 11:43:25 2020 From: kevin.k.sheppard at gmail.com (Kevin Sheppard) Date: Wed, 8 Jan 2020 16:43:25 +0000 Subject: [Numpy-discussion] Is the build-system section of pyproject.toml a maintained list of the build dependencies? In-Reply-To: References: Message-ID: With recent versions of pip it will read the pyproject.toml file to get the dependencies, and then install these in an isolated environment to build the wheel, and then install the wheel. The requires=[...] in the pyproject is not installed in the original environment, so that when run on NumPy, you would only end up with NumPy installed. Are you trying to get Cython installed after an install of NumPy? If you want this then it needs to be listed in the setup as a dependency. On Wed, Jan 8, 2020 at 4:38 PM Warren Weckesser wrote: > On 1/8/20, Warren Weckesser wrote: > > I'm doing some work on the travis-ci scripts, and I'd like to remove > > some redundant calls of 'pip install'. The scripts should get the > > build dependencies from a configuration file instead of having > > hard-coded pip commands. Is pyproject.toml the appropriate file to > > use for this? (Note: we also have test_requirements.txt, but as the > > name says, those are the dependencies for running the tests, not for > > building numpy.) > > > > Updating my question: `pyproject.toml` lists numpy's build > dependencies in the `build_system` section of the file: > > [build-system] > # Minimum requirements for the build system to execute. > requires = [ > "setuptools", > "wheel", > "Cython>=0.29.14", # Note: keep in sync with tools/cythonize.py > ] > > So the file serves the equivalent purpose of a `requirements.txt` > file. Is there an option to pip that would allow something like > > pip install -r pyproject.toml > > (with some other flag or option as needed) to install the build > requirements found in pyproject.toml? In > https://github.com/numpy/numpy/pull/15275, I wrote a few lines of > Python to get the dependencies from pyproject.toml, but it seems like > that shouldn't be necessary. > > Warren > > > > Warren > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Jan 8 12:01:39 2020 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 8 Jan 2020 12:01:39 -0500 Subject: [Numpy-discussion] Is the build-system section of pyproject.toml a maintained list of the build dependencies? In-Reply-To: References: Message-ID: On 1/8/20, Kevin Sheppard wrote: > With recent versions of pip it will read the pyproject.toml file to get the > dependencies, and then install these in an isolated environment to build > the wheel, and then install the wheel. The requires=[...] in the pyproject > is not installed in the original environment, so that when run on NumPy, > you would only end up with NumPy installed. Are you trying to get Cython > installed after an install of NumPy? If you want this then it needs to be > listed in the setup as a dependency Thanks Kevin. I'm cleaning up the shell scripts that we use on Travis-CI. There were several redundant uses of `pip install`, some of which were installing build requirements. The pull request is https://github.com/numpy/numpy/pull/15275, but I think the code can be further simplified. Does building with setup.py (instead of pip) use pyproject.toml? Warren > > On Wed, Jan 8, 2020 at 4:38 PM Warren Weckesser > > wrote: > >> On 1/8/20, Warren Weckesser wrote: >> > I'm doing some work on the travis-ci scripts, and I'd like to remove >> > some redundant calls of 'pip install'. The scripts should get the >> > build dependencies from a configuration file instead of having >> > hard-coded pip commands. Is pyproject.toml the appropriate file to >> > use for this? (Note: we also have test_requirements.txt, but as the >> > name says, those are the dependencies for running the tests, not for >> > building numpy.) >> > >> >> Updating my question: `pyproject.toml` lists numpy's build >> dependencies in the `build_system` section of the file: >> >> [build-system] >> # Minimum requirements for the build system to execute. >> requires = [ >> "setuptools", >> "wheel", >> "Cython>=0.29.14", # Note: keep in sync with tools/cythonize.py >> ] >> >> So the file serves the equivalent purpose of a `requirements.txt` >> file. Is there an option to pip that would allow something like >> >> pip install -r pyproject.toml >> >> (with some other flag or option as needed) to install the build >> requirements found in pyproject.toml? In >> https://github.com/numpy/numpy/pull/15275, I wrote a few lines of >> Python to get the dependencies from pyproject.toml, but it seems like >> that shouldn't be necessary. >> >> Warren >> >> >> > Warren >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > From ralf.gommers at gmail.com Wed Jan 8 12:15:00 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 8 Jan 2020 18:15:00 +0100 Subject: [Numpy-discussion] Is the build-system section of pyproject.toml a maintained list of the build dependencies? In-Reply-To: References: Message-ID: On Wed, Jan 8, 2020 at 6:04 PM Warren Weckesser wrote: > On 1/8/20, Kevin Sheppard wrote: > > With recent versions of pip it will read the pyproject.toml file to get > the > > dependencies, and then install these in an isolated environment to build > > the wheel, and then install the wheel. The requires=[...] in the > pyproject > > is not installed in the original environment, so that when run on NumPy, > > you would only end up with NumPy installed. Are you trying to get Cython > > installed after an install of NumPy? If you want this then it needs to > be > > listed in the setup as a dependency > > > Thanks Kevin. I'm cleaning up the shell scripts that we use on > Travis-CI. There were several redundant uses of `pip install`, some > of which were installing build requirements. The pull request is > https://github.com/numpy/numpy/pull/15275, but I think the code can be > further simplified. > > Does building with setup.py (instead of pip) use pyproject.toml? > No it doesn't. Ralf > Warren > > > > > > > On Wed, Jan 8, 2020 at 4:38 PM Warren Weckesser > > > > wrote: > > > >> On 1/8/20, Warren Weckesser wrote: > >> > I'm doing some work on the travis-ci scripts, and I'd like to remove > >> > some redundant calls of 'pip install'. The scripts should get the > >> > build dependencies from a configuration file instead of having > >> > hard-coded pip commands. Is pyproject.toml the appropriate file to > >> > use for this? (Note: we also have test_requirements.txt, but as the > >> > name says, those are the dependencies for running the tests, not for > >> > building numpy.) > >> > > >> > >> Updating my question: `pyproject.toml` lists numpy's build > >> dependencies in the `build_system` section of the file: > >> > >> [build-system] > >> # Minimum requirements for the build system to execute. > >> requires = [ > >> "setuptools", > >> "wheel", > >> "Cython>=0.29.14", # Note: keep in sync with tools/cythonize.py > >> ] > >> > >> So the file serves the equivalent purpose of a `requirements.txt` > >> file. Is there an option to pip that would allow something like > >> > >> pip install -r pyproject.toml > >> > >> (with some other flag or option as needed) to install the build > >> requirements found in pyproject.toml? In > >> https://github.com/numpy/numpy/pull/15275, I wrote a few lines of > >> Python to get the dependencies from pyproject.toml, but it seems like > >> that shouldn't be necessary. > >> > >> Warren > >> > >> > >> > Warren > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Jan 8 12:51:06 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 08 Jan 2020 11:51:06 -0600 Subject: [Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules In-Reply-To: References: Message-ID: Thanks, maybe to start discussion floating the actual usage here: ``` def add_noise(array_like): module = np.get_array_module(array_like) noise = module.random.randn(*array_like.shape) return array_like + noise ``` The above function could also include `module.asarray(array_like)` to support non-array inputs. Importantly, the random function, and especially array creation functions such as `empty` and `ones` can work. To summarize I think there are two main things that this NEP can address: 1. Some libraries are reluctant to adopt `__array_function__`, but they could adopt this NEP. 2. Libraries written for numpy (scipy, sklearn, etc.) often use `np.asarray` and `__array_function__` does not help them easily. This NEP hopefully gives them a way forward. We may need to prototype some examples, but right now it feels like this should be a step forward, especially for libraries. Of course there are other similar design options, so discussions (or criticism of this idea) are welcome. I believe this can help libraries, i.e. if skimage only feels confident that they support Dask, they can still do: ``` module = np.get_array_module(*input_arrays) if module not in {np, dask.numpy_api}: raise TypeError("This function only supports numpy and Dask.") ``` I do not think this is as cleanly possibly with `__array_function__`. Best, Sebastian On Mon, 2020-01-06 at 20:29 -0800, Stephan Hoyer wrote: > I am pleased to present a new NumPy Enhancement Proposal for > discussion: "NEP-37: A dispatch protocol for NumPy-like modules." > Feedback would be very welcome! > > The full text follows. The rendered proposal can also be found online > at https://numpy.org/neps/nep-0037-array-module.html > > Best, > Stephan Hoyer > > =================================================== > NEP 37 ? A dispatch protocol for NumPy-like modules > =================================================== > > :Author: Stephan Hoyer > :Author: Hameer Abbasi > :Author: Sebastian Berg > :Status: Draft > :Type: Standards Track > :Created: 2019-12-29 > > Abstract > -------- > > NEP-18's ``__array_function__`` has been a mixed success. Some > projects (e.g., > dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. > Others > (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose > a new > protocol, ``__array_module__``, that we expect could eventually > subsume most > use-cases for ``__array_function__``. The protocol requires explicit > adoption > by both users and library authors, which ensures backwards > compatibility, and > is also significantly simpler than ``__array_function__``, both of > which we > expect will make it easier to adopt. > > Why ``__array_function__`` hasn't been enough > --------------------------------------------- > > There are two broad ways in which NEP-18 has fallen short of its > goals: > > 1. **Maintainability concerns**. `__array_function__` has significant > implications for libraries that use it: > > - Projects like `PyTorch > `_, `JAX > `_ and even > `scipy.sparse > `_ have been > reluctant to > implement `__array_function__` in part because they are > concerned about > **breaking existing code**: users expect NumPy functions like > ``np.concatenate`` to return NumPy arrays. This is a fundamental > limitation of the ``__array_function__`` design, which we chose > to allow > overriding the existing ``numpy`` namespace. > - ``__array_function__`` currently requires an "all or nothing" > approach to > implementing NumPy's API. There is no good pathway for > **incremental > adoption**, which is particularly problematic for established > projects > for which adopting ``__array_function__`` would result in > breaking > changes. > - It is no longer possible to use **aliases to NumPy functions** > within > modules that support overrides. For example, both CuPy and JAX > set > ``result_type = np.result_type``. > - Implementing **fall-back mechanisms** for unimplemented NumPy > functions > by using NumPy's implementation is hard to get right (but see > the > `version from dask `_), > because > ``__array_function__`` does not present a consistent interface. > Converting all arguments of array type requires recursing into > generic > arguments of the form ``*args, **kwargs``. > > 2. **Limitations on what can be overridden.** ``__array_function__`` > has some > important gaps, most notably array creation and coercion > functions: > > - **Array creation** routines (e.g., ``np.arange`` and those in > ``np.random``) need some other mechanism for indicating what > type of > arrays to create. `NEP 36 < > https://github.com/numpy/numpy/pull/14715>`_ > proposed adding optional ``like=`` arguments to functions > without > existing array arguments. However, we still lack any mechanism > to > override methods on objects, such as those needed by > ``np.random.RandomState``. > - **Array conversion** can't reuse the existing coercion functions > like > ``np.asarray``, because ``np.asarray`` sometimes means "convert > to an > exact ``np.ndarray``" and other times means "convert to > something _like_ > a NumPy array." This led to the `NEP 30 > `_ > proposal for > a separate ``np.duckarray`` function, but this still does not > resolve how > to cast one duck array into a type matching another duck array. > > ``get_array_module`` and the ``__array_module__`` protocol > ---------------------------------------------------------- > > We propose a new user-facing mechanism for dispatching to a duck- > array > implementation, ``numpy.get_array_module``. ``get_array_module`` > performs the > same type resolution as ``__array_function__`` and returns a module > with an API > promised to match the standard interface of ``numpy`` that can > implement > operations on all provided array types. > > The protocol itself is both simpler and more powerful than > ``__array_function__``, because it doesn't need to worry about > actually > implementing functions. We believe it resolves most of the > maintainability and > functionality limitations of ``__array_function__``. > > The new protocol is opt-in, explicit and with local control; see > :ref:`appendix-design-choices` for discussion on the importance of > these design > features. > > The array module contract > ========================= > > Modules returned by ``get_array_module``/``__array_module__`` should > make a > best effort to implement NumPy's core functionality on new array > types(s). > Unimplemented functionality should simply be omitted (e.g., accessing > an > unimplemented function should raise ``AttributeError``). In the > future, we > anticipate codifying a protocol for requesting restricted subsets of > ``numpy``; > see :ref:`requesting-restricted-subsets` for more details. > > How to use ``get_array_module`` > =============================== > > Code that wants to support generic duck arrays should explicitly call > ``get_array_module`` to determine an appropriate array module from > which to > call functions, rather than using the ``numpy`` namespace directly. > For > example: > > .. code:: python > > # calls the appropriate version of np.something for x and y > module = np.get_array_module(x, y) > module.something(x, y) > > Both array creation and array conversion are supported, because > dispatching is > handled by ``get_array_module`` rather than via the types of function > arguments. For example, to use random number generation functions or > methods, > we can simply pull out the appropriate submodule: > > .. code:: python > > def duckarray_add_random(array): > module = np.get_array_module(array) > noise = module.random.randn(*array.shape) > return array + noise > > We can also write the duck-array ``stack`` function from `NEP 30 > `_, without > the need > for a new ``np.duckarray`` function: > > .. code:: python > > def duckarray_stack(arrays): > module = np.get_array_module(*arrays) > arrays = [module.asarray(arr) for arr in arrays] > shapes = {arr.shape for arr in arrays} > if len(shapes) != 1: > raise ValueError('all input arrays must have the same > shape') > expanded_arrays = [arr[module.newaxis, ...] for arr in > arrays] > return module.concatenate(expanded_arrays, axis=0) > > By default, ``get_array_module`` will return the ``numpy`` module if > no > arguments are arrays. This fall-back can be explicitly controlled by > providing > the ``module`` keyword-only argument. It is also possible to indicate > that an > exception should be raised instead of returning a default array > module by > setting ``module=None``. > > How to implement ``__array_module__`` > ===================================== > > Libraries implementing a duck array type that want to support > ``get_array_module`` need to implement the corresponding protocol, > ``__array_module__``. This new protocol is based on Python's dispatch > protocol > for arithmetic, and is essentially a simpler version of > ``__array_function__``. > > Only one argument is passed into ``__array_module__``, a Python > collection of > unique array types passed into ``get_array_module``, i.e., all > arguments with > an ``__array_module__`` attribute. > > The special method should either return an namespace with an API > matching > ``numpy``, or ``NotImplemented``, indicating that it does not know > how to > handle the operation: > > .. code:: python > > class MyArray: > def __array_module__(self, types): > if not all(issubclass(t, MyArray) for t in types): > return NotImplemented > return my_array_module > > Returning custom objects from ``__array_module__`` > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > ``my_array_module`` will typically, but need not always, be a Python > module. > Returning a custom objects (e.g., with functions implemented via > ``__getattr__``) may be useful for some advanced use cases. > > For example, custom objects could allow for partial implementations > of duck > array modules that fall-back to NumPy (although this is not > recommended in > general because such fall-back behavior can be error prone): > > .. code:: python > > class MyArray: > def __array_module__(self, types): > if all(issubclass(t, MyArray) for t in types): > return ArrayModule() > else: > return NotImplemented > > class ArrayModule: > def __getattr__(self, name): > import base_module > return getattr(base_module, name, getattr(numpy, name)) > > Subclassing from ``numpy.ndarray`` > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > All of the same guidance about well-defined type casting hierarchies > from > NEP-18 still applies. ``numpy.ndarray`` itself contains a matching > implementation of ``__array_module__``, which is convenient for > subclasses: > > .. code:: python > > class ndarray: > def __array_module__(self, types): > if all(issubclass(t, ndarray) for t in types): > return numpy > else: > return NotImplemented > > NumPy's internal machinery > ========================== > > The type resolution rules of ``get_array_module`` follow the same > model as > Python and NumPy's existing dispatch protocols: subclasses are called > before > super-classes, and otherwise left to right. ``__array_module__`` is > guaranteed > to be called only a single time on each unique type. > > The actual implementation of `get_array_module` will be in C, but > should be > equivalent to this Python code: > > .. code:: python > > def get_array_module(*arrays, default=numpy): > implementing_arrays, types = > _implementing_arrays_and_types(arrays) > if not implementing_arrays and default is not None: > return default > for array in implementing_arrays: > module = array.__array_module__(types) > if module is not NotImplemented: > return module > raise TypeError("no common array module found") > > def _implementing_arrays_and_types(relevant_arrays): > types = [] > implementing_arrays = [] > for array in relevant_arrays: > t = type(array) > if t not in types and hasattr(t, '__array_module__'): > types.append(t) > # Subclasses before superclasses, otherwise left to > right > index = len(implementing_arrays) > for i, old_array in enumerate(implementing_arrays): > if issubclass(t, type(old_array)): > index = i > break > implementing_arrays.insert(index, array) > return implementing_arrays, types > > Relationship with ``__array_ufunc__`` and ``__array_function__`` > ---------------------------------------------------------------- > > These older protocols have distinct use-cases and should remain > =============================================================== > > ``__array_module__`` is intended to resolve limitations of > ``__array_function__``, so it is natural to consider whether it could > entirely > replace ``__array_function__``. This would offer dual benefits: (1) > simplifying > the user-story about how to override NumPy and (2) removing the > slowdown > associated with checking for dispatch when calling every NumPy > function. > > However, ``__array_module__`` and ``__array_function__`` are pretty > different > from a user perspective: it requires explicit calls to > ``get_array_function``, > rather than simply reusing original ``numpy`` functions. This is > probably fine > for *libraries* that rely on duck-arrays, but may be frustratingly > verbose for > interactive use. > > Some of the dispatching use-cases for ``__array_ufunc__`` are also > solved by > ``__array_module__``, but not all of them. For example, it is still > useful to > be able to define non-NumPy ufuncs (e.g., from Numba or SciPy) in a > generic way > on non-NumPy arrays (e.g., with dask.array). > > Given their existing adoption and distinct use cases, we don't think > it makes > sense to remove or deprecate ``__array_function__`` and > ``__array_ufunc__`` at > this time. > > Mixin classes to implement ``__array_function__`` and > ``__array_ufunc__`` > ===================================================================== > ==== > > Despite the user-facing differences, ``__array_module__`` and a > module > implementing NumPy's API still contain sufficient functionality > needed to > implement dispatching with the existing duck array protocols. > > For example, the following mixin classes would provide sensible > defaults for > these special methods in terms of ``get_array_module`` and > ``__array_module__``: > > .. code:: python > > class ArrayUfuncFromModuleMixin: > > def __array_ufunc__(self, ufunc, method, *inputs, **kwargs): > arrays = inputs + kwargs.get('out', ()) > try: > array_module = np.get_array_module(*arrays) > except TypeError: > return NotImplemented > > try: > # Note this may have false positive matches, if > ufunc.__name__ > # matches the name of a ufunc defined by NumPy. > Unfortunately > # there is no way to determine in which module a > ufunc was > # defined. > new_ufunc = getattr(array_module, ufunc.__name__) > except AttributeError: > return NotImplemented > > try: > callable = getattr(new_ufunc, method) > except AttributeError: > return NotImplemented > > return callable(*inputs, **kwargs) > > class ArrayFunctionFromModuleMixin: > > def __array_function__(self, func, types, args, kwargs): > array_module = self.__array_module__(types) > if array_module is NotImplemented: > return NotImplemented > > # Traverse submodules to find the appropriate function > modules = func.__module__.split('.') > assert modules[0] == 'numpy' > for submodule in modules[1:]: > module = getattr(module, submodule, None) > new_func = getattr(module, func.__name__, None) > if new_func is None: > return NotImplemented > > return new_func(*args, **kwargs) > > To make it easier to write duck arrays, we could also add these mixin > classes > into ``numpy.lib.mixins`` (but the examples above may suffice). > > Alternatives considered > ----------------------- > > Naming > ====== > > We like the name ``__array_module__`` because it mirrors the existing > ``__array_function__`` and ``__array_ufunc__`` protocols. Another > reasonable > choice could be ``__array_namespace__``. > > It is less clear what the NumPy function that calls this protocol > should be > called (``get_array_module`` in this proposal). Some possible > alternatives: > ``array_module``, ``common_array_module``, ``resolve_array_module``, > ``get_namespace``, ``get_numpy``, ``get_numpylike_module``, > ``get_duck_array_module``. > > .. _requesting-restricted-subsets: > > Requesting restricted subsets of NumPy's API > ============================================ > > Over time, NumPy has accumulated a very large API surface, with over > 600 > attributes in the top level ``numpy`` module alone. It is unlikely > that any > duck array library could or would want to implement all of these > functions and > classes, because the frequently used subset of NumPy is much smaller. > > We think it would be useful exercise to define "minimal" subset(s) of > NumPy's > API, omitting rarely used or non-recommended functionality. For > example, > minimal NumPy might include ``stack``, but not the other stacking > functions > ``column_stack``, ``dstack``, ``hstack`` and ``vstack``. This could > clearly > indicate to duck array authors and users want functionality is core > and what > functionality they can skip. > > Support for requesting a restricted subset of NumPy's API would be a > natural > feature to include in ``get_array_function`` and > ``__array_module__``, e.g., > > .. code:: python > > # array_module is only guaranteed to contain "minimal" NumPy > array_module = np.get_array_module(*arrays, request='minimal') > > To facilitate testing with NumPy and use with any valid duck array > library, > NumPy itself would return restricted versions of the ``numpy`` module > when > ``get_array_module`` is called only on NumPy arrays. Omitted > functions would > simply not exist. > > Unfortunately, we have not yet figured out what these restricted > subsets should > be, so it doesn't make sense to do this yet. When/if we do, we could > either add > new keyword arguments to ``get_array_module`` or add new top level > functions, > e.g., ``get_minimal_array_module``. We would also need to add either > a new > protocol patterned off of ``__array_module__`` (e.g., > ``__array_module_minimal__``), or could add an optional second > argument to > ``__array_module__`` (catching errors with ``try``/``except``). > > A new namespace for implicit dispatch > ===================================== > > Instead of supporting overrides in the main `numpy` namespace with > ``__array_function__``, we could create a new opt-in namespace, e.g., > ``numpy.api``, with versions of NumPy functions that support > dispatching. These > overrides would need new opt-in protocols, e.g., > ``__array_function_api__`` > patterned off of ``__array_function__``. > > This would resolve the biggest limitations of ``__array_function__`` > by being > opt-in and would also allow for unambiguously overriding functions > like > ``asarray``, because ``np.api.asarray`` would always mean "convert an > array-like object." But it wouldn't solve all the dispatching needs > met by > ``__array_module__``, and would leave us with supporting a > considerably more > complex protocol both for array users and implementors. > > We could potentially implement such a new namespace *via* the > ``__array_module__`` protocol. Certainly some users would find this > convenient, > because it is slightly less boilerplate. But this would leave users > with a > confusing choice: when should they use `get_array_module` vs. > `np.api.something`. Also, we would have to add and maintain a whole > new module, > which is considerably more expensive than merely adding a function. > > Dispatching on both types and arrays instead of only types > ========================================================== > > Instead of supporting dispatch only via unique array types, we could > also > support dispatch via array objects, e.g., by passing an ``arrays`` > argument as > part of the ``__array_module__`` protocol. This could potentially be > useful for > dispatch for arrays with metadata, such provided by Dask and Pint, > but would > impose costs in terms of type safety and complexity. > > For example, a library that supports arrays on both CPUs and GPUs > might decide > on which device to create a new arrays from functions like ``ones`` > based on > input arguments: > > .. code:: python > > class Array: > def __array_module__(self, types, arrays): > useful_arrays = tuple(a in arrays if isinstance(a, > Array)) > if not useful_arrays: > return NotImplemented > prefer_gpu = any(a.prefer_gpu for a in useful_arrays) > return ArrayModule(prefer_gpu) > > class ArrayModule: > def __init__(self, prefer_gpu): > self.prefer_gpu = prefer_gpu > > def __getattr__(self, name): > import base_module > base_func = getattr(base_module, name) > return functools.partial(base_func, > prefer_gpu=self.prefer_gpu) > > This might be useful, but it's not clear if we really need it. Pint > seems to > get along OK without any explicit array creation routines (favoring > multiplication by units, e.g., ``np.ones(5) * ureg.m``), and for the > most part > Dask is also OK with existing ``__array_function__`` style overides > (e.g., > favoring ``np.ones_like`` over ``np.ones``). Choosing whether to > place an array > on the CPU or GPU could be solved by `making array creation lazy > `_. > > .. _appendix-design-choices: > > Appendix: design choices for API overrides > ------------------------------------------ > > There is a large range of possible design choices for overriding > NumPy's API. > Here we discuss three major axes of the design decision that guided > our design > for ``__array_module__``. > > Opt-in vs. opt-out for users > ============================ > > The ``__array_ufunc__`` and ``__array_function__`` protocols provide > a > mechanism for overriding NumPy functions *within NumPy's existing > namespace*. > This means that users need to explicitly opt-out if they do not want > any > overridden behavior, e.g., by casting arrays with ``np.asarray()``. > > In theory, this approach lowers the barrier for adopting these > protocols in > user code and libraries, because code that uses the standard NumPy > namespace is > automatically compatible. But in practice, this hasn't worked out. > For example, > most well-maintained libraries that use NumPy follow the best > practice of > casting all inputs with ``np.asarray()``, which they would have to > explicitly > relax to use ``__array_function__``. Our experience has been that > making a > library compatible with a new duck array type typically requires at > least a > small amount of work to accommodate differences in the data model and > operations > that can be implemented efficiently. > > These opt-out approaches also considerably complicate backwards > compatibility > for libraries that adopt these protocols, because by opting in as a > library > they also opt-in their users, whether they expect it or not. For > winning over > libraries that have been unable to adopt ``__array_function__``, an > opt-in > approach seems like a must. > > Explicit vs. implicit choice of implementation > ============================================== > > Both ``__array_ufunc__`` and ``__array_function__`` have implicit > control over > dispatching: the dispatched functions are determined via the > appropriate > protocols in every function call. This generalizes well to handling > many > different types of objects, as evidenced by its use for implementing > arithmetic > operators in Python, but it has two downsides: > > 1. *Speed*: it imposes additional overhead in every function call, > because each > function call needs to inspect each of its arguments for > overrides. This is > why arithmetic on builtin Python numbers is slow. > 2. *Readability*: it is not longer immediately evident to readers of > code what > happens when a function is called, because the function's > implementation > could be overridden by any of its arguments. > > In contrast, importing a new library (e.g., ``import dask.array as > da``) with > an API matching NumPy is entirely explicit. There is no overhead from > dispatch > or ambiguity about which implementation is being used. > > Explicit and implicit choice of implementations are not mutually > exclusive > options. Indeed, most implementations of NumPy API overrides via > ``__array_function__`` that we are familiar with (namely, dask, CuPy > and > sparse, but not Pint) also include an explicit way to use their > version of > NumPy's API by importing a module directly (``dask.array``, ``cupy`` > or > ``sparse``, respectively). > > Local vs. non-local vs. global control > ====================================== > > The final design axis is how users control the choice of API: > > - **Local control**, as exemplified by multiple dispatch and Python > protocols for > arithmetic, determines which implementation to use either by > checking types > or calling methods on the direct arguments of a function. > - **Non-local control** such as `np.errstate > < > https://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html>`_ > overrides behavior with global-state via function decorators or > context-managers. Control is determined hierarchically, via the > inner-most > context. > - **Global control** provides a mechanism for users to set default > behavior, > either via function calls or configuration files. For example, > matplotlib > allows setting a global choice of plotting backend. > > Local control is generally considered a best practice for API design, > because > control flow is entirely explicit, which makes it the easiest to > understand. > Non-local and global control are occasionally used, but generally > either due to > ignorance or a lack of better alternatives. > > In the case of duck typing for NumPy's public API, we think non-local > or global > control would be mistakes, mostly because they **don't compose > well**. If one > library sets/needs one set of overrides and then internally calls a > routine > that expects another set of overrides, the resulting behavior may be > very > surprising. Higher order functions are especially problematic, > because the > context in which functions are evaluated may not be the context in > which they > are defined. > > One class of override use cases where we think non-local and global > control are > appropriate is for choosing a backend system that is guaranteed to > have an > entirely consistent interface, such as a faster alternative > implementation of > ``numpy.fft`` on NumPy arrays. However, these are out of scope for > the current > proposal, which is focused on duck arrays. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Thu Jan 9 07:22:33 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 9 Jan 2020 13:22:33 +0100 Subject: [Numpy-discussion] Fwd: [NumFOCUS Projects] Job: Open Source Developer Advocate In-Reply-To: References: Message-ID: ---------- Forwarded message --------- From: Nicole Foster Date: Wed, Jan 8, 2020 at 10:25 PM Subject: [NumFOCUS Projects] Job: Open Source Developer Advocate To: Fiscally Sponsored Project Representatives , Affiliated Projects Hi Everyone, We recently posted an ad for an Open Source Developer Advocate position with NumFOCUS. We would be thrilled to have a member of our community work with us here at NumFOCUS so we ask that you please share this job posting with anyone you know that might be a good fit. Please feel free to reach out if you have any questions. Thank you for your time and consideration as we work to fill this role. Best, -- Nicole Foster Executive Operations Administrator, NumFOCUS nicole at numfocus.org 512-831-2870 -- You received this message because you are subscribed to the Google Groups "Fiscally Sponsored Project Representatives" group. To unsubscribe from this group and stop receiving emails from it, send an email to projects+unsubscribe at numfocus.org. To view this discussion on the web visit https://groups.google.com/a/numfocus.org/d/msgid/projects/CAJLwxPF5ezbsAEWk4pqWPXk6F7SAU8cESm6GrXpy1ak6%3DBzt6w%40mail.gmail.com . -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokul at astrome.co Thu Jan 9 09:03:32 2020 From: gokul at astrome.co (pnp) Date: Thu, 9 Jan 2020 07:03:32 -0700 (MST) Subject: [Numpy-discussion] Numpy support for Microblaze Message-ID: <1578578612004-0.post@n7.nabble.com> Are there some steps to add support to a new cpu such as microblaze? Regards. -- Sent from: http://numpy-discussion.10968.n7.nabble.com/ From matti.picus at gmail.com Thu Jan 9 17:34:19 2020 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 10 Jan 2020 06:34:19 +0800 Subject: [Numpy-discussion] Numpy support for Microblaze In-Reply-To: <1578578612004-0.post@n7.nabble.com> References: <1578578612004-0.post@n7.nabble.com> Message-ID: <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> On 9/1/20 10:03 pm, pnp wrote: > Are there some steps to add support to a new cpu such as microblaze? > > Regards. > > > > -- > Sent from: http://numpy-discussion.10968.n7.nabble.com/ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion I don't think we have a documented procedure for adding a new CPU architecture: it is not that common. The microblaze is a soft core built in HDL, targeting an FPGA. I would try to break the question into three parts: - Does the python support for the microblaze implement any the Python C-API? - Does the C compiler for microblase support enough of the c99 language so that NumPy can be synthesized? and once NumPy is running: - Are there acceleration libraries that could supply a BLAS-like interface? My approach would be to "make it run" then "make it fast", but I doubt the high-level (c) tools are up to the task of sythesizing the NumPy C code to HDL without alot of work. It might be better to approach this task in a different direction: take one of the NumPy-like projects (CuPy might be a good starting point) and rewrite the C code parts in a way that can be synthesized. Matti From fbastien at nvidia.com Fri Jan 10 08:32:54 2020 From: fbastien at nvidia.com (Frederic Bastien) Date: Fri, 10 Jan 2020 13:32:54 +0000 Subject: [Numpy-discussion] Numpy support for Microblaze In-Reply-To: <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> References: <1578578612004-0.post@n7.nabble.com> <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> Message-ID: I like your breakdown. But the fact that it is a softcore is not related. We do not need to compile it to an HDL. We need it to be compiled to the binary format that the Microblaze support. Now, after one full version is working as a software, it is possible to optimize part of NumPy by having those part synthesized to hardware. But from my understanding, it wasn't the question. This would be a bigger effort and I suppose there is no guide from NumPy on how to do this. Frederic -----Original Message----- From: NumPy-Discussion On Behalf Of Matti Picus Sent: Thursday, January 9, 2020 5:34 PM To: numpy-discussion at python.org Subject: Re: [Numpy-discussion] Numpy support for Microblaze External email: Use caution opening links or attachments On 9/1/20 10:03 pm, pnp wrote: > Are there some steps to add support to a new cpu such as microblaze? > > Regards. > > > > -- > Sent from: http://numpy-discussion.10968.n7.nabble.com/ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion I don't think we have a documented procedure for adding a new CPU architecture: it is not that common. The microblaze is a soft core built in HDL, targeting an FPGA. I would try to break the question into three parts: - Does the python support for the microblaze implement any the Python C-API? - Does the C compiler for microblase support enough of the c99 language so that NumPy can be synthesized? and once NumPy is running: - Are there acceleration libraries that could supply a BLAS-like interface? My approach would be to "make it run" then "make it fast", but I doubt the high-level (c) tools are up to the task of sythesizing the NumPy C code to HDL without alot of work. It might be better to approach this task in a different direction: take one of the NumPy-like projects (CuPy might be a good starting point) and rewrite the C code parts in a way that can be synthesized. Matti _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ----------------------------------------------------------------------------------- From robert.kern at gmail.com Fri Jan 10 12:35:46 2020 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Jan 2020 12:35:46 -0500 Subject: [Numpy-discussion] Numpy support for Microblaze In-Reply-To: References: <1578578612004-0.post@n7.nabble.com> <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> Message-ID: On Fri, Jan 10, 2020 at 8:34 AM Frederic Bastien wrote: > I like your breakdown. But the fact that it is a softcore is not related. > We do not need to compile it to an HDL. > We need it to be compiled to the binary format that the Microblaze support. > > Now, after one full version is working as a software, it is possible to > optimize part of NumPy by having those part synthesized to hardware. > But from my understanding, it wasn't the question. This would be a bigger > effort and I suppose there is no guide from NumPy on how to do this. > Indeed, there is no particular guide. Porting Python first will likely tell you a whole lot about the general process. You can search in the numpy source for "NPY_CPU" (and "PPC" and "ARM") to see where we have applied architecture-specific customizations. I suspect that you will have more trouble dealing with the OS (or lack of OS, if that's the case) than the architecture itself. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokul at astrome.co Sat Jan 11 01:29:35 2020 From: gokul at astrome.co (pnp) Date: Fri, 10 Jan 2020 23:29:35 -0700 (MST) Subject: [Numpy-discussion] Numpy support for Microblaze In-Reply-To: References: <1578578612004-0.post@n7.nabble.com> <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> Message-ID: <1578724175031-0.post@n7.nabble.com> First of, many thanks to each one of you for taking your time to post your valuable comments and suggestions. My apologies for not filling in the details. Here goes: 1.We have linux running on the Microblaze. So, what I am looking at this point is a software only solution. 2. We have cross compilation tools and buildroot. I was able to compile python3.8 externally and run it on Microblaze. Note that I compiled python outside of buildroot since the version of python built from buildroot did not work. 3. Buildroot did not have support for Numpy. So I added it to the configuration. To resolve the compilation issues, here is what I did: diff -r -u /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/numpy/npy_cpu.h python-numpy-1.16.4.BAK/numpy/core/include/numpy/npy_cpu.h --- /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/numpy/npy_cpu.h 2019-02-22 06:03:41.000000000 +0530 +++ python-numpy-1.16.4.BAK/numpy/core/include/numpy/npy_cpu.h 2019-12-22 18:52:16.468085099 +0530 @@ -102,6 +102,8 @@ #define NPY_CPU_ARCEB #elif defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64 #define NPY_CPU_RISCV64 +#elif defined(__MICROBLAZEEL__) + #define NPY_CPU_MICROBLAZEEL #else #error Unknown CPU, please report this to numpy maintainers with \ information about your platform (OS, CPU and compiler) diff -r -u /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/numpy/npy_endian.h python-numpy-1.16.4.BAK/numpy/core/include/numpy/npy_endian.h --- /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/numpy/npy_endian.h 2019-02-22 06:03:41.000000000 +0530 +++ python-numpy-1.16.4.BAK/numpy/core/include/numpy/npy_endian.h 2019-12-22 18:54:08.912859083 +0530 @@ -48,7 +48,8 @@ || defined(NPY_CPU_MIPSEL) \ || defined(NPY_CPU_PPC64LE) \ || defined(NPY_CPU_ARCEL) \ - || defined(NPY_CPU_RISCV64) + || defined(NPY_CPU_RISCV64) \ + || defined(NPY_CPU_MICROBLAZEEL) #define NPY_BYTE_ORDER NPY_LITTLE_ENDIAN #elif defined(NPY_CPU_PPC) \ || defined(NPY_CPU_SPARC) \ diff -ruN -x '*.py' /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/microblaze-fenv.h python-numpy-1.16.4/numpy/core/include/microblaze-fenv.h --- /home/gokul/numpyExpt/numpy-1.16.4/numpy/core/include/microblaze-fenv.h 1970-01-01 05:30:00.000000000 +0530 +++ python-numpy-1.16.4/numpy/core/include/microblaze-fenv.h 2020-01-02 14:06:30.410908275 +0530 @@ -0,0 +1,36 @@ +#ifndef MICROBLAZE_FENV_H +#define MICROBLAZE_FENV_H 1 + +enum + { + FE_INEXACT = +#define FE_INEXACT (0x00) + FE_INEXACT, + FE_UNDERFLOW = +#define FE_UNDERFLOW (0x00) + FE_UNDERFLOW, + FE_OVERFLOW = +#define FE_OVERFLOW (0x00) + FE_OVERFLOW, + FE_DIVBYZERO = +#define FE_DIVBYZERO (0x00) + FE_DIVBYZERO, + FE_INVALID = +#define FE_INVALID (0x00) + FE_INVALID + }; + +enum + { + FE_TOWARDZERO = +#define FE_TOWARDZERO (0x0) + FE_TOWARDZERO, + FE_DOWNWARD = +#define FE_DOWNWARD (0x0) + FE_DOWNWARD, + FE_UPWARD = +#define FE_UPWARD (0x0) + FE_UPWARD + }; + +#endif The last change above was to take care of floating point exception errors in compilation. Not sure if this is the right approach- I had to do this since Microblaze didnt support all these exceptions 4. When I run with these changes, I get the following runtime error: ~# /usr/local/bin/python Python 3.8.0 (default, Jan 3 2020, 16:10:54) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 150, in from . import random File "/usr/lib/python3.8/site-packages/numpy/random/__init__.py", line 143, in from .mtrand import * SystemError: execution of module numpy.random.mtrand failed without setting an exception >>> -- Sent from: http://numpy-discussion.10968.n7.nabble.com/ From robert.kern at gmail.com Sun Jan 12 01:48:08 2020 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 12 Jan 2020 01:48:08 -0500 Subject: [Numpy-discussion] Numpy support for Microblaze In-Reply-To: <1578724175031-0.post@n7.nabble.com> References: <1578578612004-0.post@n7.nabble.com> <565cb22a-e22d-5c9f-b535-cded2d923d43@gmail.com> <1578724175031-0.post@n7.nabble.com> Message-ID: On Sat, Jan 11, 2020 at 1:30 AM pnp wrote: > 4. When I run with these changes, I get the following runtime error: > ~# /usr/local/bin/python > > Python 3.8.0 (default, Jan 3 2020, 16:10:54) > [GCC 8.2.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 150, in > > from . import random > File "/usr/lib/python3.8/site-packages/numpy/random/__init__.py", line > 143, in > from .mtrand import * > SystemError: execution of module numpy.random.mtrand failed without setting > an exception > >>> > I'd suggest trying to build a small Cython module by itself to see if this is a platform-specific problem with the general Cython-generated code for initializing the module or a problem that we can fix in the mtrand code. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From magdalena.proszewska at gmail.com Mon Jan 13 12:08:16 2020 From: magdalena.proszewska at gmail.com (mpro) Date: Mon, 13 Jan 2020 10:08:16 -0700 (MST) Subject: [Numpy-discussion] Accuracy improvement in np.mean Message-ID: <1578935296208-0.post@n7.nabble.com> Hi, 18 days ago I submitted pull request that improves accuracy of mean function by calculating mean sequentially along certain axis. Happy to get some feedback. Magdalena -- Sent from: http://numpy-discussion.10968.n7.nabble.com/ From p.j.a.cock at googlemail.com Mon Jan 13 12:32:30 2020 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 13 Jan 2020 17:32:30 +0000 Subject: [Numpy-discussion] Accuracy improvement in np.mean In-Reply-To: <1578935296208-0.post@n7.nabble.com> References: <1578935296208-0.post@n7.nabble.com> Message-ID: To save anyone else looking, it was pull request 15185: https://github.com/numpy/numpy/pull/15185 Peter On Mon, Jan 13, 2020 at 5:25 PM mpro wrote: > > Hi, > > 18 days ago I submitted pull request that improves accuracy of mean function > by calculating mean sequentially along certain axis. > Happy to get some feedback. > > Magdalena > > > > -- > Sent from: http://numpy-discussion.10968.n7.nabble.com/ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From michael.lance at gmail.com Thu Jan 16 21:46:17 2020 From: michael.lance at gmail.com (Michael Lance) Date: Thu, 16 Jan 2020 21:46:17 -0500 Subject: [Numpy-discussion] 8 Real Distributions Message-ID: TLDR; I think this could be a useful contribution to NumPy, but I want to get feedback on where it should go (either in NumPy or elsewhere). I have functions using numpy.random which invoke the 8 "Real" data sets as estimated by Ted Micceri in 1989. These can be useful in Monte Carlo simulations. Background info: Parametric inferential statistics generally assume normal distributions (though kurtosis presents less of an issue than skew). However, in "nature", distributions are often not normal. In 1989, Ted Micceri's study ( http://psycnet.apa.org/record/1989-14214-001) on real data sets resulted in the estimation of 8 "Real" distributions. Using these distributions in simulations help to produce more realistic types I and II error rate and power estimates, particularly for smaller samples. A similar module is currently available in Fortran called realpops. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Fri Jan 17 05:39:24 2020 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Fri, 17 Jan 2020 10:39:24 +0000 Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols Message-ID: Today, numpy has a np.ma.mask_rowcols function, which stretches masks along the full length of an axis. For example, given the matrix:: >>> a2d = np.zeros((3, 3), dtype=int) >>> a2d[1, 1] = 1 >>> a2d = np.ma.masked_equal(a2d, 1) >>> print(a2d) [[0 0 0] [0 -- 0] [0 0 0]] The API allows:: >>> print(np.ma.mask_rowcols(a2d, axis=0)) [[0 0 0] [-- -- --] [0 0 0]] >>> print(np.ma.mask_rowcols(a2d, axis=1)) [[0 -- 0] [0 -- 0] [0 -- 0]] >>> print(np.ma.mask_rowcols(a2d, axis=None)) [[0 -- 0] [-- -- --] [0 -- 0]] However, this function only works for 2D arrays. It would be useful to generalize this to work on ND arrays as well. Unfortunately, the current function is messy to generalize, because axis=0 means ?spread the mask along axis 1?, and vice versa. Additionally, the name is not particularly good for an ND function. My proposal in PR 14998 is to introduce a new function, mask_extend_axis, which fixes this shortcoming. Given an 3D array:: >>> a3d = np.zeros((2, 2, 2), dtype=int) >>> a3d[0, 0, 0] = 1 >>> a3d = np.ma.masked_equal(a3d, 1) >>> print(a3d) [[[-- 0] [0 0]] [[0 0] [0 0]]] This, in my opinion, has clearer axis semantics: >>> print(np.ma.mask_extend_axis(a2d, axis=0)) [[[-- 0] [0 0]] [[-- 0] [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=1)) [[[-- 0] [-- 0]] [[0 0] [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=2)) [[[-- --] [0 0]] [[0 0] [0 0]]] Stretching over multiple axes remains possible: >>> print(np.ma.mask_extend_axis(a2d, axis=(1, 2))) [[[-- --] [-- 0]] [[0 0] [0 0]]] # extending sequentially is not the same as extending in parallel >>> print(np.ma.mask_extend_axis(np.ma.mask_extend_axis(a2d, axis=1), axis=2)) [[[-- --] [-- --]] [[0 0] [0 0]]] Questions for the mailing list then: - Can you think of a better name than mask_extend_axis? - Does my proposed meaning of axis make more sense to you than the one used by mask_rowcols? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hameerabbasi at yahoo.com Fri Jan 17 10:26:00 2020 From: hameerabbasi at yahoo.com (Hameer Abbasi) Date: Fri, 17 Jan 2020 16:26:00 +0100 Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols In-Reply-To: References: Message-ID: <6FC5458C-6011-473E-A8B6-27832583FDDE@yahoo.com> IMHO, masked arrays and extending masks like that is a weird API. I would prefer a more functional approach: Where we take in an input 1-D or N-D boolean array in addition to a masked array with multiple axes over which to extend the mask. From: NumPy-Discussion on behalf of Eric Wieser Reply to: Discussion of Numerical Python Date: Friday, 17. January 2020 at 11:41 To: Discussion of Numerical Python Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols Today, numpy has a np.ma.mask_rowcols function, which stretches masks along the full length of an axis. For example, given the matrix:: >>> a2d = np.zeros((3, 3), dtype=int) >>> a2d[1, 1] = 1 >>> a2d = np.ma.masked_equal(a2d, 1) >>> print(a2d) [[0 0 0] [0 -- 0] [0 0 0]] The API allows:: >>> print(np.ma.mask_rowcols(a2d, axis=0)) [[0 0 0] [-- -- --] [0 0 0]] >>> print(np.ma.mask_rowcols(a2d, axis=1)) [[0 -- 0] [0 -- 0] [0 -- 0]] >>> print(np.ma.mask_rowcols(a2d, axis=None)) [[0 -- 0] [-- -- --] [0 -- 0]] However, this function only works for 2D arrays. It would be useful to generalize this to work on ND arrays as well. Unfortunately, the current function is messy to generalize, because axis=0 means ?spread the mask along axis 1?, and vice versa. Additionally, the name is not particularly good for an ND function. My proposal in PR 14998 is to introduce a new function, mask_extend_axis, which fixes this shortcoming. Given an 3D array:: >>> a3d = np.zeros((2, 2, 2), dtype=int) >>> a3d[0, 0, 0] = 1 >>> a3d = np.ma.masked_equal(a3d, 1) >>> print(a3d) [[[-- 0] ? [0 0]] [[0 0] ? [0 0]]] This, in my opinion, has clearer axis semantics: >>> print(np.ma.mask_extend_axis(a2d, axis=0)) [[[-- 0] ? [0 0]] [[-- 0] ? [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=1)) [[[-- 0] ? [-- 0]] [[0 0] ? [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=2)) [[[-- --] ? [0 0]] [[0 0] ? [0 0]]] Stretching over multiple axes remains possible: >>> print(np.ma.mask_extend_axis(a2d, axis=(1, 2))) [[[-- --] ? [-- 0]] [[0 0] ? [0 0]]] # extending sequentially is not the same as extending in parallel >>> print(np.ma.mask_extend_axis(np.ma.mask_extend_axis(a2d, axis=1), axis=2)) [[[-- --] ? [-- --]] [[0 0] ? [0 0]]] Questions for the mailing list then: ? Can you think of a better name than mask_extend_axis? ? Does my proposed meaning of axis make more sense to you than the one used by mask_rowcols? _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 17 10:27:48 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 17 Jan 2020 09:27:48 -0600 Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols In-Reply-To: References: Message-ID: <9eb3cbe63c6edf3a28e57fe407303e43f0a09d98.camel@sipsolutions.net> On Fri, 2020-01-17 at 10:39 +0000, Eric Wieser wrote: > Today, numpy has a np.ma.mask_rowcols function, which stretches masks > along > the full length of an axis. For example, given the matrix:: > > Questions for the mailing list then: > The additional question: I think I am good with adding a new name if we cannot reasonably reuse the old ones. > Can you think of a better name than mask_extend_axis? Doubt it is good, but to put it there: extend_mask_along_axis "along" shows up 2-3 times, although "along" really is the default for most things in NumPy. Tried thesaurus for "extend", the main other word seemed "spread" (but it is very different from the current choice). > Does my proposed meaning of axis make more sense to you than the one > used by mask_rowcols? It does to me (although I hardly ever use masked arrays). The `axis` argument usually denotes the axis being operated on/along. - Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From einstein.edison at gmail.com Fri Jan 17 10:29:46 2020 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 17 Jan 2020 15:29:46 +0000 Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols In-Reply-To: References: Message-ID: IMHO, masked arrays and extending masks like that is a weird API. I would prefer a more functional approach: Where we take in an input 1-D or N-D boolean array in addition to a masked array with multiple axes over which to extend the mask. From: NumPy-Discussion on behalf of Eric Wieser Reply to: Discussion of Numerical Python Date: Friday, 17. January 2020 at 11:40 To: Discussion of Numerical Python Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols Today, numpy has a np.ma.mask_rowcols function, which stretches masks along the full length of an axis. For example, given the matrix:: >>> a2d = np.zeros((3, 3), dtype=int) >>> a2d[1, 1] = 1 >>> a2d = np.ma.masked_equal(a2d, 1) >>> print(a2d) [[0 0 0] [0 -- 0] [0 0 0]] The API allows:: >>> print(np.ma.mask_rowcols(a2d, axis=0)) [[0 0 0] [-- -- --] [0 0 0]] >>> print(np.ma.mask_rowcols(a2d, axis=1)) [[0 -- 0] [0 -- 0] [0 -- 0]] >>> print(np.ma.mask_rowcols(a2d, axis=None)) [[0 -- 0] [-- -- --] [0 -- 0]] However, this function only works for 2D arrays. It would be useful to generalize this to work on ND arrays as well. Unfortunately, the current function is messy to generalize, because axis=0 means ?spread the mask along axis 1?, and vice versa. Additionally, the name is not particularly good for an ND function. My proposal in PR 14998 is to introduce a new function, mask_extend_axis, which fixes this shortcoming. Given an 3D array:: >>> a3d = np.zeros((2, 2, 2), dtype=int) >>> a3d[0, 0, 0] = 1 >>> a3d = np.ma.masked_equal(a3d, 1) >>> print(a3d) [[[-- 0] [0 0]] [[0 0] [0 0]]] This, in my opinion, has clearer axis semantics: >>> print(np.ma.mask_extend_axis(a2d, axis=0)) [[[-- 0] [0 0]] [[-- 0] [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=1)) [[[-- 0] [-- 0]] [[0 0] [0 0]]] >>> print(np.ma.mask_extend_axis(a2d, axis=2)) [[[-- --] [0 0]] [[0 0] [0 0]]] Stretching over multiple axes remains possible: >>> print(np.ma.mask_extend_axis(a2d, axis=(1, 2))) [[[-- --] [-- 0]] [[0 0] [0 0]]] # extending sequentially is not the same as extending in parallel >>> print(np.ma.mask_extend_axis(np.ma.mask_extend_axis(a2d, axis=1), axis=2)) [[[-- --] [-- --]] [[0 0] [0 0]]] Questions for the mailing list then: ? Can you think of a better name than mask_extend_axis? ? Does my proposed meaning of axis make more sense to you than the one used by mask_rowcols? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Fri Jan 17 11:07:55 2020 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Fri, 17 Jan 2020 16:07:55 +0000 Subject: [Numpy-discussion] Adding an nd generalization of np.ma.mask_rowscols In-Reply-To: References: Message-ID: IMHO, masked arrays and extending masks like that is a weird API. To give some context, I needed this nd generalization internally in order to fix the issue on these lines . I would prefer a more functional approach Can you elaborate on that with an example of the API you?d prefer instead of np.ma.mask_extend_axis(a, axis=(0, 1))? On Fri, 17 Jan 2020 at 15:30, Hameer Abbasi wrote: > IMHO, masked arrays and extending masks like that is a weird API. I would > prefer a more functional approach: Where we take in an input 1-D or N-D > boolean array in addition to a masked array with multiple axes over which > to extend the mask. > > > > *From: *NumPy-Discussion gmail.com at python.org> on behalf of Eric Wieser < > wieser.eric+numpy at gmail.com> > *Reply to: *Discussion of Numerical Python > *Date: *Friday, 17. January 2020 at 11:40 > *To: *Discussion of Numerical Python > *Subject: *[Numpy-discussion] Adding an nd generalization of > np.ma.mask_rowscols > > > > Today, numpy has a np.ma.mask_rowcols function, which stretches masks > along > the full length of an axis. For example, given the matrix:: > > >>> a2d = np.zeros((3, 3), dtype=int) > > >>> a2d[1, 1] = 1 > > >>> a2d = np.ma.masked_equal(a2d, 1) > > >>> print(a2d) > > [[0 0 0] > > [0 -- 0] > > [0 0 0]] > > The API allows:: > > >>> print(np.ma.mask_rowcols(a2d, axis=0)) > > [[0 0 0] > > [-- -- --] > > [0 0 0]] > > > > >>> print(np.ma.mask_rowcols(a2d, axis=1)) > > [[0 -- 0] > > [0 -- 0] > > [0 -- 0]] > > > > >>> print(np.ma.mask_rowcols(a2d, axis=None)) > > [[0 -- 0] > > [-- -- --] > > [0 -- 0]] > > However, this function only works for 2D arrays. > It would be useful to generalize this to work on ND arrays as well. > > Unfortunately, the current function is messy to generalize, because axis=0 > means ?spread the mask along axis 1?, and vice versa. Additionally, the > name is not particularly good for an ND function. > > My proposal in PR 14998 is to > introduce a new function, mask_extend_axis, which fixes this shortcoming. > Given an 3D array:: > > >>> a3d = np.zeros((2, 2, 2), dtype=int) > > >>> a3d[0, 0, 0] = 1 > > >>> a3d = np.ma.masked_equal(a3d, 1) > > >>> print(a3d) > > [[[-- 0] > > [0 0]] > > > > [[0 0] > > [0 0]]] > > This, in my opinion, has clearer axis semantics: > > >>> print(np.ma.mask_extend_axis(a2d, axis=0)) > > [[[-- 0] > > [0 0]] > > > > [[-- 0] > > [0 0]]] > > > > >>> print(np.ma.mask_extend_axis(a2d, axis=1)) > > [[[-- 0] > > [-- 0]] > > > > [[0 0] > > [0 0]]] > > > > >>> print(np.ma.mask_extend_axis(a2d, axis=2)) > > [[[-- --] > > [0 0]] > > > > [[0 0] > > [0 0]]] > > Stretching over multiple axes remains possible: > > >>> print(np.ma.mask_extend_axis(a2d, axis=(1, 2))) > > [[[-- --] > > [-- 0]] > > > > [[0 0] > > [0 0]]] > > > > # extending sequentially is not the same as extending in parallel > > >>> print(np.ma.mask_extend_axis(np.ma.mask_extend_axis(a2d, axis=1), axis=2)) > > [[[-- --] > > [-- --]] > > > > [[0 0] > > [0 0]]] > > Questions for the mailing list then: > > ? Can you think of a better name than mask_extend_axis? > > ? Does my proposed meaning of axis make more sense to you than > the one used by mask_rowcols? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Jan 18 12:31:24 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 18 Jan 2020 18:31:24 +0100 Subject: [Numpy-discussion] 8 Real Distributions In-Reply-To: References: Message-ID: On Fri, Jan 17, 2020 at 3:46 AM Michael Lance wrote: > TLDR; > I think this could be a useful contribution to NumPy, but I want to get > feedback on where it should go (either in NumPy or elsewhere). > I have functions using numpy.random which invoke the 8 "Real" data sets as > estimated by Ted Micceri in 1989. These can be useful in Monte Carlo > simulations. > Thanks for the suggestion Michael. This seems too specialized for NumPy. Also, it's not 100% clear whether you want to add functions or data sets; NumPy doesn't want to ship any data sets. It sounds to me like these would be best in their own package. Cheers, Ralf > Background info: > > Parametric inferential statistics generally assume normal distributions > (though kurtosis presents less of an issue than skew). However, in > "nature", distributions are often not normal. In 1989, Ted Micceri's study ( > http://psycnet.apa.org/record/1989-14214-001) on real data sets resulted > in the estimation of 8 "Real" distributions. Using these distributions in > simulations help to produce more realistic types I and II error rate and > power estimates, particularly for smaller samples. > A similar module is currently available in Fortran called realpops. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From compl.yue at icloud.com Mon Jan 20 09:58:51 2020 From: compl.yue at icloud.com (YueCompl) Date: Mon, 20 Jan 2020 22:58:51 +0800 Subject: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? Message-ID: Hello folks, Myself a long time Numpy user, and recently done some Haskell stuff, Please have a look at https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate and/or http://hackage.haskell.org/package/repa doing the number crunching? Best regards, Compl -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.lance at gmail.com Mon Jan 20 10:36:03 2020 From: michael.lance at gmail.com (Michael Lance) Date: Mon, 20 Jan 2020 10:36:03 -0500 Subject: [Numpy-discussion] 8 Real Distributions In-Reply-To: References: Message-ID: Hi Ralf, These are functions that generate data sets when invoked, like Numpy already does with mathematical distributions. On Sat, Jan 18, 2020, 12:32 PM Ralf Gommers wrote: > > > On Fri, Jan 17, 2020 at 3:46 AM Michael Lance > wrote: > >> TLDR; >> I think this could be a useful contribution to NumPy, but I want to get >> feedback on where it should go (either in NumPy or elsewhere). >> I have functions using numpy.random which invoke the 8 "Real" data sets >> as estimated by Ted Micceri in 1989. These can be useful in Monte Carlo >> simulations. >> > > Thanks for the suggestion Michael. This seems too specialized for NumPy. > Also, it's not 100% clear whether you want to add functions or data sets; > NumPy doesn't want to ship any data sets. It sounds to me like these would > be best in their own package. > > Cheers, > Ralf > > >> Background info: >> >> Parametric inferential statistics generally assume normal distributions >> (though kurtosis presents less of an issue than skew). However, in >> "nature", distributions are often not normal. In 1989, Ted Micceri's study ( >> http://psycnet.apa.org/record/1989-14214-001) on real data sets resulted >> in the estimation of 8 "Real" distributions. Using these distributions in >> simulations help to produce more realistic types I and II error rate and >> power estimates, particularly for smaller samples. >> A similar module is currently available in Fortran called realpops. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jan 20 10:53:06 2020 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 21 Jan 2020 00:53:06 +0900 Subject: [Numpy-discussion] 8 Real Distributions In-Reply-To: References: Message-ID: On Tue, Jan 21, 2020 at 12:37 AM Michael Lance wrote: > Hi Ralf, > > These are functions that generate data sets when invoked, like Numpy > already does with mathematical distributions. > It's something of a judgement call. These distribution functions are defined by tables. The most efficient way to implement them in Python would be to just make them static arrays in a module that one uses `Generator.choice()` on instead of a function that generates the arrays, just to pick out one value and throw away the array again. Regardless of what you want to call them or how you want to implement them, a third-party module is the right place for them. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.lance at gmail.com Mon Jan 20 11:39:27 2020 From: michael.lance at gmail.com (Michael Lance) Date: Mon, 20 Jan 2020 11:39:27 -0500 Subject: [Numpy-discussion] 8 Real Distributions In-Reply-To: References: Message-ID: Okay, thanks! 3rd party it is. On Mon, Jan 20, 2020, 10:54 AM Robert Kern wrote: > On Tue, Jan 21, 2020 at 12:37 AM Michael Lance > wrote: > >> Hi Ralf, >> >> These are functions that generate data sets when invoked, like Numpy >> already does with mathematical distributions. >> > > It's something of a judgement call. These distribution functions are > defined by tables. The most efficient way to implement them in Python would > be to just make them static arrays in a module that one uses > `Generator.choice()` on instead of a function that generates the arrays, > just to pick out one value and throw away the array again. > > Regardless of what you want to call them or how you want to implement > them, a third-party module is the right place for them. > > -- > Robert Kern > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Tue Jan 21 00:36:49 2020 From: mikofski at berkeley.edu (Dr. Mark Alexander Mikofski PhD) Date: Mon, 20 Jan 2020 21:36:49 -0800 Subject: [Numpy-discussion] [ANN] pvlib python v0.7.1: predicting power for solar energy Message-ID: pvlib has a new minor release, v0.7.1 Release Notes: https://pvlib-python.readthedocs.io/en/v0.7.1/whatsnew.html PyPI: https://pypi.org/project/pvlib/ Read the Docs: https://pvlib-python.readthedocs.io/en/latest/ GitHub: https://github.com/pvlib/pvlib-python New examples gallery: https://pvlib-python.readthedocs.io/en/stable/auto_examples/index.html -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Jan 21 09:20:48 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Jan 2020 14:20:48 +0000 Subject: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? In-Reply-To: References: Message-ID: Hi, On Mon, Jan 20, 2020 at 3:00 PM YueCompl wrote: > > Hello folks, > > Myself a long time Numpy user, and recently done some Haskell stuff, > > Please have a look at https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, > with http://hackage.haskell.org/package/accelerate and/or http://hackage.haskell.org/package/repa doing the number crunching? Interesting - do you have any feeling for the differences in speed for Haskell accelerate array processing as compared to Numpy with default OpenBLAS? Cheers, Matthew From compl.yue at icloud.com Tue Jan 21 10:32:50 2020 From: compl.yue at icloud.com (Compl Yue) Date: Tue, 21 Jan 2020 23:32:50 +0800 Subject: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? In-Reply-To: References: Message-ID: <62654f07-ae49-f196-a76d-72707c6f20e2@icloud.com> Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively? ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 ??10:20, Matthew Brett wrote: > Hi, > > On Mon, Jan 20, 2020 at 3:00 PM YueCompl wrote: >> Hello folks, >> >> Myself a long time Numpy user, and recently done some Haskell stuff, >> >> Please have a look at https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, >> with http://hackage.haskell.org/package/accelerate and/or http://hackage.haskell.org/package/repa doing the number crunching? > Interesting - do you have any feeling for the differences in speed for > Haskell accelerate array processing as compared to Numpy with default > OpenBLAS? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From tyler.je.reddy at gmail.com Tue Jan 21 12:53:40 2020 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 21 Jan 2020 10:53:40 -0700 Subject: [Numpy-discussion] ANN: SciPy 1.2.3 (LTS) Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.2.3, which is a bug fix release. This is part of the long-term support (LTS) branch that includes Python 2.7. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.2.3 One of a few ways to install this release with pip: pip install scipy==1.2.3 ========================== SciPy 1.2.3 Release Notes ========================== SciPy 1.2.3 is a bug-fix release with no new features compared to 1.2.2. It is part of the long-term support (LTS) release series for Python 2.7. Authors ======= * Geordie McBain * Matt Haberland * David Hagen * Tyler Reddy * Pauli Virtanen * Eric Larson * Yu Feng * ananyashreyjain * Nikolay Mayorov * Evgeni Burovski * Warren Weckesser Issues closed for 1.2.3 -------------------------------- * `#4915 `__: Bug in unique_roots in scipy.signal.signaltools.py for roots with same magnitude * `#5546 `__: ValueError raised if scipy.sparse.linalg.expm recieves array larger than 200x200 * `#7117 `__: Warn users when using float32 input data to curve_fit and friends * `#7906 `__: Wrong result from scipy.interpolate.UnivariateSpline.integral for out-of-bounds * `#9581 `__: Least-squares minimization fails silently when x and y data are different types * `#9901 `__: lsoda fails to detect stiff problem when called from solve_ivp * `#9988 `__: doc build broken with Sphinx 2.0.0 * `#10303 `__: BUG: optimize: `linprog` failing TestLinprogSimplexBland::test_unbounded_below_no_presolve_corrected * `#10376 `__: TST: Travis CI fails (with pytest 5.0 ?) * `#10384 `__: CircleCI doc build failing on new warnings * `#10535 `__: TST: master branch CI failures * `#11121 `__: Calls to `scipy.interpolate.splprep` increase RAM usage. * `#11198 `__: BUG: sparse eigs (arpack) shift-invert drops the smallest eigenvalue for some k * `#11266 `__: Sparse matrix constructor data type detection changes on Numpy 1.18.0 Pull requests for 1.2.3 -------------------------------- * `#9992 `__: MAINT: Undo Sphinx pin * `#10071 `__: DOC: reconstruct SuperLU permutation matrices avoiding SparseEfficiencyWarning * `#10076 `__: BUG: optimize: fix curve_fit for mixed float32/float64 input * `#10138 `__: BUG: special: Invalid arguments to ellip_harm can crash Python. * `#10306 `__: BUG: optimize: Fix for 10303 * `#10309 `__: BUG: Pass jac=None directly to lsoda * `#10377 `__: TST, MAINT: adjustments for pytest 5.0 * `#10379 `__: BUG: sparse: set writeability to be forward-compatible with numpy>=1.17 * `#10426 `__: MAINT: Fix doc build bugs * `#10540 `__: MAINT: Fix Travis and Circle * `#10633 `__: BUG: interpolate: integral(a, b) should be zero when both limits are outside of the interpolation range * `#10833 `__: BUG: Fix subspace_angles for complex values * `#10882 `__: BUG: sparse/arpack: fix incorrect code for complex hermitian M * `#10906 `__: BUG: sparse/linalg: fix expm for np.matrix inputs * `#10961 `__: BUG: Fix signal.unique_roots * `#11126 `__: BUG: interpolate/fitpack: fix memory leak in splprep * `#11199 `__: BUG: sparse.linalg: mistake in unsymm. real shift-invert ARPACK eigenvalue selection * `#11269 `__: Fix: Sparse matrix constructor data type detection changes on Numpy 1.18.0 Checksums ========= MD5 ~~~ 702e7f68e024359d5cf0627337944520 scipy-1.2.3-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 53035df3bf9c688fb7803df11b74eb97 scipy-1.2.3-cp27-cp27m-manylinux1_i686.whl 6a07e4864228d4d6bc8d396b9e09f71c scipy-1.2.3-cp27-cp27m-manylinux1_x86_64.whl 645b4211dc2f2b3bdc4e5cf85454e164 scipy-1.2.3-cp27-cp27m-win32.whl 49e6c189ca1a0d92f426e4efaa782f37 scipy-1.2.3-cp27-cp27m-win_amd64.whl a875abf6f3f52fac916739dd556ccefb scipy-1.2.3-cp27-cp27mu-manylinux1_i686.whl 46138092ed3b9e9f0b67afb3765ca041 scipy-1.2.3-cp27-cp27mu-manylinux1_x86_64.whl 608650168cfeda8fb9c1f44ccfb9b6a7 scipy-1.2.3-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 3acbfaf35ad246e4cffd0f7e35f88483 scipy-1.2.3-cp34-cp34m-manylinux1_i686.whl 807bca9d17cf3221a062c6b72192a1ed scipy-1.2.3-cp34-cp34m-manylinux1_x86_64.whl 327a19bd65112ee59425f57cf65dce67 scipy-1.2.3-cp34-cp34m-win32.whl a1b627a5b0b1adb3c5418d3c9081615b scipy-1.2.3-cp34-cp34m-win_amd64.whl 80d65fd4266bcb29e02b03545ae80b7a scipy-1.2.3-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl bd1e8f80f9e0427855ff10810e68118f scipy-1.2.3-cp35-cp35m-manylinux1_i686.whl 35f03968403ad3049db956b90a346a59 scipy-1.2.3-cp35-cp35m-manylinux1_x86_64.whl bf489f87f8bfeb6a14c5e606b539f16a scipy-1.2.3-cp35-cp35m-win32.whl 7a618445b53f4f8671352ea52df5cc9f scipy-1.2.3-cp35-cp35m-win_amd64.whl 7b7e8889babc121b4c1340f4b8081423 scipy-1.2.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 8339ebba078ee752b2bd15eb3fc64f2f scipy-1.2.3-cp36-cp36m-manylinux1_i686.whl 4dd7be670ce0b1bd2b4c3f5038cf6fb9 scipy-1.2.3-cp36-cp36m-manylinux1_x86_64.whl 9130c940f3b811b9281ac64a3bd610a6 scipy-1.2.3-cp36-cp36m-win32.whl adeb6e4e9c270df4a5e55d0f1a4a72f0 scipy-1.2.3-cp36-cp36m-win_amd64.whl 19f6644944227c64c50dedbb04f8d91d scipy-1.2.3-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl f75450b9399acd7eb3c0fc1724de4be3 scipy-1.2.3-cp37-cp37m-manylinux1_i686.whl fc1cb1164479e070af722e579604bea2 scipy-1.2.3-cp37-cp37m-manylinux1_x86_64.whl 6c2057a41b73f17daf100f3fc1252903 scipy-1.2.3-cp37-cp37m-win32.whl 27f26a031ac1884ac5cb3e35343aba7c scipy-1.2.3-cp37-cp37m-win_amd64.whl 43b42a507472dfa1dff4c91d58a6543f scipy-1.2.3.tar.gz 561ee26a6d0a9b31d644db5e8244bc76 scipy-1.2.3.tar.xz 5b9f47308d06b22078810fca4f97fad2 scipy-1.2.3.zip SHA256 ~~~~~~ 2ff82db1393bd5d8ddeb9134e8f77a8e971e635452d7e65f7238f40c71d385a8 scipy-1.2.3-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 5d950892a6a2da6dae2b5a3021b329dbf04483a7fb0bd3011685db7f51578ae7 scipy-1.2.3-cp27-cp27m-manylinux1_i686.whl 487a61a7e477923c7a9e8fa06f27e3f2dbc7ce9450a970a2e5d902b0a305d028 scipy-1.2.3-cp27-cp27m-manylinux1_x86_64.whl 3397ce30e240e9a543d81f623a9e8e98ae39012cf72e42e95929530a03f20791 scipy-1.2.3-cp27-cp27m-win32.whl 8533e8c2e467eed913a0aa4fac09c9fb824f32d1ab1121db3a50845f9b347825 scipy-1.2.3-cp27-cp27m-win_amd64.whl b6cbb7125b0c742e0f31fb293d19e9f1a03db58f6ddefc51a2025ee15ae607cb scipy-1.2.3-cp27-cp27mu-manylinux1_i686.whl e870dfd006ab657f9cc48b099646c5ceb4e812e59a7d460dc80ac9a659f089dc scipy-1.2.3-cp27-cp27mu-manylinux1_x86_64.whl b8b0e81a2b87d68acf4a54aea800edafbb5bc9a04f38256718826f95f625fb75 scipy-1.2.3-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl ecc27dda36d7172bb4ed7f245499e28251fa17909b314b4e8956a942f302e1a3 scipy-1.2.3-cp34-cp34m-manylinux1_i686.whl ee213d8c8eee0540ba91efca61605dc0e2b5bd90ca35ab49c85f0ad7038df00f scipy-1.2.3-cp34-cp34m-manylinux1_x86_64.whl 57a0ee083b94944ba6329e755d112bfd53a98bd9a6a4faf10bc7722c955a7653 scipy-1.2.3-cp34-cp34m-win32.whl 1bc0720f149fbb69d19156cf91730aa21455c58949aea56bfaa2b74c06868100 scipy-1.2.3-cp34-cp34m-win_amd64.whl d39ae9cfd7bdea7753fd617e2edfc68b94a019a65c0153c3fced35cf657b79e7 scipy-1.2.3-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 7e6c485c3f2789c790fe4cc5c6c3b4554000d284fd05be90479b6fbd8795d064 scipy-1.2.3-cp35-cp35m-manylinux1_i686.whl 92fbdcf9acebf2007502adbed1b22e3a3e9445aceb5e7a568f9759d76461368b scipy-1.2.3-cp35-cp35m-manylinux1_x86_64.whl b2fad2430232f8c2faeb0abff7ed5191f0a534bd1bd482c71a9616c44e674c76 scipy-1.2.3-cp35-cp35m-win32.whl 8b3ac1e50188792fdf811e5a747df2b784c65d9a17f59609a73c8285424a48ad scipy-1.2.3-cp35-cp35m-win_amd64.whl b38f2e6d53f852ae1de1aacead2c26a69c91b36f833e744557ca370979b81652 scipy-1.2.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 39bf06b2536e69f08a257b260aaa25d088d755b73cf3751b44e5838ccb1d0a82 scipy-1.2.3-cp36-cp36m-manylinux1_i686.whl c08013e0fe1554372da9312d5bad588402f71c0636f0f86a9b9b61d507c59bac scipy-1.2.3-cp36-cp36m-manylinux1_x86_64.whl cac83647970115dfb6d29dc3b4ab44b3aa11254e8aeba115f88ad0ccbb273085 scipy-1.2.3-cp36-cp36m-win32.whl 0c23e5b3a314dce4049b969c81ad801cf05e1fe699760846c7567deaa9b8c548 scipy-1.2.3-cp36-cp36m-win_amd64.whl 503e25b8da22b1be6f2f81e1dcf26f42bfb13fe89bccbf8bc48e1b6f2a4789c8 scipy-1.2.3-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 19904399501ecf56ead21b307cc52c8ff03a2103343f554f3fbc189bb2cf1609 scipy-1.2.3-cp37-cp37m-manylinux1_i686.whl f658f90db1530f70128bbeb2c1e61006de2eb2382408080a211d885f501af3a7 scipy-1.2.3-cp37-cp37m-manylinux1_x86_64.whl 7aeb232273b38e74d89181d9165ca47314e1c4269afe7c3a1926aea3e63345a2 scipy-1.2.3-cp37-cp37m-win32.whl 53353a504ad2181eb27c9d61e91ce29324f8b7550876736a08a36e6b60c43407 scipy-1.2.3-cp37-cp37m-win_amd64.whl ecbe6413ca90b8e19f8475bfa303ac001e81b04ec600d17fa7f816271f7cca57 scipy-1.2.3.tar.gz 94ef2ac3c9c83cbbda0a5c5552fffa22993c1c21313c0eb7a2e7102b4629bc31 scipy-1.2.3.tar.xz a4f09f8c6f5924582fc61d3c5cd8174e8f8857a45dfa4a81c0d134b7c4af74cc scipy-1.2.3.zip -----BEGIN PGP SIGNATURE----- iQGzBAEBCAAdFiEEuinQzfUTSK0iykiguEZ+VI8xggYFAl4nNasACgkQuEZ+VI8x ggYXiQwArWCb+6PjBb/vyjDO8KOjYmVCCNm8OxC1OvJ3Nvm+CMOY9z8nurfImpZ3 3mL/pl3mqmvMj7u7i30pOVXlfdVUaAoRKbmyjOnXDGTOTogbPEkkf6PPJOKgdXYo 1ml+FmdAFPIyNJhJOMv9ME5rssZlLqpwxbwqKsRBnC3/eHLLXgJ2/kFL9dunPSDN upfSpptbBbuQJpsA6+OxUB3DKZ3VsxbOG7+b+2dIiKfsRkt1N2nQi4g/HJqbQ85P 8G7s4Twqo6DToUZdDcud/C+hsUsFkqRRh/0osLjuu99H3z4y0+F6rZSedwf5T4Wa zFU5IFiU92T4pA5Y0EnZxJS4MnB7jaVeSe+FLf23q7Y6Hi1nC3s4/rLxqUhKzO0V T02G+/hAyeFol2/6quwephzn3Vv+fiZ+hg7zEvDnnjP/TlmSwW/csLRTHXb1qu/z FXxb/Xs1f+/jPbjwqb0gA7vpzs7OgodWZtML+0FOxPwjyNBxWQQe/pbuppjtszQq 2iysS9UM =E9q4 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From fbastien at nvidia.com Tue Jan 21 12:58:24 2020 From: fbastien at nvidia.com (Frederic Bastien) Date: Tue, 21 Jan 2020 17:58:24 +0000 Subject: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? In-Reply-To: <62654f07-ae49-f196-a76d-72707c6f20e2@icloud.com> References: <62654f07-ae49-f196-a76d-72707c6f20e2@icloud.com> Message-ID: The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up. [1] https://github.com/pydata/numexpr -----Original Message----- From: NumPy-Discussion On Behalf Of Compl Yue Sent: Tuesday, January 21, 2020 10:33 AM To: numpy-discussion at python.org Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? External email: Use caution opening links or attachments Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 ??10:20, Matthew Brett wrote: > Hi, > > On Mon, Jan 20, 2020 at 3:00 PM YueCompl wrote: >> Hello folks, >> >> Myself a long time Numpy user, and recently done some Haskell stuff, >> >> Please have a look at >> https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate and/or http://hackage.haskell.org/package/repa doing the number crunching? > Interesting - do you have any feeling for the differences in speed for > Haskell accelerate array processing as compared to Numpy with default > OpenBLAS? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ----------------------------------------------------------------------------------- From ralf.gommers at gmail.com Tue Jan 21 14:19:09 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 21 Jan 2020 20:19:09 +0100 Subject: [Numpy-discussion] new absolute beginners tutorial merged Message-ID: Hi all, I think this is worth announcing: we have a new tutorial in the user guide titled "NumPy: the absolute basics for beginners". This was Anne Bonner's Season of Docs project, and it is a big step in the right direction for making are docs more accessible to new users. Up at https://numpy.org/devdocs/user/absolute_beginners.html Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Jan 21 16:13:14 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 22 Jan 2020 08:13:14 +1100 Subject: [Numpy-discussion] Re-enable deprecation warning for "ragged arrays" Message-ID: <706b9ac0-80c0-e7de-1ee7-85d823d0f798@gmail.com> A few months ago we merged an implementation of "NEP 34 - Disallow inferring dtype=object from sequences" https://numpy.org/neps/nep-0034.html but it broke downstream packages like scipy, pandas, matplotlib, astropy (those are the ones we know about). Since then, the downstream projects seem to have fixed the problematic code. We have re-merged the implementation. Please check your package against the latest NumPy HEAD and let us know if there are problems. Matti From sebastian at sipsolutions.net Tue Jan 21 16:47:32 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 21 Jan 2020 13:47:32 -0800 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, Jan. 22 Message-ID: Hi all, There will be a NumPy Community meeting Wednesday January 22 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From baker.alexander at gmail.com Tue Jan 21 17:19:31 2020 From: baker.alexander at gmail.com (alexander baker) Date: Tue, 21 Jan 2020 22:19:31 +0000 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: References: Message-ID: This looks really helpful, thanks to all those whom took the time to write. Alex b. > On 21 Jan 2020, at 19:19, Ralf Gommers wrote: > > Hi all, > > I think this is worth announcing: we have a new tutorial in the user guide titled "NumPy: the absolute basics for beginners". This was Anne Bonner's Season of Docs project, and it is a big step in the right direction for making are docs more accessible to new users. > > Up at https://numpy.org/devdocs/user/absolute_beginners.html > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jan 21 18:06:27 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Jan 2020 16:06:27 -0700 Subject: [Numpy-discussion] Re-enable deprecation warning for "ragged arrays" In-Reply-To: <706b9ac0-80c0-e7de-1ee7-85d823d0f798@gmail.com> References: <706b9ac0-80c0-e7de-1ee7-85d823d0f798@gmail.com> Message-ID: On Tue, Jan 21, 2020 at 2:14 PM Matti Picus wrote: > A few months ago we merged an implementation of "NEP 34 - Disallow > inferring dtype=object from sequences" > https://numpy.org/neps/nep-0034.html but it broke downstream packages > like scipy, pandas, matplotlib, astropy (those are the ones we know > about). Since then, the downstream projects seem to have fixed the > problematic code. We have re-merged the implementation. Please check > your package against the latest NumPy HEAD and let us know if there are > problems. > > I went ahead and pushed to numpy-wheels master to build new pre-release wheels. Scheduled builds don't make wheels for Windows so it is useful to make explicit pushes when new features need testing downstream. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From anne at contentsimplicity.com Tue Jan 21 18:20:06 2020 From: anne at contentsimplicity.com (Anne Bonner) Date: Tue, 21 Jan 2020 15:20:06 -0800 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: References: Message-ID: What an exciting day!!! Thank you! ~Anne On Tue, Jan 21, 2020 at 2:20 PM alexander baker wrote: > This looks really helpful, thanks to all those whom took the time to > write. Alex b. > > On 21 Jan 2020, at 19:19, Ralf Gommers wrote: > > Hi all, > > I think this is worth announcing: we have a new tutorial in the user guide > titled "NumPy: the absolute basics for beginners". This was Anne Bonner's > Season of Docs project, and it is a big step in the right direction for > making are docs more accessible to new users. > > Up at https://numpy.org/devdocs/user/absolute_beginners.html > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- *Anne Bonner | **Content Simplicity | **Making complicated concepts seem simple.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Jan 21 19:12:58 2020 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 21 Jan 2020 19:12:58 -0500 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: References: Message-ID: <20200122001258.tnij7q7xuo4fabe6@phare.normalesup.org> This is great!! Did someone tweet about this, so that I retweet? I'm happy crafting a tweet myself, but I'd rather shine light on someone who deserves the attention. Thanks a lot to all involved, in the name of the community, Ga?l On Tue, Jan 21, 2020 at 03:20:06PM -0800, Anne Bonner wrote: > What an exciting day!!! Thank you!? > ~Anne > On Tue, Jan 21, 2020 at 2:20 PM alexander baker > wrote: > This looks really helpful, thanks to all those whom took the time to write. > Alex b. > On 21 Jan 2020, at 19:19, Ralf Gommers wrote: > Hi all, > I think this is worth announcing: we have a new tutorial in the user > guide titled "NumPy: the absolute basics for beginners". This was Anne > Bonner's Season of Docs project, and it is a big step in the right > direction for making are docs more accessible to new users. > Up at https://numpy.org/devdocs/user/absolute_beginners.html > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -- Gael Varoquaux Research Director, INRIA Visiting professor, McGill http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From charlesr.harris at gmail.com Tue Jan 21 21:11:17 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Jan 2020 19:11:17 -0700 Subject: [Numpy-discussion] pip 20 release broke wheels builds. Message-ID: Hi All, Just a note that latest pip release broke the numpy wheel builds. The fix is to update multibuild to the latest version. I'm sending this out in case others are affected. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From compl.yue at icloud.com Wed Jan 22 01:02:12 2020 From: compl.yue at icloud.com (YueCompl) Date: Wed, 22 Jan 2020 14:02:12 +0800 Subject: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? In-Reply-To: References: <62654f07-ae49-f196-a76d-72707c6f20e2@icloud.com> Message-ID: <46D9AA7D-0BEE-455B-AD2B-13F4A16CB499@icloud.com> Yes, I do feel NumExpr does very similar things as Repa, while Repa works rather statically for Haskell code, NumExpr is much more interactive. Well, honestly as a long time Numpy user, I'm aware of NumExpr since early days I learning Numpy, but never ever resorted to NumExpr, it's neither as complete as Numpy, nor as flexible as Numba. I'm neither proficient in Haskell at data exploration for now, attempting improving the tooling (e.g. [hadui](https://github.com/complyue/hadui ) ), while still working on an ndarray interface for Haskell to an in house array database (mmap'ed virtual large datafile per se). I miss very much the workflow with Pandas/Numpy/Numba, the interactive aspect is crucial to me and my team's daily work. We have to resort to Haskell because some time ago we needed to start processing irregular event streams (no common dimension along time axis), for thousands of series (instruments in the stock market). While in handling this many number of inputs (passing in a tuple of 1d arrays, Numba treats each array as a separate input variable), Numba will swallow 20+GB ram with 10+ hours in jit compilation without a result. I'm aware of Julia, and feels very pity for my org not able to adopt it for hiring difficulties, while luckily Haskell is agreed to be the base of our next gen internal platform. So I'm currently working hard to bring sufficient interaction abilities with a Haskell tool chain. > On 2020-01-22, at 01:58, Frederic Bastien wrote: > > The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up. > > > > [1] https://github.com/pydata/numexpr > > -----Original Message----- > From: NumPy-Discussion On Behalf Of Compl Yue > Sent: Tuesday, January 21, 2020 10:33 AM > To: numpy-discussion at python.org > Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? > > External email: Use caution opening links or attachments > > > Hi Matthew, > > I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. > > Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. > > btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. > > btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. > > btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. > > Cheers, > > Compl > > > On 2020/1/21 ??10:20, Matthew Brett wrote: >> Hi, >> >> On Mon, Jan 20, 2020 at 3:00 PM YueCompl wrote: >>> Hello folks, >>> >>> Myself a long time Numpy user, and recently done some Haskell stuff, >>> >>> Please have a look at >>> https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate and/or http://hackage.haskell.org/package/repa doing the number crunching? >> Interesting - do you have any feeling for the differences in speed for >> Haskell accelerate array processing as compared to Numpy with default >> OpenBLAS? >> >> Cheers, >> >> Matthew >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > ----------------------------------------------------------------------------------- > This email message is for the sole use of the intended recipient(s) and may contain > confidential information. Any unauthorized review, use, disclosure or distribution > is prohibited. If you are not the intended recipient, please contact the sender by > reply email and destroy all copies of the original message. > ----------------------------------------------------------------------------------- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 22 11:02:01 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 22 Jan 2020 17:02:01 +0100 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: <20200122001258.tnij7q7xuo4fabe6@phare.normalesup.org> References: <20200122001258.tnij7q7xuo4fabe6@phare.normalesup.org> Message-ID: On Wed, Jan 22, 2020 at 1:13 AM Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > This is great!! > > Did someone tweet about this, so that I retweet? I'm happy crafting a > tweet myself, but I'd rather shine light on someone who deserves the > attention. > Thanks Gael. Here is Anne's tweet: https://twitter.com/annebonnerdata/status/1219778893537669121. Cheers, Ralf > Thanks a lot to all involved, in the name of the community, > > Ga?l > > On Tue, Jan 21, 2020 at 03:20:06PM -0800, Anne Bonner wrote: > > What an exciting day!!! Thank you! > > > ~Anne > > > On Tue, Jan 21, 2020 at 2:20 PM alexander baker < > baker.alexander at gmail.com> > > wrote: > > > This looks really helpful, thanks to all those whom took the time to > write. > > Alex b. > > > On 21 Jan 2020, at 19:19, Ralf Gommers > wrote: > > > > Hi all, > > > I think this is worth announcing: we have a new tutorial in the > user > > guide titled "NumPy: the absolute basics for beginners". This > was Anne > > Bonner's Season of Docs project, and it is a big step in the > right > > direction for making are docs more accessible to new users. > > > Up at https://numpy.org/devdocs/user/absolute_beginners.html > > > Cheers, > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > -- > Gael Varoquaux > Research Director, INRIA Visiting professor, McGill > http://gael-varoquaux.info http://twitter.com/GaelVaroquaux > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Wed Jan 22 11:42:48 2020 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 22 Jan 2020 11:42:48 -0500 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: References: <20200122001258.tnij7q7xuo4fabe6@phare.normalesup.org> Message-ID: <20200122164248.vehov7k6lfhlnkno@phare.normalesup.org> Thanks. Exactly what I needed. I don't know why I did not find it myself :). Cheers, Ga?l On Wed, Jan 22, 2020 at 05:02:01PM +0100, Ralf Gommers wrote: > On Wed, Jan 22, 2020 at 1:13 AM Gael Varoquaux > wrote: > Did someone tweet about this, so that I retweet? I'm happy crafting a > tweet myself, but I'd rather shine light on someone who deserves the > attention. > Thanks Gael. Here is Anne's tweet: https://twitter.com/annebonnerdata/status/ > 1219778893537669121. > Cheers, > Ralf > Thanks a lot to all involved, in the name of the community, > Ga?l > On Tue, Jan 21, 2020 at 03:20:06PM -0800, Anne Bonner wrote: > > What an exciting day!!! Thank you!? > > ~Anne > > On Tue, Jan 21, 2020 at 2:20 PM alexander baker < > baker.alexander at gmail.com> > > wrote: > >? ? ?This looks really helpful, thanks to all those whom took the time to > write. > >? ? ?Alex b. > >? ? ?On 21 Jan 2020, at 19:19, Ralf Gommers > wrote: > >? ? ? ? ?Hi all, > >? ? ? ? ?I think this is worth announcing: we have a new tutorial in the > user > >? ? ? ? ?guide titled "NumPy: the absolute basics for beginners". This was > Anne > >? ? ? ? ?Bonner's Season of Docs project, and it is a big step in the > right > >? ? ? ? ?direction for making are docs more accessible to new users. > >? ? ? ? ?Up at https://numpy.org/devdocs/user/absolute_beginners.html > >? ? ? ? ?Cheers, > >? ? ? ? ?Ralf > >? ? ? ? ?_______________________________________________ > >? ? ? ? ?NumPy-Discussion mailing list > >? ? ? ? ?NumPy-Discussion at python.org > >? ? ? ? ?https://mail.python.org/mailman/listinfo/numpy-discussion > >? ? ?_______________________________________________ > >? ? ?NumPy-Discussion mailing list > >? ? ?NumPy-Discussion at python.org > >? ? ?https://mail.python.org/mailman/listinfo/numpy-discussion -- Gael Varoquaux Research Director, INRIA Visiting professor, McGill http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From anne at contentsimplicity.com Wed Jan 22 17:08:43 2020 From: anne at contentsimplicity.com (Anne Bonner) Date: Wed, 22 Jan 2020 14:08:43 -0800 Subject: [Numpy-discussion] new absolute beginners tutorial merged In-Reply-To: <20200122164248.vehov7k6lfhlnkno@phare.normalesup.org> References: <20200122001258.tnij7q7xuo4fabe6@phare.normalesup.org> <20200122164248.vehov7k6lfhlnkno@phare.normalesup.org> Message-ID: Wow!!!! Thank you so much for the Twitter love! I am so grateful. Thank you!!! Anne On Wed, Jan 22, 2020 at 8:43 AM Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > Thanks. Exactly what I needed. I don't know why I did not find it myself > :). > > Cheers, > > Ga?l > > On Wed, Jan 22, 2020 at 05:02:01PM +0100, Ralf Gommers wrote: > > On Wed, Jan 22, 2020 at 1:13 AM Gael Varoquaux < > gael.varoquaux at normalesup.org> > > wrote: > > > Did someone tweet about this, so that I retweet? I'm happy crafting a > > tweet myself, but I'd rather shine light on someone who deserves the > > attention. > > > > Thanks Gael. Here is Anne's tweet: > https://twitter.com/annebonnerdata/status/ > > 1219778893537669121. > > > Cheers, > > Ralf > > > > > Thanks a lot to all involved, in the name of the community, > > > Ga?l > > > On Tue, Jan 21, 2020 at 03:20:06PM -0800, Anne Bonner wrote: > > > What an exciting day!!! Thank you! > > > > ~Anne > > > > On Tue, Jan 21, 2020 at 2:20 PM alexander baker < > > baker.alexander at gmail.com> > > > wrote: > > > > This looks really helpful, thanks to all those whom took the > time to > > write. > > > Alex b. > > > > On 21 Jan 2020, at 19:19, Ralf Gommers > > > wrote: > > > > > Hi all, > > > > I think this is worth announcing: we have a new tutorial > in the > > user > > > guide titled "NumPy: the absolute basics for beginners". > This was > > Anne > > > Bonner's Season of Docs project, and it is a big step in > the > > right > > > direction for making are docs more accessible to new users. > > > > Up at > https://numpy.org/devdocs/user/absolute_beginners.html > > > > Cheers, > > > Ralf > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > -- > Gael Varoquaux > Research Director, INRIA Visiting professor, McGill > http://gael-varoquaux.info http://twitter.com/GaelVaroquaux > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- *Anne Bonner | **Content Simplicity | **Making complicated concepts seem simple.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jan 24 11:45:41 2020 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 24 Jan 2020 17:45:41 +0100 Subject: [Numpy-discussion] what to clean up, what to leave as is Message-ID: Hi all, It's great to see that people are jumping at the chance to clean up Python 2 support. I would however caution about overdoing it on other cleanups. As a reminder, we normally do not want pure style PRs (e.g. PEP8 cleanups), because they make the code history (git blame, commits on particular files, etc.) harder to work with, have review overhead, and may introduce new bugs for little gain. Imho that same rationale applies to things like converting strings to f-strings. There's of course some gray area, for example removing "from ... import *" can guard against accidentally exposing new API, so can be considered a valuable cleanup. As a separate/additional point: numpy.distutils and numpy.f2py are largely untested, PRs are hard to test locally because of platform-specific code, and changes often introduce regressions. So even for some cleanups that are okay for other files, please do not do them on those modules. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 24 12:28:29 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 24 Jan 2020 10:28:29 -0700 Subject: [Numpy-discussion] what to clean up, what to leave as is In-Reply-To: References: Message-ID: On Fri, Jan 24, 2020 at 9:46 AM Ralf Gommers wrote: > Hi all, > > It's great to see that people are jumping at the chance to clean up Python > 2 support. I would however caution about overdoing it on other cleanups. As > a reminder, we normally do not want pure style PRs (e.g. PEP8 cleanups), > because they make the code history (git blame, commits on particular files, > etc.) harder to work with, have review overhead, and may introduce new bugs > for little gain. > > Imho that same rationale applies to things like converting strings to > f-strings. There's of course some gray area, for example removing "from ... > import *" can guard against accidentally exposing new API, so can be > considered a valuable cleanup. > > As a separate/additional point: numpy.distutils and numpy.f2py are largely > untested, PRs are hard to test locally because of platform-specific code, > and changes often introduce regressions. So even for some cleanups that are > okay for other files, please do not do them on those modules. > > I do like f-strings, they can make the code simpler and more readable. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.k.sheppard at gmail.com Fri Jan 24 12:41:33 2020 From: kevin.k.sheppard at gmail.com (Kevin Sheppard) Date: Fri, 24 Jan 2020 17:41:33 +0000 Subject: [Numpy-discussion] what to clean up, what to leave as is In-Reply-To: References: Message-ID: I think some types of clean-ups, for example, imports, are pretty low cost, low risk and don't have much bearing on and it might be best to do them all at once. f-strings are also pretty simple but can be abused to the detriment of code around (for example, moving a string defined in a variable outside a function into the function just so it can have an f-string). I think more general code clean-up, for example, removing list around iterators (e.g., list(map(f, i)) should be mostly avoided unless there is a compelling case to prefer the iterator for performance reasons. Kevin On Fri, Jan 24, 2020 at 5:29 PM Charles R Harris wrote: > > > On Fri, Jan 24, 2020 at 9:46 AM Ralf Gommers > wrote: > >> Hi all, >> >> It's great to see that people are jumping at the chance to clean up >> Python 2 support. I would however caution about overdoing it on other >> cleanups. As a reminder, we normally do not want pure style PRs (e.g. PEP8 >> cleanups), because they make the code history (git blame, commits on >> particular files, etc.) harder to work with, have review overhead, and may >> introduce new bugs for little gain. >> >> Imho that same rationale applies to things like converting strings to >> f-strings. There's of course some gray area, for example removing "from ... >> import *" can guard against accidentally exposing new API, so can be >> considered a valuable cleanup. >> >> As a separate/additional point: numpy.distutils and numpy.f2py are >> largely untested, PRs are hard to test locally because of platform-specific >> code, and changes often introduce regressions. So even for some cleanups >> that are okay for other files, please do not do them on those modules. >> >> > I do like f-strings, they can make the code simpler and more readable. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 24 12:57:47 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 24 Jan 2020 10:57:47 -0700 Subject: [Numpy-discussion] what to clean up, what to leave as is In-Reply-To: References: Message-ID: On Fri, Jan 24, 2020 at 10:42 AM Kevin Sheppard wrote: > I think some types of clean-ups, for example, imports, are pretty low > cost, low risk and don't have much bearing on and it might be best to do > them all at once. > > f-strings are also pretty simple but can be abused to the detriment of > code around (for example, moving a string defined in a variable outside a > function into the function just so it can have an f-string). > > I think more general code clean-up, for example, removing list around > iterators (e.g., list(map(f, i)) should be mostly avoided unless there is a > compelling case to prefer the iterator for performance reasons. > > I tend to see that as a matter of timing. At some point such things become archaic and unfamiliar as developers who grew up with modern Python become more common. But there is no rush. The problem with code history is trickier, and almost unavoidable over time. The information is still there, but it is harder to dig it out. Probably there need to be better tools to deal with that. This is especially apparent from the early C code cleanup that I did just to get more modern and readable code styling. My name is all over code that I didn't write, just formatted. But the cleanup was necessary for maintenance if nothing else. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 24 13:15:03 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 24 Jan 2020 10:15:03 -0800 Subject: [Numpy-discussion] what to clean up, what to leave as is In-Reply-To: References: Message-ID: <978df365d6fa6ea25a68b8b88f7895623d667150.camel@sipsolutions.net> On Fri, 2020-01-24 at 10:57 -0700, Charles R Harris wrote: > > > On Fri, Jan 24, 2020 at 10:42 AM Kevin Sheppard < > kevin.k.sheppard at gmail.com> wrote: > > I think some types of clean-ups, for example, imports, are pretty > > low cost, low risk and don't have much bearing on and it might be > > best to do them all at once. > > > > f-strings are also pretty simple but can be abused to the detriment > > of code around (for example, moving a string defined in a variable > > outside a function into the function just so it can have an f- > > string). > > > > I think more general code clean-up, for example, removing list > > around iterators (e.g., list(map(f, i)) should be mostly avoided > > unless there is a compelling case to prefer the iterator for > > performance reasons. > > > > > > I tend to see that as a matter of timing. At some point such things > become archaic and unfamiliar as developers who grew up with modern > Python become more common. But there is no rush. > Honestly, I am OK with simply not cleaning up some stuff. It was a quick thought. But, if we ever get around doing it, it seems like a good time now when we already have a big code churn do to Py2 removal. > The problem with code history is trickier, and almost unavoidable > over time. The information is still there, but it is harder to dig it > out. Probably there need to be better tools to deal with that. This > is especially apparent from the early C code cleanup that I did just > to get more modern and readable code styling. My name is all over > code that I didn't write, just formatted. But the cleanup was > necessary for maintenance if nothing else. Code history is a problem and it is annoying that code attribution is obfuscated by it. The argument for doing style cleanups is that if we do not use modern patterns in large chunks of the code, the old patterns are likely to repeated indefinitely. At least within their files [0]. Tools are maybe getting a bit better (e.g. github allows to click on "view blame before this, although I doubt we ever get somewhere were they will figure out that something was "just" a style cleanup. - Sebastian [0] An example are pytest vs. nose testing styles, where it is unlikely we ever lose the complexity of keeping random snippets of nose style around (which definitely will confuse new contributors). > > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Fri Jan 24 13:28:54 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 24 Jan 2020 10:28:54 -0800 Subject: [Numpy-discussion] Deprecate unused but exposed C-API functions Message-ID: Hi all, I would like to deprecate three C-API functions (more may come later) in https://github.com/numpy/numpy/pull/15427 The functions are: * PyArray_GetArrayParamsFromObject * PyUfunc_GenericCall * PyUFunc_SetUsesArraysAsData I could not find downstream usage for any of these and they seem useful only on first glance to me. The first one is too complex (regarding scalars), simply converting to an array first should be just as well. The second one is almost identical to `PyObject_Call`. And the third one is not even documented and while the idea is sound, I am not sure that there are any cases where it is useful [0] My idea is to add the deprecation warning now, and if nobody notices it remove them as soon as 1.19 is released. - Sebastian [0] It seems useful for user datatypes/loop, but using it for those does not seem possible (or at least straight forward), since they are usually stored in a way not accessible through this function. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Fri Jan 24 19:56:43 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 24 Jan 2020 17:56:43 -0700 Subject: [Numpy-discussion] Deprecate unused but exposed C-API functions In-Reply-To: References: Message-ID: On Fri, Jan 24, 2020 at 11:29 AM Sebastian Berg wrote: > Hi all, > > I would like to deprecate three C-API functions (more may come later) > in https://github.com/numpy/numpy/pull/15427 > > The functions are: > * PyArray_GetArrayParamsFromObject > * PyUfunc_GenericCall > * PyUFunc_SetUsesArraysAsData > > I could not find downstream usage for any of these and they seem useful > only on first glance to me. > > The first one is too complex (regarding scalars), simply converting to > an array first should be just as well. > The second one is almost identical to `PyObject_Call`. And the third > one is not even documented and while the idea is sound, I am not sure > that there are any cases where it is useful [0] > > My idea is to add the deprecation warning now, and if nobody notices it > remove them as soon as 1.19 is released. > > I don't think C-API functions should be removed. Deprecation is fine, but removal changes the customary order of the functions. If they cost nothing to maintain, just leave them alone with a note/documentation that there are preferable alternatives. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 24 20:04:16 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 24 Jan 2020 17:04:16 -0800 Subject: [Numpy-discussion] Deprecate unused but exposed C-API functions In-Reply-To: References: Message-ID: <00fbba31e21993def28a66e09537bc192398ffdf.camel@sipsolutions.net> On Fri, 2020-01-24 at 17:56 -0700, Charles R Harris wrote: > > > On Fri, Jan 24, 2020 at 11:29 AM Sebastian Berg < > sebastian at sipsolutions.net> wrote: > > Hi all, > > > > > > My idea is to add the deprecation warning now, and if nobody > > notices it > > remove them as soon as 1.19 is released. > > > > I don't think C-API functions should be removed. Deprecation is fine, > but removal changes the customary order of the functions. If they > cost nothing to maintain, just leave them alone with a > note/documentation that there are preferable alternatives. > To be clear: removal in this sense for me would mean that they exist as a stub which always raises an error. In principle they could be set to NULL as well, but I think the always- error solution is just as good probably. - Sebastian > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Tue Jan 28 14:42:04 2020 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 28 Jan 2020 11:42:04 -0800 Subject: [Numpy-discussion] NumPy Development Meeting - Triage Focus Message-ID: <120c800a8301a9b1b80b39b71ac9cb94d7f3944f.camel@sipsolutions.net> Hi all, Our bi-weekly triage-focused NumPy development meeting is tomorrow (Wednesday, January 29) at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg Best regards, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From melissawm at gmail.com Wed Jan 29 07:49:25 2020 From: melissawm at gmail.com (=?UTF-8?Q?Melissa_Mendon=C3=A7a?=) Date: Wed, 29 Jan 2020 09:49:25 -0300 Subject: [Numpy-discussion] SciPy 2020 Call for Submissions is open! In-Reply-To: References: Message-ID: Apologies for crossposting. ===== SciPy 2020, the 19th annual Scientific Computing with Python conference, will be held July 6-12, 2020 in Austin, Texas. The annual SciPy Conference brings together over 900 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for SciPy 2020 talks, posters, and tutorials is now open through February 11, 2020. Talks and Posters (July 8-10, 2020) In addition to the general track, this year will have the following specialized tracks, mini symposia, and sessions: Tracks High Performance Python Machine Learning and Data Science Mini Symposia Astronomy and Astrophysics Biology and Bioinformatics Earth, Ocean, Geo and Atmospheric Science Materials Science Special Sessions Maintainers Track SciPy Tools Plenary Session For additional details and instructions, please see the conference website. Tutorials (July 6-7, 2020) Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for awesome techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend. For examples of content and format, you can refer to tutorials from past SciPy tutorial sessions (SciPy 2018 , SciPy2019 ) and some accepted submissions . For additional details and instructions see the conference website . Submission page: https://easychair.org/conferences/?conf=scipy2020 Submission Deadline: February 11, 2020 - Melissa Weber Mendon?a Diversity Committee Co-Chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jan 30 21:53:05 2020 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 30 Jan 2020 19:53:05 -0700 Subject: [Numpy-discussion] Python 3.9 Message-ID: Hi All, Out of curiosity, I ran the numpy tests on Python3.9a3 to see if we hit any of the expired deprecations. The only error looked like a Python bug in fractions.Fraction. For others who may want to try this for a bit of fun: - Download and altinstall Python3.9a3 - Clone and install cython in python3.9 from the cython master branch - Install pytest in python3.9 This worked for me on Fedora 31. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: