From tyler.je.reddy at gmail.com Tue Feb 5 21:12:26 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 5 Feb 2019 18:12:26 -0800 Subject: [Numpy-discussion] Reminder: weekly status meeting Feb. 6 at 12:00 Pacific Time Message-ID: Draft agenda: https://hackmd.io/5AUINxNfS3iN0txIaN32HA?view There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. BIDS team -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Feb 8 09:18:04 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 8 Feb 2019 07:18:04 -0700 Subject: [Numpy-discussion] New sorting routines. Message-ID: Hi All, I've put up gh-12945 that preserves forward compatibility of NumPy after the addition of timsort by reusing the mergesort slot in PyArray_ArrFuncs for stable sorts in general, not just mergesort. The same method can be used when we add radixsort for integer types. See the discussion at the PR link for more details on the nature of the problem and considerations for the dtype proposals. I'm waiting on agreement that this is the way to go before proceeding. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sat Feb 9 01:05:17 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Fri, 8 Feb 2019 22:05:17 -0800 Subject: [Numpy-discussion] ANN: SciPy 1.2.1 Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.2.1, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.2.1 One of a few ways to install this release with pip: pip install scipy==1.2.1 ========================== SciPy 1.2.1 Release Notes ========================== SciPy 1.2.1 is a bug-fix release with no new features compared to 1.2.0. Most importantly, it solves the issue that 1.2.0 cannot be installed from source on Python 2.7 because of non-ASCII character issues. It is also notable that SciPy 1.2.1 wheels were built with OpenBLAS 0.3.5.dev, which may alleviate some linear algebra issues observed in SciPy 1.2.0. Authors ======= * Eric Larson * Mark Mikofski * Evgeni Burovski * Ralf Gommers * Eric Moore * Tyler Reddy Issues closed for 1.2.1 ------------------------ * `#9606 `__: SyntaxError: Non-ASCII character '\xe2' in file scipy/stats/_continuous_distns.py on line 3346, but no encoding declared * `#9608 `__: Version 1.2.0 introduces `too many indices for array` error in... * `#9709 `__: scipy.stats.gaussian_kde normalizes the weights keyword argument... * `#9733 `__: scipy.linalg.qr_update gives NaN result * `#9724 `__: CI: Is scipy.scipy Windows Python36-32bit-full working? Pull requests for 1.2.1 ------------------------ * `#9612 `__: BUG: don't use array newton unless size is greater than 1 * `#9615 `__: ENH: Add test for encoding * `#9720 `__: BUG: stats: weighted KDE does not modify the weights array * `#9739 `__: BUG: qr_updates fails if u is exactly in span Q * `#9725 `__: TST: pin mingw for Azure Win CI * `#9736 `__: TST: adjust to vmImage dispatch in Azure * `#9681 `__: BUG: Fix failing stats tests (partial backport) * `#9662 `__: TST: interpolate: avoid pytest deprecations Checksums ========= MD5 ~~~ 982810997da9daab2e512a6c27918889 scipy-1.2.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 9500cad49b7eac9786c90cba146ad357 scipy-1.2.1-cp27-cp27m-manylinux1_i686.whl fe9ba7f16e0e7f7c9cd59d39ea8e9545 scipy-1.2.1-cp27-cp27m-manylinux1_x86_64.whl f1823b26b2afda2027f78df427791700 scipy-1.2.1-cp27-cp27m-win32.whl c0692c60b4baaafd99fb2bf0c689bbf1 scipy-1.2.1-cp27-cp27m-win_amd64.whl b5a5b9b146bac064a7921c5121d5e19c scipy-1.2.1-cp27-cp27mu-manylinux1_i686.whl fd637f4dc4308ae9cf7655e3e5f3a27d scipy-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl 8faae90250009230cd2f7845ba4735f2 scipy-1.2.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl a28a93cb20987666f788d43c50e2ab79 scipy-1.2.1-cp34-cp34m-manylinux1_i686.whl 26264ebb568728065e3ef73b5419793a scipy-1.2.1-cp34-cp34m-manylinux1_x86_64.whl fbb79da3d5823ba0f23b0f8d50023bfd scipy-1.2.1-cp34-cp34m-win32.whl f82cf6834e4e2a2e4008fec5da2f4b0b scipy-1.2.1-cp34-cp34m-win_amd64.whl cc248fd969b11589d98bcb550330e08d scipy-1.2.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 3b5e8eb89a49fb7bc506cf17a54b926b scipy-1.2.1-cp35-cp35m-manylinux1_i686.whl 0cd66785fd9f8390494f42beb499f56a scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl 379d37ae932af2096e037b4defd4cf60 scipy-1.2.1-cp35-cp35m-win32.whl 5dd4c47c501c96dc0c20b72521058133 scipy-1.2.1-cp35-cp35m-win_amd64.whl 6fa013443295e55c96eb174578dcffee scipy-1.2.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 5069317ab38e3dfbb8923341c2b23a69 scipy-1.2.1-cp36-cp36m-manylinux1_i686.whl 8e6a1fd1354428074d8457fe46cd4f7c scipy-1.2.1-cp36-cp36m-manylinux1_x86_64.whl 27142bff898a18368032165c27c11466 scipy-1.2.1-cp36-cp36m-win32.whl 489fd885aefd07bded69a8e0452c16b0 scipy-1.2.1-cp36-cp36m-win_amd64.whl 84ef33013d9871f23db0ecad25552636 scipy-1.2.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 7b7ae846b70e60188b3d772e9d60d3d2 scipy-1.2.1-cp37-cp37m-manylinux1_i686.whl 21b29788cd2c037366d2ff5d54ae7517 scipy-1.2.1-cp37-cp37m-manylinux1_x86_64.whl e11ce933d012481ea27fe49314b606aa scipy-1.2.1-cp37-cp37m-win32.whl 23424815af8e69f945bb406a0c1be6c6 scipy-1.2.1-cp37-cp37m-win_amd64.whl fe88877767563e50078df86dadf7b1f4 scipy-1.2.1.tar.gz 312110f3ea37269003e587207eb72855 scipy-1.2.1.tar.xz b4e2b70f7885a746618e8bcdf77857d7 scipy-1.2.1.zip SHA256 ~~~~~~ f06819b028b8ef9010281e74c59cb35483933583043091ed6b261bb1540f11cc scipy-1.2.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 70e37cec0ac0fe95c85b74ca4e0620169590fd5d3f44765f3c3a532cedb0e5fd scipy-1.2.1-cp27-cp27m-manylinux1_i686.whl 9726791484f08e394af0b59eb80489ad94d0a53bbb58ab1837dcad4d58489863 scipy-1.2.1-cp27-cp27m-manylinux1_x86_64.whl 790cbd3c8d09f3a6d9c47c4558841e25bac34eb7a0864a9def8f26be0b8706af scipy-1.2.1-cp27-cp27m-win32.whl 014cb900c003b5ac81a53f2403294e8ecf37aedc315b59a6b9370dce0aa7627a scipy-1.2.1-cp27-cp27m-win_amd64.whl 78e12972e144da47326958ac40c2bd1c1cca908edc8b01c26a36f9ffd3dce466 scipy-1.2.1-cp27-cp27mu-manylinux1_i686.whl f15f2d60a11c306de7700ee9f65df7e9e463848dbea9c8051e293b704038da60 scipy-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl e2cfcbab37c082a5087aba5ff00209999053260441caadd4f0e8f4c2d6b72088 scipy-1.2.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl b86ae13c597fca087cb8c193870507c8916cefb21e52e1897da320b5a35075e5 scipy-1.2.1-cp34-cp34m-manylinux1_i686.whl 865afedf35aaef6df6344bee0de391ee5e99d6e802950a237f9fb9b13e441f91 scipy-1.2.1-cp34-cp34m-manylinux1_x86_64.whl e742f1f5dcaf222e8471c37ee3d1fd561568a16bb52e031c25674ff1cf9702d5 scipy-1.2.1-cp34-cp34m-win32.whl f6b88c8d302c3dac8dff7766955e38d670c82e0d79edfc7eae47d6bb2c186594 scipy-1.2.1-cp34-cp34m-win_amd64.whl 7274735fb6fb5d67d3789ddec2cd53ed6362539b41aa6cc0d33a06c003aaa390 scipy-1.2.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 963815c226b29b0176d5e3d37fc9de46e2778ce4636a5a7af11a48122ef2577c scipy-1.2.1-cp35-cp35m-manylinux1_i686.whl 281a34da34a5e0de42d26aed692ab710141cad9d5d218b20643a9cb538ace976 scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl 5a10661accd36b6e2e8855addcf3d675d6222006a15795420a39c040362def66 scipy-1.2.1-cp35-cp35m-win32.whl def0e5d681dd3eb562b059d355ae8bebe27f5cc455ab7c2b6655586b63d3a8ea scipy-1.2.1-cp35-cp35m-win_amd64.whl 79792c8fe8e9d06ebc50fe23266522c8c89f20aa94ac8e80472917ecdce1e5ba scipy-1.2.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 6dcc43a88e25b815c2dea1c6fac7339779fc988f5df8396e1de01610604a7c38 scipy-1.2.1-cp36-cp36m-manylinux1_i686.whl f31338ee269d201abe76083a990905473987371ff6f3fdb76a3f9073a361cf37 scipy-1.2.1-cp36-cp36m-manylinux1_x86_64.whl 9de84a71bb7979aa8c089c4fb0ea0e2ed3917df3fb2a287a41aaea54bbad7f5d scipy-1.2.1-cp36-cp36m-win32.whl b2c324ddc5d6dbd3f13680ad16a29425841876a84a1de23a984236d1afff4fa6 scipy-1.2.1-cp36-cp36m-win_amd64.whl 588f9cc4bfab04c45fbd19c1354b5ade377a8124d6151d511c83730a9b6b2338 scipy-1.2.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 628f60be272512ca1123524969649a8cb5ae8b31cca349f7c6f8903daf9034d7 scipy-1.2.1-cp37-cp37m-manylinux1_i686.whl 870fd401ec7b64a895cff8e206ee16569158db00254b2f7157b4c9a5db72c722 scipy-1.2.1-cp37-cp37m-manylinux1_x86_64.whl d78702af4102a3a4e23bb7372cec283e78f32f5573d92091aa6aaba870370fe1 scipy-1.2.1-cp37-cp37m-win32.whl ba0488d4dbba2af5bf9596b849873102d612e49a118c512d9d302ceafa36e01a scipy-1.2.1-cp37-cp37m-win_amd64.whl e085d1babcb419bbe58e2e805ac61924dac4ca45a07c9fa081144739e500aa3c scipy-1.2.1.tar.gz be0e1b322b4b9f2ab88a35db32c494f12639937cf671d4cc210238feb8d5a89d scipy-1.2.1.tar.xz 8140810cf19707c732f330ba6e54968bc85160768850bfbf38d995193e597e2e scipy-1.2.1.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJcXk5KAAoJELD/41ZX0J71zZ0P/3/6ZgXpYGEA013LKNh+/PrV YSA2G7b4jGYExim4cgFHDrwBPoFwiVcfdBTPMxSYn8mtvuqyn4TlcmxqGKQkOwoS zdOEUHdxPFLt8nAy5rWjiCYHzC4qCpGnt+3ROVzRjGA2BLSdzqBg+njT2+9FKNrH 5UoZP4s3TCA8Yn4z2bYV+2viHTAXFsEhHFraNuK5MQu045E57gt5VwYm+ylEeE3Q npcO2U8vio7Hr8ARVOyvQqL8HsQy5/8q8lGTBLbBAP+ujPXi3lRwdb2KqPl3ojfE 7afkEdC3W44CCxPoJfLOdwFnsaOUGVtFeNKmcS2YNWregMpjeG68AjjHO1vCeM34 twNXrwhO2gROwYnp6Q22TTiMsPTTUvce+qGmAyD1d4osOG5zz1A/YQWxHw8Runl0 LPrVt46aOUJTvjCAFs6c7McY04j6lILMyGZlxpyGwLEkP9YAABGvbOtHDMBBWvJs CFxLAo4gEWfJKcJ0bnAKQ9HNeJ45JJ3PQLD5eEmOl+TOQjl0nzY4c7D4LYB0wpl2 NY7tr/TcASX7TlwbjzmlccJUDmNhekcZVTbx65NFjmb2/nUgqAB9ZVjJKyQDpZng rLXZitBmd8IRb8xXp5Smbgt0VccwbMfJF4fulnrVokwILhsIbwZlYQDCG6KOPV27 mObh7S2ZazRQ76Gd4rcu =HBJ8 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Tue Feb 12 17:05:06 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 12 Feb 2019 14:05:06 -0800 Subject: [Numpy-discussion] Reminder: weekly status meeting Feb. 13 at 12:00 Pacific Time Message-ID: Draft agenda: https://hackmd.io/f_e6dnssTkuIC0Pa0TEV6w?view There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. BIDS team -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Tue Feb 12 18:15:49 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Tue, 12 Feb 2019 21:15:49 -0200 Subject: [Numpy-discussion] Porting code for Numpy 1.13+ to get rid of "boolean index did not match indexed array along dimension 1" error Message-ID: Dear ALL, I am trying to port an eigenalysis function that runs smoothly on Numpy 1.12 but fail miserably on Numpy 1.13 or higher with the dreadful error "boolean index did not match indexed array along dimension 1". Here is a fragment of the code, where the error occurrs: evals, evecs = np.linalg.eig(Syy) idx = evals.argsort()[::-1] evals = np.real(evals[idx]) U = np.real(evecs[:, idx]) evals = evals[evals > tolerance] U = U[:, evals > tolerance] # Here is where the error occurs So, I ask: is there a way out of this? Thanks in advance for any assistance you can provide. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Tue Feb 12 20:05:28 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Tue, 12 Feb 2019 17:05:28 -0800 Subject: [Numpy-discussion] Porting code for Numpy 1.13+ to get rid of "boolean index did not match indexed array along dimension 1" error In-Reply-To: References: Message-ID: It looks like your code is wrong, and numpy 1.12 happened to let you get away with it This line: evals = evals[evals > tolerance] Reduces the eigenvalues to only those which are greater than the tolerance When you do U[:, evals > tolerance], evals > tolerance is just going to be [True, True, ...]. You need to swap the last two lines, to U = U[:, evals > tolerance] evals = evals[evals > tolerance] Or better yet, introduce an intermediate variable: keep = evals > tolerance evals = evals[keep] U = U[:, keep] Eric ? On Tue, 12 Feb 2019 at 15:16 Mauro Cavalcanti wrote: > Dear ALL, > > I am trying to port an eigenalysis function that runs smoothly on Numpy > 1.12 but fail miserably on Numpy 1.13 or higher with the dreadful error > "boolean index did not match indexed array along dimension 1". > > Here is a fragment of the code, where the error occurrs: > > evals, evecs = np.linalg.eig(Syy) > idx = evals.argsort()[::-1] > evals = np.real(evals[idx]) > U = np.real(evecs[:, idx]) > evals = evals[evals > tolerance] > U = U[:, evals > tolerance] # Here is where the error occurs > > So, I ask: is there a way out of this? > > Thanks in advance for any assistance you can provide. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Wed Feb 13 05:58:55 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Wed, 13 Feb 2019 08:58:55 -0200 Subject: [Numpy-discussion] Porting code for Numpy 1.13+ to get rid of "boolean index did not match indexed array along dimension 1" error In-Reply-To: References: Message-ID: Eric, Implementing either of your suggestions (swapping the lines or using an intermediate variable) worked fine under the latest Numpy (v1.16.1)! Thanks a lot for your help! Best regards, Em ter, 12 de fev de 2019 ?s 23:06, Eric Wieser escreveu: > It looks like your code is wrong, and numpy 1.12 happened to let you get > away with it > > This line: > > evals = evals[evals > tolerance] > > Reduces the eigenvalues to only those which are greater than the tolerance > > When you do U[:, evals > tolerance], evals > tolerance is just going to > be [True, True, ...]. > > You need to swap the last two lines, to > > U = U[:, evals > tolerance] > evals = evals[evals > tolerance] > > Or better yet, introduce an intermediate variable: > > keep = evals > tolerance > evals = evals[keep] > U = U[:, keep] > > Eric > ? > > On Tue, 12 Feb 2019 at 15:16 Mauro Cavalcanti wrote: > >> Dear ALL, >> >> I am trying to port an eigenalysis function that runs smoothly on Numpy >> 1.12 but fail miserably on Numpy 1.13 or higher with the dreadful error >> "boolean index did not match indexed array along dimension 1". >> >> Here is a fragment of the code, where the error occurrs: >> >> evals, evecs = np.linalg.eig(Syy) >> idx = evals.argsort()[::-1] >> evals = np.real(evals[idx]) >> U = np.real(evecs[:, idx]) >> evals = evals[evals > tolerance] >> U = U[:, evals > tolerance] # Here is where the error occurs >> >> So, I ask: is there a way out of this? >> >> Thanks in advance for any assistance you can provide. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Dr. Mauro J. Cavalcanti E-mail: maurobio at gmail.com Web: http://sites.google.com/site/maurobio "Life is complex. It consists of real and imaginary parts." -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Wed Feb 13 15:34:43 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Wed, 13 Feb 2019 18:34:43 -0200 Subject: [Numpy-discussion] 'nansqrt' function? Message-ID: Dear ALL, In the process of porting an existing (but abandoned) package to the latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' function, which seems not to exist. Here is the specific code: def normTrans(y): denom = np.nansqrt(np.nansum(y**2)) return y/denom As far as I could find, there is no such 'nansqrt' function in the current version of Numpy, so I suspect that the above code has not been properly tested. Am I right, or that function had existed in some past version of Numpy? Thanks in advance for any comments or suggestions. Best regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Feb 13 16:08:20 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Feb 2019 14:08:20 -0700 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: On Wed, Feb 13, 2019 at 1:35 PM Mauro Cavalcanti wrote: > Dear ALL, > > In the process of porting an existing (but abandoned) package to the > latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' > function, which seems not to exist. > > Here is the specific code: > > def normTrans(y): > denom = np.nansqrt(np.nansum(y**2)) > return y/denom > > As far as I could find, there is no such 'nansqrt' function in the current > version of Numpy, so I suspect that the above code has not been properly > tested. > > Am I right, or that function had existed in some past version of Numpy? > > Thanks in advance for any comments or suggestions. > > I don't recall any such function, but nansum will not result in any nans, so plain old sqrt should work. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Wed Feb 13 16:13:26 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Wed, 13 Feb 2019 19:13:26 -0200 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: Chuck, Sure, using numpy.sqrt works fine. Thank you very much. Best regards, Em qua, 13 de fev de 2019 ?s 19:09, Charles R Harris < charlesr.harris at gmail.com> escreveu: > > > On Wed, Feb 13, 2019 at 1:35 PM Mauro Cavalcanti > wrote: > >> Dear ALL, >> >> In the process of porting an existing (but abandoned) package to the >> latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' >> function, which seems not to exist. >> >> Here is the specific code: >> >> def normTrans(y): >> denom = np.nansqrt(np.nansum(y**2)) >> return y/denom >> >> As far as I could find, there is no such 'nansqrt' function in the >> current version of Numpy, so I suspect that the above code has not been >> properly tested. >> >> Am I right, or that function had existed in some past version of Numpy? >> >> Thanks in advance for any comments or suggestions. >> >> > I don't recall any such function, but nansum will not result in any nans, > so plain old sqrt should work. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Dr. Mauro J. Cavalcanti E-mail: maurobio at gmail.com Web: http://sites.google.com/site/maurobio "Life is complex. It consists of real and imaginary parts." -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Feb 13 17:09:46 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Feb 2019 15:09:46 -0700 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: On Wed, Feb 13, 2019 at 2:14 PM Mauro Cavalcanti wrote: > Chuck, > > Sure, using numpy.sqrt works fine. > > Thank you very much. > > Best regards, > > Em qua, 13 de fev de 2019 ?s 19:09, Charles R Harris < > charlesr.harris at gmail.com> escreveu: > >> >> >> On Wed, Feb 13, 2019 at 1:35 PM Mauro Cavalcanti >> wrote: >> >>> Dear ALL, >>> >>> In the process of porting an existing (but abandoned) package to the >>> latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' >>> function, which seems not to exist. >>> >>> Here is the specific code: >>> >>> def normTrans(y): >>> denom = np.nansqrt(np.nansum(y**2)) >>> return y/denom >>> >>> As far as I could find, there is no such 'nansqrt' function in the >>> current version of Numpy, so I suspect that the above code has not been >>> properly tested. >>> >>> Am I right, or that function had existed in some past version of Numpy? >>> >>> Thanks in advance for any comments or suggestions. >>> >>> >> I don't recall any such function, but nansum will not result in any >> nans, so plain old sqrt should work. >> >> Note that there are various nan stat functions: - `nanmin` -- minimum non-NaN value - `nanmax` -- maximum non-NaN value - `nanargmin` -- index of minimum non-NaN value - `nanargmax` -- index of maximum non-NaN value - `nansum` -- sum of non-NaN values - `nanprod` -- product of non-NaN values - `nancumsum` -- cumulative sum of non-NaN values - `nancumprod` -- cumulative product of non-NaN values - `nanmean` -- mean of non-NaN values - `nanvar` -- variance of non-NaN values - `nanstd` -- standard deviation of non-NaN values - `nanmedian` -- median of non-NaN values - `nanquantile` -- qth quantile of non-NaN values - `nanpercentile` -- qth percentile of non-NaN values Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Wed Feb 13 17:44:26 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Wed, 13 Feb 2019 20:44:26 -0200 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: Chuck, I attempted to find such a list from the Numpy website. A complete list like yours should be quite handy for users if available there. Best regards, Em qua, 13 de fev de 2019 ?s 20:10, Charles R Harris < charlesr.harris at gmail.com> escreveu: > > > On Wed, Feb 13, 2019 at 2:14 PM Mauro Cavalcanti > wrote: > >> Chuck, >> >> Sure, using numpy.sqrt works fine. >> >> Thank you very much. >> >> Best regards, >> >> Em qua, 13 de fev de 2019 ?s 19:09, Charles R Harris < >> charlesr.harris at gmail.com> escreveu: >> >>> >>> >>> On Wed, Feb 13, 2019 at 1:35 PM Mauro Cavalcanti >>> wrote: >>> >>>> Dear ALL, >>>> >>>> In the process of porting an existing (but abandoned) package to the >>>> latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' >>>> function, which seems not to exist. >>>> >>>> Here is the specific code: >>>> >>>> def normTrans(y): >>>> denom = np.nansqrt(np.nansum(y**2)) >>>> return y/denom >>>> >>>> As far as I could find, there is no such 'nansqrt' function in the >>>> current version of Numpy, so I suspect that the above code has not been >>>> properly tested. >>>> >>>> Am I right, or that function had existed in some past version of Numpy? >>>> >>>> Thanks in advance for any comments or suggestions. >>>> >>>> >>> I don't recall any such function, but nansum will not result in any >>> nans, so plain old sqrt should work. >>> >>> > Note that there are various nan stat functions: > > - `nanmin` -- minimum non-NaN value > - `nanmax` -- maximum non-NaN value > - `nanargmin` -- index of minimum non-NaN value > - `nanargmax` -- index of maximum non-NaN value > - `nansum` -- sum of non-NaN values > - `nanprod` -- product of non-NaN values > - `nancumsum` -- cumulative sum of non-NaN values > - `nancumprod` -- cumulative product of non-NaN values > - `nanmean` -- mean of non-NaN values > - `nanvar` -- variance of non-NaN values > - `nanstd` -- standard deviation of non-NaN values > - `nanmedian` -- median of non-NaN values > - `nanquantile` -- qth quantile of non-NaN values > - `nanpercentile` -- qth percentile of non-NaN values > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Dr. Mauro J. Cavalcanti E-mail: maurobio at gmail.com Web: http://sites.google.com/site/maurobio "Life is complex. It consists of real and imaginary parts." -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmrsg11 at gmail.com Wed Feb 13 17:50:39 2019 From: tmrsg11 at gmail.com (C W) Date: Wed, 13 Feb 2019 17:50:39 -0500 Subject: [Numpy-discussion] Why slicing Pandas column and then subtract gives NaN? Message-ID: Dear list, I have the following to Pandas Series: a, b. I want to slice and then subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works in Numpy. Example 1: did not work >>>a = pd.Series([85, 86, 87, 86]) >>>b = pd.Series([15, 72, 2, 3]) >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN >>> type(a[1:4]) Example 2: worked If I use values() method, it's converted to a Numpy object. And it works! >>> a.values[1:4]-b.values[0:3] array([71, 15, 84]) >>> type(a.values[1:4]) What's the reason that Pandas in example 1 did not work? Isn't Numpy built on top of Pandas? So, why is everything ok in Numpy, but not in Pandas? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmhobson at gmail.com Wed Feb 13 18:49:03 2019 From: pmhobson at gmail.com (Paul Hobson) Date: Wed, 13 Feb 2019 15:49:03 -0800 Subject: [Numpy-discussion] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: This is more a question for the pandas list, but since i'm here i'll take a crack. - numpy aligns arrays by position. - pandas aligns by label. So what you did in pandas is roughly equivalent to the following: a = pandas.Series([85, 86, 87, 86], name='a').iloc[1:4].to_frame() b = pandas.Series([15, 72, 2, 3], name='b').iloc[0:3].to_frame() result = a.join(b,how='outer').assign(diff=lambda df: df['a'] - df['b']) print(result) a b diff 0 NaN 15.0 NaN 1 86.0 72.0 14.0 2 87.0 2.0 85.0 3 86.0 NaN NaN So what I think you want would be the following: a = pandas.Series([85, 86, 87, 86], name='a') b = pandas.Series([15, 72, 2, 3], name='b') result = a.subtract(b.shift()).dropna() print(result) 1 71.0 2 15.0 3 84.0 dtype: float64 On Wed, Feb 13, 2019 at 2:51 PM C W wrote: > Dear list, > > I have the following to Pandas Series: a, b. I want to slice and then > subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works > in Numpy. > > Example 1: did not work > >>>a = pd.Series([85, 86, 87, 86]) > >>>b = pd.Series([15, 72, 2, 3]) > >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN > >>> type(a[1:4]) > > > Example 2: worked > If I use values() method, it's converted to a Numpy object. And it works! > >>> a.values[1:4]-b.values[0:3] > array([71, 15, 84]) > >>> type(a.values[1:4]) > > > What's the reason that Pandas in example 1 did not work? Isn't Numpy built > on top of Pandas? So, why is everything ok in Numpy, but not in Pandas? > > Thanks in advance! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbmcleod at gmail.com Wed Feb 13 19:09:46 2019 From: robbmcleod at gmail.com (Robert McLeod) Date: Wed, 13 Feb 2019 16:09:46 -0800 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: The `bottleneck` library is a very good package if there's some function in NumPy that you want to handle `nan`s in reductions without exploding. https://github.com/kwgoodman/bottleneck On Wed, Feb 13, 2019 at 12:35 PM Mauro Cavalcanti wrote: > Dear ALL, > > In the process of porting an existing (but abandoned) package to the > latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' > function, which seems not to exist. > > Here is the specific code: > > def normTrans(y): > denom = np.nansqrt(np.nansum(y**2)) > return y/denom > > As far as I could find, there is no such 'nansqrt' function in the current > version of Numpy, so I suspect that the above code has not been properly > tested. > > Am I right, or that function had existed in some past version of Numpy? > > Thanks in advance for any comments or suggestions. > > Best regards, > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Robert McLeod, Ph.D. robbmcleod at gmail.com robbmcleod at protonmail.com robert.mcleod at hitachi-hhtc.ca www.entropyreduction.al -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Wed Feb 13 19:21:27 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Wed, 13 Feb 2019 22:21:27 -0200 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: Thanks! I didn't heard of this package and will look into it. Best regards, Em Qua, 13 de fev de 2019 22:10, Robert McLeod The `bottleneck` library is a very good package if there's some function > in NumPy that you want to handle `nan`s in reductions without exploding. > > https://github.com/kwgoodman/bottleneck > > On Wed, Feb 13, 2019 at 12:35 PM Mauro Cavalcanti > wrote: > >> Dear ALL, >> >> In the process of porting an existing (but abandoned) package to the >> latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' >> function, which seems not to exist. >> >> Here is the specific code: >> >> def normTrans(y): >> denom = np.nansqrt(np.nansum(y**2)) >> return y/denom >> >> As far as I could find, there is no such 'nansqrt' function in the >> current version of Numpy, so I suspect that the above code has not been >> properly tested. >> >> Am I right, or that function had existed in some past version of Numpy? >> >> Thanks in advance for any comments or suggestions. >> >> Best regards, >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > -- > Robert McLeod, Ph.D. > robbmcleod at gmail.com > robbmcleod at protonmail.com > robert.mcleod at hitachi-hhtc.ca > www.entropyreduction.al > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Feb 13 21:43:50 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Feb 2019 19:43:50 -0700 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: On Wed, Feb 13, 2019 at 3:45 PM Mauro Cavalcanti wrote: > Chuck, > > I attempted to find such a list from the Numpy website. A complete list > like yours should be quite handy for users if available there. > > In ipython In [1]: numpy.lib.nanfunctions? will give it to you. But it looks like a module level entry should be added to the documentation under "NaN functions (numpy.lib.nanfunctions)" in a "Routines" entry. Maybe "Histograms" also. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurobio at gmail.com Thu Feb 14 04:45:44 2019 From: maurobio at gmail.com (Mauro Cavalcanti) Date: Thu, 14 Feb 2019 07:45:44 -0200 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: Chuck, IPython is full of secrets! More traditional users (myself included) usually look for the official documentation, so it would be really useful if such hints were also available there. Best regards, Em qui, 14 de fev de 2019 ?s 00:44, Charles R Harris < charlesr.harris at gmail.com> escreveu: > > > On Wed, Feb 13, 2019 at 3:45 PM Mauro Cavalcanti > wrote: > >> Chuck, >> >> I attempted to find such a list from the Numpy website. A complete list >> like yours should be quite handy for users if available there. >> >> > In ipython > > In [1]: numpy.lib.nanfunctions? > > will give it to you. But it looks like a module level entry should be > added to the documentation under "NaN functions (numpy.lib.nanfunctions)" > in a "Routines" entry. Maybe "Histograms" also. > > > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Dr. Mauro J. Cavalcanti E-mail: maurobio at gmail.com Web: http://sites.google.com/site/maurobio "Life is complex. It consists of real and imaginary parts." -------------- next part -------------- An HTML attachment was scrubbed... URL: From deak.andris at gmail.com Thu Feb 14 05:07:12 2019 From: deak.andris at gmail.com (Andras Deak) Date: Thu, 14 Feb 2019 11:07:12 +0100 Subject: [Numpy-discussion] 'nansqrt' function? In-Reply-To: References: Message-ID: On Thu, Feb 14, 2019 at 10:46 AM Mauro Cavalcanti wrote: > > Chuck, > > IPython is full of secrets! More traditional users (myself included) usually look for the official documentation, so it would be really useful if such hints were also available there. Sorry if I'm stating the obvious, but what ipython does is merely give you the appropriate docstring, in this case the module docstring of numpy.lib.nanfunctions (https://github.com/numpy/numpy/blob/master/numpy/lib/nanfunctions.py#L1-L22). So you have to know where to look to find it, but it's official. I believe Chuck's remark of "looks like a module level entry should be added to the documentation under 'NaN functions (numpy.lib.nanfunctions)'" suggests exactly to have this information in the online docs, somewhere at https://docs.scipy.org/doc/numpy/reference/routines.html Regards, Andr?s > > Best regards, > > Em qui, 14 de fev de 2019 ?s 00:44, Charles R Harris escreveu: >> >> >> >> On Wed, Feb 13, 2019 at 3:45 PM Mauro Cavalcanti wrote: >>> >>> Chuck, >>> >>> I attempted to find such a list from the Numpy website. A complete list like yours should be quite handy for users if available there. >>> >> >> In ipython >> >> In [1]: numpy.lib.nanfunctions? >> >> will give it to you. But it looks like a module level entry should be added to the documentation under "NaN functions (numpy.lib.nanfunctions)" in a "Routines" entry. Maybe "Histograms" also. >> >> >> >> Chuck >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > > > -- > Dr. Mauro J. Cavalcanti > E-mail: maurobio at gmail.com > Web: http://sites.google.com/site/maurobio > "Life is complex. It consists of real and imaginary parts." > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From tmrsg11 at gmail.com Thu Feb 14 15:20:20 2019 From: tmrsg11 at gmail.com (C W) Date: Thu, 14 Feb 2019 15:20:20 -0500 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: Hi Paul, Thanks for your response! I did not find a Pandas list for users, only for developers. I'd love to be on there. result = a.subtract(b.shift()).dropna() This seems verbose, several layers of parenthesis follow by a dot method. I'm new to Python, I thought Python code would be pity and short. Is this what everyone will write? Thank you! On Wed, Feb 13, 2019 at 6:50 PM Paul Hobson wrote: > This is more a question for the pandas list, but since i'm here i'll take > a crack. > > > - numpy aligns arrays by position. > - pandas aligns by label. > > So what you did in pandas is roughly equivalent to the following: > > a = pandas.Series([85, 86, 87, 86], name='a').iloc[1:4].to_frame() > b = pandas.Series([15, 72, 2, 3], name='b').iloc[0:3].to_frame() > result = a.join(b,how='outer').assign(diff=lambda df: df['a'] - df['b']) > print(result) > > a b diff > 0 NaN 15.0 NaN > 1 86.0 72.0 14.0 > 2 87.0 2.0 85.0 > 3 86.0 NaN NaN > > So what I think you want would be the following: > > a = pandas.Series([85, 86, 87, 86], name='a') > b = pandas.Series([15, 72, 2, 3], name='b') > result = a.subtract(b.shift()).dropna() > print(result) > 1 71.0 > 2 15.0 > 3 84.0 > dtype: float64 > > > > On Wed, Feb 13, 2019 at 2:51 PM C W wrote: > >> Dear list, >> >> I have the following to Pandas Series: a, b. I want to slice and then >> subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works >> in Numpy. >> >> Example 1: did not work >> >>>a = pd.Series([85, 86, 87, 86]) >> >>>b = pd.Series([15, 72, 2, 3]) >> >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN >> >>> type(a[1:4]) >> >> >> Example 2: worked >> If I use values() method, it's converted to a Numpy object. And it works! >> >>> a.values[1:4]-b.values[0:3] >> array([71, 15, 84]) >> >>> type(a.values[1:4]) >> >> >> What's the reason that Pandas in example 1 did not work? Isn't Numpy >> built on top of Pandas? So, why is everything ok in Numpy, but not in >> Pandas? >> >> Thanks in advance! >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at python.org > https://mail.python.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmrsg11 at gmail.com Fri Feb 15 01:02:01 2019 From: tmrsg11 at gmail.com (Mike C) Date: Fri, 15 Feb 2019 06:02:01 +0000 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: , Message-ID: Thanks a lot, Thomas. I don?t have index when I read in the data. I just want to slice two series to the same length, and subtract. That?s it! I also don?t what numpy methods wrapped within methods. They work, but hard do understand. How would you do it? In Matlab or R, it?s very simple, one line. ________________________________ From: SciPy-User on behalf of Thomas Kluyver Sent: Thursday, February 14, 2019 4:54 PM To: SciPy Users List Cc: Discussion of Numerical Python Subject: Re: [SciPy-User] [Numpy-discussion] Why slicing Pandas column and then subtract gives NaN? Maybe it's useful to look a bit more at what pandas is doing and why. The 'index' on a series or dataframe labels each row - e.g. if your series is measuring total sales for each day, its index would be the dates. When you combine (e.g. subtract) two series, pandas automatically lines up the indices. So it will join up the numbers for February 14th, even if they're not in the same position in the data. In your example, you haven't specified an index, so pandas generates an integer index which doesn't really mean anything, and aligning on it doesn't do what you want. What are you trying to do? If Numpy does exactly what you want, then the answer might be to use Numpy. > Isn't Numpy built on top of Pandas? It's the other way round: pandas is built on Numpy. Pandas indices are an extra layer of functionality on top of what Numpy does. On Thu, 14 Feb 2019 at 20:22, C W > wrote: Hi Paul, Thanks for your response! I did not find a Pandas list for users, only for developers. I'd love to be on there. result = a.subtract(b.shift()).dropna() This seems verbose, several layers of parenthesis follow by a dot method. I'm new to Python, I thought Python code would be pity and short. Is this what everyone will write? Thank you! On Wed, Feb 13, 2019 at 6:50 PM Paul Hobson > wrote: This is more a question for the pandas list, but since i'm here i'll take a crack. * numpy aligns arrays by position. * pandas aligns by label. So what you did in pandas is roughly equivalent to the following: a = pandas.Series([85, 86, 87, 86], name='a').iloc[1:4].to_frame() b = pandas.Series([15, 72, 2, 3], name='b').iloc[0:3].to_frame() result = a.join(b,how='outer').assign(diff=lambda df: df['a'] - df['b']) print(result) a b diff 0 NaN 15.0 NaN 1 86.0 72.0 14.0 2 87.0 2.0 85.0 3 86.0 NaN NaN So what I think you want would be the following: a = pandas.Series([85, 86, 87, 86], name='a') b = pandas.Series([15, 72, 2, 3], name='b') result = a.subtract(b.shift()).dropna() print(result) 1 71.0 2 15.0 3 84.0 dtype: float64 On Wed, Feb 13, 2019 at 2:51 PM C W > wrote: Dear list, I have the following to Pandas Series: a, b. I want to slice and then subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works in Numpy. Example 1: did not work >>>a = pd.Series([85, 86, 87, 86]) >>>b = pd.Series([15, 72, 2, 3]) >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN >>> type(a[1:4]) Example 2: worked If I use values() method, it's converted to a Numpy object. And it works! >>> a.values[1:4]-b.values[0:3] array([71, 15, 84]) >>> type(a.values[1:4]) What's the reason that Pandas in example 1 did not work? Isn't Numpy built on top of Pandas? So, why is everything ok in Numpy, but not in Pandas? Thanks in advance! _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ SciPy-User mailing list SciPy-User at python.org https://mail.python.org/mailman/listinfo/scipy-user _______________________________________________ SciPy-User mailing list SciPy-User at python.org https://mail.python.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Fri Feb 15 04:14:09 2019 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Fri, 15 Feb 2019 20:14:09 +1100 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: > I don?t have index when I read in the data. I just want to slice two series to the same length, and subtract. That?s it! > > I also don?t what numpy methods wrapped within methods. They work, but hard do understand. > > How would you do it? In Matlab or R, it?s very simple, one line. Why are you using pandas at all? If you want the Matlab equivalent, use NumPy from the beginning (or as soon as possible). I personally agree with you that pandas is too verbose, which is why I mostly use NumPy for this kind of arithmetic, and reserve pandas for advanced data table type functionality (like groupbys and joining on indices). As you saw yourself, a.values[1:4] - b.values[0:3] works great. If you read in your data into NumPy from the beginning, it?ll be a[1:4] - b[0:3] just like in Matlab. (Or even better: a[1:] - b[:-1]). -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmrsg11 at gmail.com Fri Feb 15 08:11:37 2019 From: tmrsg11 at gmail.com (Mike C) Date: Fri, 15 Feb 2019 13:11:37 +0000 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: , Message-ID: The original data was in CSV format. I read it in using pd.read_csv(). It does have column names, but no row names. I don?t think numpy reads csv files. And also, when I do a[2:5]-b[:3], it does not throw any ?index out of range? error. I was able to catch that, but in both Matlab and R. You get an error. This is frustrating!! ________________________________ From: NumPy-Discussion on behalf of Juan Nunez-Iglesias Sent: Friday, February 15, 2019 4:15 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? I don?t have index when I read in the data. I just want to slice two series to the same length, and subtract. That?s it! I also don?t what numpy methods wrapped within methods. They work, but hard do understand. How would you do it? In Matlab or R, it?s very simple, one line. Why are you using pandas at all? If you want the Matlab equivalent, use NumPy from the beginning (or as soon as possible). I personally agree with you that pandas is too verbose, which is why I mostly use NumPy for this kind of arithmetic, and reserve pandas for advanced data table type functionality (like groupbys and joining on indices). As you saw yourself, a.values[1:4] - b.values[0:3] works great. If you read in your data into NumPy from the beginning, it?ll be a[1:4] - b[0:3] just like in Matlab. (Or even better: a[1:] - b[:-1]). -------------- next part -------------- An HTML attachment was scrubbed... URL: From deak.andris at gmail.com Fri Feb 15 12:49:07 2019 From: deak.andris at gmail.com (Andras Deak) Date: Fri, 15 Feb 2019 18:49:07 +0100 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: > The original data was in CSV format. I read it in using pd.read_csv(). It does have column names, but no row names. I don?t think numpy reads csv files I routinely read csv files using numpy.loadtxt https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html > And also, when I do a[2:5]-b[:3], it does not throw any ?index out of range? error. I was able to catch that, but in both Matlab and R. You get an error. This is frustrating!! That's basic slicing behaviour of python. You might like it or not, but it's baked into the language: >>> [1,2][:10], [1,2][5:7] ([1, 2], []) One would need very good reasons to break this in case of a third-party library. Andr?s > ________________________________ > From: NumPy-Discussion on behalf of Juan Nunez-Iglesias > Sent: Friday, February 15, 2019 4:15 AM > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? > > > I don?t have index when I read in the data. I just want to slice two series to the same length, and subtract. That?s it! > > I also don?t what numpy methods wrapped within methods. They work, but hard do understand. > > How would you do it? In Matlab or R, it?s very simple, one line. > > > Why are you using pandas at all? If you want the Matlab equivalent, use NumPy from the beginning (or as soon as possible). I personally agree with you that pandas is too verbose, which is why I mostly use NumPy for this kind of arithmetic, and reserve pandas for advanced data table type functionality (like groupbys and joining on indices). > > As you saw yourself, a.values[1:4] - b.values[0:3] works great. If you read in your data into NumPy from the beginning, it?ll be a[1:4] - b[0:3] just like in Matlab. (Or even better: a[1:] - b[:-1]). > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From pmhobson at gmail.com Fri Feb 15 12:50:44 2019 From: pmhobson at gmail.com (Paul Hobson) Date: Fri, 15 Feb 2019 09:50:44 -0800 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: On Fri, Feb 15, 2019 at 5:12 AM Mike C wrote: > The original data was in CSV format. I read it in using pd.read_csv(). It > does have column names, but no row names. I don?t think numpy reads csv > files. > If you read a file into a pandas structure, it will have row labels. The default labels are integers that correspond to the ordinal positions of the values. Numpy reads files. https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.loadtxt.html https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.genfromtxt.html I prefer file IO in pandas, so I don't know which function will better suite your needs. And also, when I do a[2:5]-b[:3], it does not throw any ?index out of > range? error. I was able to catch that, but in both Matlab and R. You get > an error. This is frustrating!! > That's a feature of python in general, not numpy in particular. Every language has its own quirks. The more you immerse yourself in them, the quick you'll learn to adapt to them. -paul > > > ------------------------------ > *From:* NumPy-Discussion gmail.com at python.org> on behalf of Juan Nunez-Iglesias > > *Sent:* Friday, February 15, 2019 4:15 AM > *To:* Discussion of Numerical Python > *Subject:* Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column > and then subtract gives NaN? > > > I don?t have index when I read in the data. I just want to slice two > series to the same length, and subtract. That?s it! > > I also don?t what numpy methods wrapped within methods. They work, but > hard do understand. > > How would you do it? In Matlab or R, it?s very simple, one line. > > > Why are you using pandas at all? If you want the Matlab equivalent, use > NumPy from the beginning (or as soon as possible). I personally agree with > you that pandas is too verbose, which is why I mostly use NumPy for this > kind of arithmetic, and reserve pandas for advanced data table type > functionality (like groupbys and joining on indices). > > As you saw yourself, a.values[1:4] - b.values[0:3] works great. If you > read in your data into NumPy from the beginning, it?ll be a[1:4] - b[0:3] > just like in Matlab. (Or even better: a[1:] - b[:-1]). > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmrsg11 at gmail.com Fri Feb 15 16:48:25 2019 From: tmrsg11 at gmail.com (C W) Date: Fri, 15 Feb 2019 16:48:25 -0500 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: Fair enough. Python has been called the #1 language for data science. If I'm slicing a[2:5] out of range, why not throw an error. This is disappointing! I mean, why would you design a language to slice outside of range? Also, no other language I know have this strange behavior. On Fri, Feb 15, 2019 at 12:51 PM Paul Hobson wrote: > > > On Fri, Feb 15, 2019 at 5:12 AM Mike C wrote: > >> The original data was in CSV format. I read it in using pd.read_csv(). It >> does have column names, but no row names. I don?t think numpy reads csv >> files. >> > > If you read a file into a pandas structure, it will have row labels. The > default labels are integers that correspond to the ordinal positions of the > values. > > Numpy reads files. > > https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.loadtxt.html > > https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.genfromtxt.html > > I prefer file IO in pandas, so I don't know which function will better > suite your needs. > > And also, when I do a[2:5]-b[:3], it does not throw any ?index out of >> range? error. I was able to catch that, but in both Matlab and R. You get >> an error. This is frustrating!! >> > > That's a feature of python in general, not numpy in particular. Every > language has its own quirks. The more you immerse yourself in them, the > quick you'll learn to adapt to them. > -paul > > >> >> >> ------------------------------ >> *From:* NumPy-Discussion > gmail.com at python.org> on behalf of Juan Nunez-Iglesias < >> jni.soma at gmail.com> >> *Sent:* Friday, February 15, 2019 4:15 AM >> *To:* Discussion of Numerical Python >> *Subject:* Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column >> and then subtract gives NaN? >> >> >> I don?t have index when I read in the data. I just want to slice two >> series to the same length, and subtract. That?s it! >> >> I also don?t what numpy methods wrapped within methods. They work, but >> hard do understand. >> >> How would you do it? In Matlab or R, it?s very simple, one line. >> >> >> Why are you using pandas at all? If you want the Matlab equivalent, use >> NumPy from the beginning (or as soon as possible). I personally agree with >> you that pandas is too verbose, which is why I mostly use NumPy for this >> kind of arithmetic, and reserve pandas for advanced data table type >> functionality (like groupbys and joining on indices). >> >> As you saw yourself, a.values[1:4] - b.values[0:3] works great. If you >> read in your data into NumPy from the beginning, it?ll be a[1:4] - b[0:3] >> just like in Matlab. (Or even better: a[1:] - b[:-1]). >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele at grinta.net Fri Feb 15 18:51:35 2019 From: daniele at grinta.net (Daniele Nicolodi) Date: Fri, 15 Feb 2019 16:51:35 -0700 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: On 15-02-2019 14:48, C W wrote: > Fair enough. Python has been called the #1 language for data science. If > I'm slicing?a[2:5] out of range, why not throw an error. This is> disappointing! No one here is trying to convince you to use Python. If you don't like it, don't use it. Complain in this venue about how you don't like the language is not productive and it is not going to change Python's (or Numpy's) design. I suggest you to instead invest the time to understand why things work the way they do. Cheers, Dan From tmrsg11 at gmail.com Sat Feb 16 23:40:39 2019 From: tmrsg11 at gmail.com (C W) Date: Sat, 16 Feb 2019 23:40:39 -0500 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: Dan, > > No one here is trying to convince you to use Python. If you don't like > it, don't use it. > The problem is not me, it's the language. No need to take it out on me personally. I've used other languages, Python is lacking in this area. I'm being very frank here, just think about it. On Fri, Feb 15, 2019 at 6:53 PM Daniele Nicolodi wrote: > On 15-02-2019 14:48, C W wrote: > > Fair enough. Python has been called the #1 language for data science. If > > I'm slicing a[2:5] out of range, why not throw an error. This is> > disappointing! > > No one here is trying to convince you to use Python. If you don't like > it, don't use it. Complain in this venue about how you don't like the > language is not productive and it is not going to change Python's (or > Numpy's) design. I suggest you to instead invest the time to understand > why things work the way they do. > > Cheers, > Dan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Sun Feb 17 12:12:17 2019 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 17 Feb 2019 19:12:17 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Tues Feb 19 12:00 PST Message-ID: <94bfa41d-d3c2-d4f6-a246-15464295843c@gmail.com> This week the status meeting will be on Tues Feb 19 at 12:00 PST, not at the usual time The draft agenda is at?https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both Everyone is invited to join. Matti, Tyler and Stefan From robert.kern at gmail.com Sun Feb 17 15:14:34 2019 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 17 Feb 2019 12:14:34 -0800 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? In-Reply-To: References: Message-ID: On Sat, Feb 16, 2019 at 8:42 PM C W wrote: > Dan, >> >> No one here is trying to convince you to use Python. If you don't like >> it, don't use it. >> > The problem is not me, it's the language. No need to take it out on me > personally. I've used other languages, Python is lacking in this area. I'm > being very frank here, just think about it. > No one was taking it out on you personally. We're just stating that we're not interested in having a discussion about which semantics are best, much less convincing you that Python's choice is the right one. They have been that way for a long time, and the time for making those decisions is long past. We could not change them now if we wanted to. Empirically, I've been on these lists for about 20 years now, and I have not seen this pop up as a frequent issue causing bugs in real code, so I would submit that if there is a lack (or benefit) compared to other languages, it is small. That said, Python is not alone here. Perl and Ruby have Python's semantics. R introduces NaNs but does not raise an error. Matlab and Julia do raise errors. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From avigross at verizon.net Sun Feb 17 19:24:17 2019 From: avigross at verizon.net (Avi Gross) Date: Sun, 17 Feb 2019 19:24:17 -0500 Subject: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? Message-ID: <003f01d4c720$492b1c70$db815550$@verizon.net> Changing horses mid-stream is not generally a good idea. But python is easy to extend so there may be other choices out there. The reality is that what matters more than what choices a language makes is consistency across the language. If you can learn the choices and even quirks and work with them, you can get things done. I have used languages where vectors or lists are zero-based and some where they are one-based. Both work, albeit you may need to adjust an algorithm to add or subtract one from the index you use or leave the first entry empty. If you use a pandas data structure without specifying your own index, you get stuck with the default behavior. If you wanted a 1 or 0 based index, you could add that carefully and so on. In theory, you could create an alternate numpy or alternate pandas just like many of the alternate modules you can find which often have very different designs for graphics and there probably already are some out there. Integrating them with other tools may not be as simple. If you want to use the machine learning tools available, some functions demand they be given the data as a pandas DataFrame or a numpy array. Giving them something else like a python list, even if it is just an object extended from the above, may well break things. This makes it very hard to port some things to or from other languages with different design constraints as an algorithm that looks straightforward in one because it goes with the overall grain of the language may need major surgery to fit into the other and perhaps might best be rewritten using some other algorithm that fits better. A different discussion elsewhere about errors was instructive. Many programmers used to write complex code that checks for all kinds of errors or conditions. Some python programmers take an approach of expecting errors to be something you catch so why bother checking for things like an array index being non-existent. Well, if an error you expect in pandas is not caught, consider checking before using software that does not enforce it. And some "errors" are not errors. As mentioned, in R it is NOT a design error that using an NA value in a calculation silently propagates the NA. Throughout the language you must either check if there is an NA using the is.na() function or by telling routines to drop/ignore any NA as in sum(something, na.rm=TRUE). There are many design decisions you make this way. So, if your language does something you don't want when it uses an implicit index, fine. Give it an explicit index. If it does not properly check the range you provide for validity, do it yourself. And if the language is so unsuitable for your needs, you can often switch. In particular, there are now even ways you can start your code in one language like python or R and then shunt some data structures across a divide where the other language can do things another way and if needed, shunt it back and repeat. That may not work well for your application and, as noted, switching rides in midstream has its dangers. From: NumPy-Discussion On Behalf Of Robert Kern Sent: Sunday, February 17, 2019 3:15 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? On Sat, Feb 16, 2019 at 8:42 PM C W > wrote: Dan, No one here is trying to convince you to use Python. If you don't like it, don't use it. The problem is not me, it's the language. No need to take it out on me personally. I've used other languages, Python is lacking in this area. I'm being very frank here, just think about it. No one was taking it out on you personally. We're just stating that we're not interested in having a discussion about which semantics are best, much less convincing you that Python's choice is the right one. They have been that way for a long time, and the time for making those decisions is long past. We could not change them now if we wanted to. Empirically, I've been on these lists for about 20 years now, and I have not seen this pop up as a frequent issue causing bugs in real code, so I would submit that if there is a lack (or benefit) compared to other languages, it is small That said, Python is not alone here. Perl and Ruby have Python's semantics. R introduces NaNs but does not raise an error. Matlab and Julia do raise errors. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Feb 18 12:56:08 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 18 Feb 2019 09:56:08 -0800 Subject: [Numpy-discussion] Stepping down as NumFOCUS liason Message-ID: Hi all, As part of our relationship with NumFOCUS, we have to have some liasons so they know who to contact etc. So far that's been Ralf and me. But, I haven't had as much time for numpy recently, and I think it would be better to hand this off to someone who's more active. Is anyone interested? Someone on the steering council would probably be most natural. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Feb 19 19:32:40 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Feb 2019 17:32:40 -0700 Subject: [Numpy-discussion] Stepping down as NumFOCUS liason In-Reply-To: References: Message-ID: Hi Nathaniel, On Mon, Feb 18, 2019 at 11:19 AM Nathaniel Smith wrote: > Hi all, > > As part of our relationship with NumFOCUS, we have to have some liasons so > they know who to contact etc. So far that's been Ralf and me. But, I > haven't had as much time for numpy recently, and I think it would be better > to hand this off to someone who's more active. Is anyone interested? > Someone on the steering council would probably be most natural. > Thanks for the ping. I hope you are doing well, we miss your input. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Feb 21 16:37:52 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 21 Feb 2019 22:37:52 +0100 Subject: [Numpy-discussion] Adding count parameter to unpackbits (PR-10855) Message-ID: <7f0f13de5c2b0a12f4c8e672d5b7cbeaf1ae4a48.camel@sipsolutions.net> Hi all, I was about to merge it, but was noot sure it was really discussed before (and it would be a long time ago). The pull requests: https://github.com/numpy/numpy/pull/10855 Proposes to add a count parameter to unpackbits: ``count`` allows subsetting the number of bits that will be unpacked up-front, rather than reshaping and subsetting later, making the ``packbits`` operation invertible, and the unpacking less wasteful. Counts larger than the number of available bits add zero padding. Negative counts trim bits off the end instead of counting from the beginning. None counts implement the existing behavior of unpacking everything. This seems like a pretty small API addition, but I thought I would ping the list again before merging in case anyone objects to it or feels it should be modified. In other words, the count parameter is much the same as indexing the result: np.unpackbits(x)[:count] although it will zero pad if count is out of bounds. Which is necessary if it is to invert a `np.packbits` when less than 8 bits are packed. If no one speaks up, I plan to merge it soon. Best, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Thu Feb 21 20:51:50 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 21 Feb 2019 17:51:50 -0800 Subject: [Numpy-discussion] Stepping down as NumFOCUS liason In-Reply-To: References: Message-ID: On Tue, Feb 19, 2019 at 4:33 PM Charles R Harris wrote: > Hi Nathaniel, > > On Mon, Feb 18, 2019 at 11:19 AM Nathaniel Smith wrote: > >> Hi all, >> >> As part of our relationship with NumFOCUS, we have to have some liasons >> so they know who to contact etc. So far that's been Ralf and me. But, I >> haven't had as much time for numpy recently, and I think it would be better >> to hand this off to someone who's more active. Is anyone interested? >> Someone on the steering council would probably be most natural. >> > > Thanks for the ping. I hope you are doing well, we miss your input. > +1 to that. And thanks for being one of the NumFOCUS contacts until now Nathaniel. Any other takers? It's very little work. We get a monthly email from Gina asking for input on the NumFOCUS projects newsletter (new releases, sprints, etc.), and an occasional request that needs answering or forwarding to the steering committee. So far I think we've only sent one serious request to NumFOCUS - a check on a license thing with MKL I believe. Finally you can just the NumFOCUS projects mailing lists and Slack channel, which is quite informative and interesting (sharing by other projects of relevant things mainly). Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Feb 26 15:00:17 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Feb 2019 13:00:17 -0700 Subject: [Numpy-discussion] NumPy 1.16.2 released. Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.16.2. This is a quick release fixing several problems encountered on Windows. The Python versions supported are 2.7 and 3.5-3.7. The Windows problems addressed are: - DLL load problems for NumPy wheels on Windows, - distutils command line parsing on Windows. There is also a regression fix correcting signed zeros produced by divmod, see the release notes for details. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.4. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ``ImportError``. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 for discussion of the issue. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 5 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Eric Wieser * Matti Picus * Tyler Reddy * Tony LaTorre + *Pull requests merged* A total of 7 pull requests were merged for this release. * #12909: TST: fix vmImage dispatch in Azure * #12923: MAINT: remove complicated test of multiarray import failure mode * #13020: BUG: fix signed zero behavior in npy_divmod * #13026: MAINT: Add functions to parse shell-strings in the platform-native... * #13028: BUG: Fix regression in parsing of F90 and F77 environment variables * #13038: BUG: parse shell escaping in extra_compile_args and extra_link_args * #13041: BLD: Windows absolute path DLL loading Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Feb 26 18:36:53 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 27 Feb 2019 01:36:53 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Feb 27 at 12:00 pacific time Message-ID: The draft agenda is at https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. Past meeting notes can be seen at https://github.com/BIDS-numpy/docs/tree/master/status_meetings The BIDS team From stefanv at berkeley.edu Wed Feb 27 01:19:15 2019 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 26 Feb 2019 22:19:15 -0800 Subject: [Numpy-discussion] Outreachy internship Message-ID: <20190227061915.yodq75qoqrfox2ts@carbo> Hi everyone, The team at BIDS would like to take on an intern from Outreachy (https://www.outreachy.org), as part of our effort to grow the NumPy developer community. The internship is similar to a GSoC, except that we will pay their salary, as well as do the mentoring. We wanted to check in beforehand to ensure that everyone is comfortable with us doing this on behalf of the NumPy community. Your thoughts and recommendations are welcome! Best regards, St?fan From ralf.gommers at gmail.com Wed Feb 27 01:59:34 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 26 Feb 2019 22:59:34 -0800 Subject: [Numpy-discussion] Outreachy internship In-Reply-To: <20190227061915.yodq75qoqrfox2ts@carbo> References: <20190227061915.yodq75qoqrfox2ts@carbo> Message-ID: On Tue, Feb 26, 2019 at 10:19 PM Stefan van der Walt wrote: > Hi everyone, > > The team at BIDS would like to take on an intern from Outreachy > (https://www.outreachy.org), as part of our effort to grow the NumPy > developer community. > > The internship is similar to a GSoC, except that we will pay their > salary, as well as do the mentoring. We wanted to check in beforehand > to ensure that everyone is comfortable with us doing this on behalf of > the NumPy community. > > Your thoughts and recommendations are welcome! > Sounds like a great idea to me! Unlike for GSoC, you can look for a good candidate first and see what their interests are - could work better, GSoC like ideas have proven to be pretty hard to come by for NumPy. And arguably tackling different smaller tasks could be more educational. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Feb 27 02:49:53 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 26 Feb 2019 23:49:53 -0800 Subject: [Numpy-discussion] Outreachy internship In-Reply-To: References: <20190227061915.yodq75qoqrfox2ts@carbo> Message-ID: This sounds great to me as well! Thank you for your willingness to take this on. On Tue, Feb 26, 2019 at 10:59 PM Ralf Gommers wrote: > On Tue, Feb 26, 2019 at 10:19 PM Stefan van der Walt > wrote: > >> Hi everyone, >> >> The team at BIDS would like to take on an intern from Outreachy >> (https://www.outreachy.org), as part of our effort to grow the NumPy >> developer community. >> >> The internship is similar to a GSoC, except that we will pay their >> salary, as well as do the mentoring. We wanted to check in beforehand >> to ensure that everyone is comfortable with us doing this on behalf of >> the NumPy community. >> >> Your thoughts and recommendations are welcome! >> > > Sounds like a great idea to me! > > Unlike for GSoC, you can look for a good candidate first and see what > their interests are - could work better, GSoC like ideas have proven to be > pretty hard to come by for NumPy. And arguably tackling different smaller > tasks could be more educational. > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Feb 27 03:20:41 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 27 Feb 2019 10:20:41 +0200 Subject: [Numpy-discussion] Removing undocumented __buffer__ attribute lookup Message-ID: <210da5f6-cf94-79cc-4b74-d798f96196c0@gmail.com> In digging around the code, I found a gem in PyArray_FromBuffer (exposed to python as numpy.frombuffer). If a PyObject* does not have a tp_as_buffer->bf_getbuffer function, we check if the python object has a __buffer__ attribute. If so we use that as buf in PyObject_GetBuffer(buf, ...). This seems to stem back to the original numerics code, where getBuffer would look up the attribute and call it as a method. PyArray_FromBuffer does not call the attribute as a method, it simply passes it on to PyObject_GetBuffer, which will then raise an error saying it cannot convert a method. You can try this out by creating a class with a __buffer__ method and calling numpy.frombuffer on it. I submitted a pull request to remove the code. Since it is undocumented and (as far as I can tell) broken, I do not think we need a deprecation cycle. More details, including links to the original numeric code from 2005, in the PR https://github.com/numpy/numpy/pull/13049 Any thoughts or objections? Matti From einstein.edison at gmail.com Wed Feb 27 03:37:32 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 27 Feb 2019 09:37:32 +0100 Subject: [Numpy-discussion] Removing undocumented __buffer__ attribute lookup In-Reply-To: <210da5f6-cf94-79cc-4b74-d798f96196c0@gmail.com> References: <210da5f6-cf94-79cc-4b74-d798f96196c0@gmail.com> Message-ID: <97dd993a-a4d3-498d-b6df-1471fd1b4d98@Canary> Cc-ing in Travis, because he was the original author of the buffer protocol, and this is most definitely related. Best Regards, Hameer Abbasi > On Wednesday, Feb 27, 2019 at 9:20 AM, Matti Picus wrote: > In digging around the code, I found a gem in PyArray_FromBuffer (exposed > to python as numpy.frombuffer). If a PyObject* does not have a > tp_as_buffer->bf_getbuffer function, we check if the python object has a > __buffer__ attribute. If so we use that as buf in > PyObject_GetBuffer(buf, ...). > > > This seems to stem back to the original numerics code, where getBuffer > would look up the attribute and call it as a method. PyArray_FromBuffer > does not call the attribute as a method, it simply passes it on to > PyObject_GetBuffer, which will then raise an error saying it cannot > convert a method. You can try this out by creating a class with a > __buffer__ method and calling numpy.frombuffer on it. > > > I submitted a pull request to remove the code. Since it is undocumented > and (as far as I can tell) broken, I do not think we need a deprecation > cycle. > > > More details, including links to the original numeric code from 2005, in > the PR https://github.com/numpy/numpy/pull/13049 > > > Any thoughts or objections? > > > Matti > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Feb 27 16:37:43 2019 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 27 Feb 2019 16:37:43 -0500 Subject: [Numpy-discussion] Outreachy internship In-Reply-To: <20190227061915.yodq75qoqrfox2ts@carbo> References: <20190227061915.yodq75qoqrfox2ts@carbo> Message-ID: Fantastic! -- Marten On Wed, Feb 27, 2019 at 1:19 AM Stefan van der Walt wrote: > Hi everyone, > > The team at BIDS would like to take on an intern from Outreachy > (https://www.outreachy.org), as part of our effort to grow the NumPy > developer community. > > The internship is similar to a GSoC, except that we will pay their > salary, as well as do the mentoring. We wanted to check in beforehand > to ensure that everyone is comfortable with us doing this on behalf of > the NumPy community. > > Your thoughts and recommendations are welcome! > > Best regards, > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dillon.niederhut at gmail.com Thu Feb 28 08:00:56 2019 From: dillon.niederhut at gmail.com (Dillon Niederhut) Date: Thu, 28 Feb 2019 07:00:56 -0600 Subject: [Numpy-discussion] Outreachy internship In-Reply-To: References: <20190227061915.yodq75qoqrfox2ts@carbo> Message-ID: Super cool! Thanks, Stefan! Yours, Dillon On Wed, Feb 27, 2019, 15:38 Marten van Kerkwijk wrote: > Fantastic! > > -- Marten > > On Wed, Feb 27, 2019 at 1:19 AM Stefan van der Walt > wrote: > >> Hi everyone, >> >> The team at BIDS would like to take on an intern from Outreachy >> (https://www.outreachy.org), as part of our effort to grow the NumPy >> developer community. >> >> The internship is similar to a GSoC, except that we will pay their >> salary, as well as do the mentoring. We wanted to check in beforehand >> to ensure that everyone is comfortable with us doing this on behalf of >> the NumPy community. >> >> Your thoughts and recommendations are welcome! >> >> Best regards, >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From opossumnano at gmail.com Thu Feb 28 10:35:49 2019 From: opossumnano at gmail.com (Tiziano Zito) Date: Thu, 28 Feb 2019 07:35:49 -0800 (PST) Subject: [Numpy-discussion] =?utf-8?b?W0FOTl0gMTLhtZfKsCBBZHZhbmNlZCBT?= =?utf-8?q?cientific_Programming_in_Python_in_Camerino=2C_Italy=2C_2?= =?utf-8?q?=E2=80=947_September=2C_2019?= Message-ID: <5c77ffd5.1c69fb81.7cfce.8e9c@mx.google.com> 12?? Advanced Scientific Programming in Python ============================================== a Summer School by the G-Node and the University of Camerino https://python.g-node.org Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project ? an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 2?7 September, 2019. Camerino, Italy. Application =========== You can apply online: https://python.g-node.org/wiki/applications Application deadline: 23:59 UTC, 26 May, 2019. There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://python.g-node.org/wiki/faq Participation is for free, i.e. no fee is charged! Participants however should take care of travel, living, and accommodation expenses by themselves. Program ======= ? Version control with git and how to contribute to open source projects with GitHub ? Tidy data analysis and visualization ? Testing and debugging scientific code ? Advanced NumPy ? Organizing, documenting, and distributing scientific code ? Advanced scientific Python: context managers and generators ? Writing parallel applications in Python ? Profiling and speeding up scientific code with Cython and numba ? Programming in teams Faculty ======= ? Caterina Buizza, Personal Robotics Lab, Imperial College London, UK ? Jenni Rinker, Department of Wind Energy, Technical University of Denmark, Roskilde, Denmark ? Juan Nunez-Iglesias, Bioimage Analysis Research Fellow, Monash University, Australia ? Nelle Varoquaux, Department of Statistics, UC Berkeley, CA, USA ? Pamela Hathway, Neural Reckoning, Imperial College London, UK ? Pietro Berkes, NAGRA Kudelski, Lausanne, Switzerland ? Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany ? St?fan van der Walt, Berkeley Institute for Data Science, UC Berkeley, CA, USA ? Tiziano Zito, Department of Psychology, Humboldt-Universit?t zu Berlin, Germany Organizers ========== For the German Neuroinformatics Node of the INCF (G-Node), Germany: ? Tiziano Zito, Department of Psychology, Humboldt-Universit?t zu Berlin, Germany ? Caterina Buizza, Personal Robotics Lab, Imperial College London, UK ? Zbigniew J?drzejewski-Szmek, Red Hat Inc., Warsaw, Poland ? Jakob Jordan, Department of Physiology, University of Bern, Switzerland For the University of Camerino, Italy: ? Barbara Re, Computer Science Division, School of Science and Technology, University of Camerino Italy Website: https://python.g-node.org Contact: python-info at g-node.org From charlesr.harris at gmail.com Thu Feb 28 13:43:41 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 28 Feb 2019 11:43:41 -0700 Subject: [Numpy-discussion] Removing undocumented __buffer__ attribute lookup In-Reply-To: <97dd993a-a4d3-498d-b6df-1471fd1b4d98@Canary> References: <210da5f6-cf94-79cc-4b74-d798f96196c0@gmail.com> <97dd993a-a4d3-498d-b6df-1471fd1b4d98@Canary> Message-ID: On Wed, Feb 27, 2019 at 1:37 AM Hameer Abbasi wrote: > Cc-ing in Travis, because he was the original author of the buffer > protocol, and this is most definitely related. > > Best Regards, > Hameer Abbasi > > On Wednesday, Feb 27, 2019 at 9:20 AM, Matti Picus > wrote: > In digging around the code, I found a gem in PyArray_FromBuffer (exposed > to python as numpy.frombuffer). If a PyObject* does not have a > tp_as_buffer->bf_getbuffer function, we check if the python object has a > __buffer__ attribute. If so we use that as buf in > PyObject_GetBuffer(buf, ...). > > > This seems to stem back to the original numerics code, where getBuffer > would look up the attribute and call it as a method. PyArray_FromBuffer > does not call the attribute as a method, it simply passes it on to > PyObject_GetBuffer, which will then raise an error saying it cannot > convert a method. You can try this out by creating a class with a > __buffer__ method and calling numpy.frombuffer on it. > > > I submitted a pull request to remove the code. Since it is undocumented > and (as far as I can tell) broken, I do not think we need a deprecation > cycle. > > I vaguely recall using that many, many years ago for some C code, but don't see anything using it these days. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: