From matthias.geier at gmail.com Wed Jan 2 05:27:48 2019 From: matthias.geier at gmail.com (Matthias Geier) Date: Wed, 2 Jan 2019 11:27:48 +0100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> Message-ID: Hi Sebastian. Thanks for the clarification. On Sun, Dec 30, 2018 at 5:25 PM Sebastian Berg wrote: > On Sun, 2018-12-30 at 16:03 +0100, Matthias Geier wrote: > > On Sat, Dec 29, 2018 at 6:00 PM Sebastian Berg wrote: > > > On Sat, 2018-12-29 at 17:16 +0100, Matthias Geier wrote: > > > > Hi Sebastian. > > > > > > > > I don't have an opinion (yet) about this matter, but I have a > > > > question: > > > > > > > > On Thu, Dec 27, 2018 at 12:30 AM Sebastian Berg wrote: > > > > > > > > [...] > > > > > > > > > new_arr = arr.reshape(new_shape) > > > > > assert np.may_share_memory(arr, new_arr) > > > > > > > > > > # Which is sometimes -- but should not be -- written as: > > > > > arr.shape = new_shape # unnecessary container modification > > > > > > > > [...] > > > > > > > > Why is this discouraged? > > > > > > > > Why do you call this "unnecessary container modification"? > > > > > > > > I've used this idiom in the past for exactly those cases where I > > > > wanted to make sure no copy is made. > > > > > > > > And if we are not supposed to assign to arr.shape, why is it > > > > allowed > > > > in the first place? > > > > > > Well, this may be a matter of taste, but say you have an object > > > that > > > stores an array: > > > > > > class MyObject: > > > def __init__(self): > > > self.myarr = some_array > > > > > > > > > Now, lets say I do: > > > > > > def some_func(arr): > > > # Do something with the array: > > > arr.shape = -1 > > > > > > myobject = MyObject() > > > some_func(myobject) > > > > > > then myobject will suddenly have the wrong shape stored. In most > > > cases > > > this is harmless, but I truly believe this is exactly why we have > > > views > > > and why they are so awesome. > > > The content of arrays is mutable, but the array object itself > > > should > > > not be muted normally. > > > > Thanks for the example! I don't understand its point, though. > > Also, it's not working since MyObject doesn't have a .shape > > attribute. > > > > The example should have called `some_func(myobject.arr)`. The thing is > that if you have more references to the same array around, you change > all their shapes. And if those other references are there for a reason, > that is not what you want. > > That does not matter much in most cases, but it could change the shape > of an array in a completely different place then intended. Creating a > new view is cheap, so I think such things should be avoided. > > I admit, most code will effectively do: > arr = input_arr[...] # create a new view > arr.shape = ... > > so that there is no danger. But conceptually, I do not think there > should be a danger of magically changing the shape of a stored array in > a different part of the code. > > Does that make some sense? Maybe shorter example: > > arr = np.arange(10) > arr2 = arr > arr2.shape = (5, 2) > > print(arr.shape) # also (5, 2) > > so the arr container (shape, dtype) is changed/muted. I think we expect > that for content here, but not for the shape. Thanks for the clarification, I think I now understand your example. However, the behavior you are describing is just like the normal reference semantics of Python itself. If you have multiple identifiers bound to the same (mutable) object, you'll always have this "problem". I think every Python user should be aware of this behavior, but I don't think it is reason to discourage assigning to arr.shape. Coming back to the original suggestion of this thread: Since assigning to arr.shape makes sure no copy of the array data is made, I don't think it's necessary to add a new no-copy argument to reshape(). But the bug you mentioned ("on error the `arr.shape = ...` code currently creates the copy temporarily") should probably be fixed at some point ... cheers, Matthias > > - Sebastian > > > > > There may be some corner cases, but a lot of the > > > "than why is it allowed" questions are answered with: for history > > > reasons. > > > > OK, that's a good point. > > > > > By the way, on error the `arr.shape = ...` code currently creates > > > the > > > copy temporarily. > > > > That's interesting and it should probably be fixed. > > > > But it is not reason enough for me not to use it. > > I find it important that is doesn't make a copy in the success case, > > I > > don't care very much for the error case. > > > > Would you mind elaborating on the real reasons why I shouldn't use > > it? > > > > cheers, > > Matthias > > > > > - Sebastian > > > > > > > > > > cheers, > > > > Matthias > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From sebastian at sipsolutions.net Wed Jan 2 08:22:15 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 02 Jan 2019 14:22:15 +0100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> Message-ID: <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> On Wed, 2019-01-02 at 11:27 +0100, Matthias Geier wrote: > Hi Sebastian. > > Thanks for the clarification. > > > print(arr.shape) # also (5, 2) > > > > so the arr container (shape, dtype) is changed/muted. I think we > > expect > > that for content here, but not for the shape. > > Thanks for the clarification, I think I now understand your example. > > However, the behavior you are describing is just like the normal > reference semantics of Python itself. > > If you have multiple identifiers bound to the same (mutable) object, > you'll always have this "problem". > > I think every Python user should be aware of this behavior, but I > don't think it is reason to discourage assigning to arr.shape. Well, I doubt I will convince you. But want to point out that a numpy array is: * underlying data * shape/strides (pointing to the exact data) * data type (interpret the data) Arrays are mutable, but this is only half true from my perspective. Everyone using numpy should be aware of "views", i.e. that the content of the underlying data can change. However, if I have a read-only array, and pass it around, I would not expect it to change. That is because while the underlying data is muted, how this data is accessed and interpreted is not. In other words, I see array objects as having two sides to them [0]: * Underlying data -> normally mutable and often muted * container: -> not muted by almost all code * shape/strides * data type I realize that in some cases muting the container metadata happens. But I do believe it should be as minimal as possible. And frankly, probably one could do away with it completely. Another example for where it is bad would be a threaded environment. If a python function temporarily changes the shape of an array to read from it without creating a view first, this will break multi-threaded access to that array. - Sebastian [0] I tried to find other examples for such a split. Maybe a categorical/state object which is allowed change value/state. But the list of possible states cannot change. > Coming back to the original suggestion of this thread: > Since assigning to arr.shape makes sure no copy of the array data is > made, I don't think it's necessary to add a new no-copy argument to > reshape(). > > But the bug you mentioned ("on error the `arr.shape = ...` code > currently creates the copy temporarily") should probably be fixed at > some point ... > > cheers, > Matthias > > > - Sebastian > > > > > > > > There may be some corner cases, but a lot of the > > > > "than why is it allowed" questions are answered with: for > > > > history > > > > reasons. > > > > > > OK, that's a good point. > > > > > > > By the way, on error the `arr.shape = ...` code currently > > > > creates > > > > the > > > > copy temporarily. > > > > > > That's interesting and it should probably be fixed. > > > > > > But it is not reason enough for me not to use it. > > > I find it important that is doesn't make a copy in the success > > > case, > > > I > > > don't care very much for the error case. > > > > > > Would you mind elaborating on the real reasons why I shouldn't > > > use > > > it? > > > > > > cheers, > > > Matthias > > > > > > > - Sebastian > > > > > > > > > > > > > cheers, > > > > > Matthias > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at python.org > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From einstein.edison at gmail.com Thu Jan 3 07:42:21 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Thu, 3 Jan 2019 13:42:21 +0100 Subject: [Numpy-discussion] Testing radix sort on big-endian architectures Message-ID: <7f7ebf97-b47e-4f25-a318-34dbf3ee7956@Canary> Hello. I?m writing a PR (#12586 (https://github.com/numpy/numpy/pull/12586)) for radix sort, and since some operations involve bit-shifts, the concern came up that big-endian architectures might not support it. If someone has access to a machine with a big-endian architecture, it would be nice if they could run the test suite on that PR. Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagru at mailbox.org Fri Jan 4 08:34:39 2019 From: lagru at mailbox.org (Lars Grueter) Date: Fri, 4 Jan 2019 14:34:39 +0100 Subject: [Numpy-discussion] Debugging NumPy development build in conda environment Message-ID: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> Dear devs, If this is the wrong place for this kind of question I apologize and feel free to ignore this or point me to the right place. I'm using the conda package manager for my development environment which used to work just fine with these steps: > conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage > source activate dev-numpy > conda uninstall --force numpy numpy-base > pip install -e . This would yield me a nice isolated playground and I could use > python setup.py build_ext --incplace -j 4 at anytime to rebuild the binaries. Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with > RuntimeError: Broken toolchain: cannot link a simple C program (full output in [1]) on a clean repository. No recent changes to my tool chain come to mind which could help me pinpoint the problem. Because it used to work just fine I have the annoying feeling that the fix is something small and trivial and have the hope that it's immediately obvious to someone here or that you can suggest a way to debug this. In any case your help would be appreciated. I'm aware that this may be an exotic dev setup. Using a virtual environment instead of the conda one works just fine. The build script seems to use a different linear algebra library in that case: blas from my OS as opposed to mkl in the conda environment. This is not my preferred solution but it would do if nothing comes of this. Some additional facts and research I've done: - Using conda environments with asv has stopped working as well due to not finding any LA library at all. - If I'm understanding [1] correctly the build script finds the mkl library provided by the conda environment. - Trying earlier versions of NumPy shows the same behavior. I tested the tags v1.14.6, v1.15.4. - However creating a development environment for SciPy and scikit-image using this approach still works fine. - Searching the internet or looking at NumPy's build guide didn't yield me a solution. - I'm on linux (Manjaro) if that is of any relevance. - Miniconda was reinstalled in my home directory with the installation script provided through Anaconda's website. - Haven't yet tried this approach in a fresh virtual machine. I hope that I haven't overlooked something obvious. In any case thank you for your time and attention. Cheers, Lars [1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5 From einstein.edison at gmail.com Fri Jan 4 08:53:51 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 4 Jan 2019 14:53:51 +0100 Subject: [Numpy-discussion] Debugging NumPy development build in conda environment In-Reply-To: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> References: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> Message-ID: <1dfdf9da-0917-46e5-ad52-0d85a81f761c@Canary> > On Friday, Jan 04, 2019 at 2:44 PM, Lars Grueter wrote: > Dear devs, > > If this is the wrong place for this kind of question I apologize and > feel free to ignore this or point me to the right place. > > I'm using the conda package manager for my development environment which > used to work just fine with these steps: > > > conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage > > source activate dev-numpy > > conda uninstall --force numpy numpy-base > > pip install -e . > > This would yield me a nice isolated playground and I could use > > > python setup.py build_ext --incplace -j 4 > > at anytime to rebuild the binaries. Unfortunately the last two commands > stopped working out of the blue with my existing environment and fail with > > > RuntimeError: Broken toolchain: cannot link a simple C program > > (full output in [1]) on a clean repository. No recent changes to my tool > chain come to mind which could help me pinpoint the problem. Because it > used to work just fine I have the annoying feeling that the fix is > something small and trivial and have the hope that it's immediately > obvious to someone here or that you can suggest a way to debug this. In > any case your help would be appreciated. > > I'm aware that this may be an exotic dev setup. Using a virtual > environment instead of the conda one works just fine. The build script > seems to use a different linear algebra library in that case: blas from > my OS as opposed to mkl in the conda environment. This is not my > preferred solution but it would do if nothing comes of this. > > Some additional facts and research I've done: > > - Using conda environments with asv has stopped working as well due to > not finding any LA library at all. > - If I'm understanding [1] correctly the build script finds the mkl > library provided by the conda environment. > - Trying earlier versions of NumPy shows the same behavior. I tested the > tags v1.14.6, v1.15.4. > - However creating a development environment for SciPy and scikit-image > using this approach still works fine. > - Searching the internet or looking at NumPy's build guide didn't yield > me a solution. > - I'm on linux (Manjaro) if that is of any relevance. > - Miniconda was reinstalled in my home directory with the installation > script provided through Anaconda's website. > - Haven't yet tried this approach in a fresh virtual machine. > > I hope that I haven't overlooked something obvious. In any case thank > you for your time and attention. > > Cheers, Lars > > [1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Hi Lars, What I?m used to doing is simply the following (inside the NumPy root): > conda create -n numpy-dev python=3[.x] pytest > conda activate numpy-dev > > > python setup.py build_ext --inplace -j 4 > > > pip install -e . > > Often, when I run into issues with the build, I simply do this: > [backup site.cfg] > git clean -xfd > > > [restore site.cfg] > > python setup.py build_ext --inplace -j 4 > > pip install -e . > > In your case, it might be the second that helps. If it?s not finding any LA library, you can try compiling with the site.cfg that points to MKL. Cheers and Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Fri Jan 4 08:55:15 2019 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 4 Jan 2019 15:55:15 +0200 Subject: [Numpy-discussion] Debugging NumPy development build in conda environment In-Reply-To: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> References: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> Message-ID: On 4/1/19 3:34 pm, Lars Grueter wrote: > Unfortunately the last two commands > stopped working out of the blue with my existing environment and fail with > >> RuntimeError: Broken toolchain: cannot link a simple C program The compiler error is a few lines above this (copied below). When I have gotten that kind of output, it is because I am using a 32 bit compiler on 64 bit or the opposite. C compiler: gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c' gcc: _configtest.c gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ _configtest.o -o _configtest /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info /home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: unable to initialize decompress status for section .debug_info _configtest.o: file not recognized: file format not recognized Can you compile anything with that gcc? Matti From wieser.eric+numpy at gmail.com Fri Jan 4 11:16:43 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Fri, 4 Jan 2019 08:16:43 -0800 Subject: [Numpy-discussion] Adding more detailed exception types to numpy Message-ID: PR #12593 adds a handful of new exception types for . Some consequences of this change are that: 1. The string formatting is moved to python, allowing us to give better error messages without a lot of work 2. The formatting is dispatched lazily, meaning that users trying ufuncs, but catching unsupported loop types don?t have to pay the cost of string formatting 3. Users can catch a specific exception type. This might expose more of our internals than we?re willing to though. 4. We need to expose these new exception names in the public API. 3 & 4 raise some questions, which I?d like some feedback on: - Should we actually expose the detailed exception types to the user, or should they be kept an implementation detail? One way to hide the implementation details would be class TypeError(builtins.TypeError): """actually UFuncCastingError""" _UFuncCastingError = TypeError # for internal use when raising _UFuncCastingError.__module__ = None class TypeError(builtins.TypeError): """actually UFuncLoopError""" _UFuncLoopError= TypeError # for internal use when raising _UFuncLoopError .__module__ = None del TypeError This gives us all the advantages of 1 & 2 without the user noticing that they?re receiving anything other than TypeError, which their tracebacks will continue to show. - If we do expose them, where should these exceptions go? In the past, we also added AxisError and TooHardError - it would be nice to start being consistent about where to expose these things 1. np.UFuncCastingError (expands the global namespace even further) 2. np.core._exceptions.UFuncCastingError (shouldn?t really be private, since users won?t know how to catch it) 3. np.core.UFuncCastingError 4. np.core.umath.CastingError 5. A dedicated namespace for exceptions: - np.errors.UFuncCastingError (matches pandas) - np.exceptions.UFuncCastingError - np.exc.UFuncCastingError (matches sqlalchemy) Eric ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Jan 4 11:53:08 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 4 Jan 2019 08:53:08 -0800 Subject: [Numpy-discussion] Adding more detailed exception types to numpy In-Reply-To: References: Message-ID: On Fri, Jan 4, 2019 at 8:18 AM Eric Wieser wrote: > PR #12593 adds a handful of > new exception types for . Some consequences of this change are that: > > 1. The string formatting is moved to python, allowing us to give > better error messages without a lot of work > 2. The formatting is dispatched lazily, meaning that users trying > ufuncs, but catching unsupported loop types don?t have to pay the cost of > string formatting > 3. Users can catch a specific exception type. This might expose more > of our internals than we?re willing to though. > 4. We need to expose these new exception names in the public API. > > Wonderful! This sounds great. > > > 3 & 4 raise some questions, which I?d like some feedback on: > > - > > Should we actually expose the detailed exception types to the user, or > should they be kept an implementation detail? One way to hide the > implementation details would be > > class TypeError(builtins.TypeError): > """actually UFuncCastingError""" > _UFuncCastingError = TypeError # for internal use when raising > _UFuncCastingError.__module__ = None > class TypeError(builtins.TypeError): > """actually UFuncLoopError""" > _UFuncLoopError= TypeError # for internal use when raising > _UFuncLoopError .__module__ = None > del TypeError > > This gives us all the advantages of 1 & 2 without the user noticing > that they?re receiving anything other than TypeError, which their > tracebacks will continue to show. > > I don't think this is a good idea -- think about how much confusion has been caused by how NumPy gives scalars the same repr as Python builtins. It's great to subclass TypeError for our needs, but we shouldn't add more disguised subclasses of builtin Python types. If we're really concerned about exposing too much API surface, we should group these errors into logical groups inheriting from NumPy-specific baseclasses, e.g., _UFuncCastingError and _UFuncLoopError could inherit from UFuncError. Then at least users would at least know where to look (at numpy.UFuncError) when their error doesn't serialize properly. > > - If we do expose them, where should these exceptions go? In the past, > we also added AxisError and TooHardError - it would be nice to start > being consistent about where to expose these things > - > 1. np.UFuncCastingError (expands the global namespace even further) > 2. np.core._exceptions.UFuncCastingError (shouldn?t really be > private, since users won?t know how to catch it) > 3. np.core.UFuncCastingError > 4. np.core.umath.CastingError > 5. A dedicated namespace for exceptions: > - np.errors.UFuncCastingError (matches pandas) > - np.exceptions.UFuncCastingError > - np.exc.UFuncCastingError (matches sqlalchemy) > > numpy.errors sounds like a good namespace to me. It's short and descriptive. We can put aliases to AxisError and TooHardError in this namespace, too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 4 12:36:18 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2019 10:36:18 -0700 Subject: [Numpy-discussion] Ia `-fno-strict-aliasing` still required? Message-ID: Hi All, Just asking if the `-fno-strict-aliasing` flag is still required for gcc. Supposedly `-fstrict-aliasing` is enabled by default with optimization levels >= `-O2` and that used to result in errors. However, I have noticed that numpy wheels are being built without the `-fno-strict-aliasing` flag and doing fine. Same on my local machine. So maybe the compiler has gotten better at knowing when pointers may be aliased? I cannot find informative documentation on how `-fstrict-aliasing` is actually implemented, so thought I'd ask here if someone knows more about it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Jan 4 13:24:47 2019 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 4 Jan 2019 13:24:47 -0500 Subject: [Numpy-discussion] Ia `-fno-strict-aliasing` still required? In-Reply-To: References: Message-ID: Define "doing fine", does it pass all tests? On Fri, Jan 4, 2019 at 12:36 PM Charles R Harris wrote: > Hi All, > > Just asking if the `-fno-strict-aliasing` flag is still required for gcc. > Supposedly `-fstrict-aliasing` is enabled by default with optimization > levels >= `-O2` and that used to result in errors. However, I have noticed > that numpy wheels are being built without the `-fno-strict-aliasing` flag > and doing fine. Same on my local machine. So maybe the compiler has gotten > better at knowing when pointers may be aliased? I cannot find informative > documentation on how `-fstrict-aliasing` is actually implemented, so > thought I'd ask here if someone knows more about it. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 4 13:32:23 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2019 11:32:23 -0700 Subject: [Numpy-discussion] Ia `-fno-strict-aliasing` still required? In-Reply-To: References: Message-ID: On Fri, Jan 4, 2019 at 11:26 AM Neal Becker wrote: > Define "doing fine", does it pass all tests? > Yes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Fri Jan 4 14:28:42 2019 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Fri, 4 Jan 2019 14:28:42 -0500 Subject: [Numpy-discussion] Adding more detailed exception types to numpy In-Reply-To: References: Message-ID: Since numpy generally does not expose parts as modules, I think a separate namespace for the exceptions makes sense. I prefer `np.exceptions` over `np.errors`. It might still make sense for that namespace to import from the different parts, i.e., also have `np.core.exceptions`, np.polynomial.exceptions`, etc., in part because the ones in `core` will have C helper functions. But that is a bit of an implementation detail. -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 4 15:14:11 2019 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Jan 2019 12:14:11 -0800 Subject: [Numpy-discussion] Ia `-fno-strict-aliasing` still required? In-Reply-To: References: Message-ID: This is a pretty good article on what -fno-strict-aliasing actually does: https://blog.regehr.org/archives/1307 -n On Fri, Jan 4, 2019, 09:36 Charles R Harris Hi All, > > Just asking if the `-fno-strict-aliasing` flag is still required for gcc. > Supposedly `-fstrict-aliasing` is enabled by default with optimization > levels >= `-O2` and that used to result in errors. However, I have noticed > that numpy wheels are being built without the `-fno-strict-aliasing` flag > and doing fine. Same on my local machine. So maybe the compiler has gotten > better at knowing when pointers may be aliased? I cannot find informative > documentation on how `-fstrict-aliasing` is actually implemented, so > thought I'd ask here if someone knows more about it. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 4 16:25:12 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2019 14:25:12 -0700 Subject: [Numpy-discussion] Ia `-fno-strict-aliasing` still required? In-Reply-To: References: Message-ID: On Fri, Jan 4, 2019 at 1:14 PM Nathaniel Smith wrote: > This is a pretty good article on what -fno-strict-aliasing actually does: > https://blog.regehr.org/archives/1307 > > That was pretty much as I recalled. I'm guessing that what changed for NumPy was going to separate compilation and the wide use of `char *`. All in all, it looks like we have been lucky and should continue compiling with `-fno-strict-aliasing`, it will probably not affect the speed much. Perhaps what I am missing is looking for compiler warnings rather than errors... AFAICT, MSVC doesn't implement strict aliasing, so no worry there (yet). The gcc manual doesn't say much about `-fno-strict-aliasing`, just ... -fno-strict-aliasing are passed through to the link stage and merged conservatively for conflicting translation units But I am going to assume that it also turns off `-fstrict-aliasing`. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 4 21:35:55 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Jan 2019 19:35:55 -0700 Subject: [Numpy-discussion] NumPy 1.16.0rc2 released Message-ID: Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.0rc2. This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are - Experimental support for overriding numpy functions in downstream projects. - The matmul function is now a ufunc and can be overridden using __array_ufunc__. - Improved support for the ARM and POWER architectures. - Improved support for AIX and PyPy. - Improved interoperation with ctypes. - Improved support for PEP 3118. The supported Python versions are 2.7 and 3.5-3.7, support for 3.4 has been dropped. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4. Wheels for this release can be downloaded from PyPI , source archives are available from Github . *Contributors* A total of 112 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Alan Fontenot + * Allan Haldane * Alon Hershenhorn + * Alyssa Quek + * Andreas Nussbaumer + * Anner + * Anthony Sottile + * Antony Lee * Ayappan P + * Bas van Schaik + * C.A.M. Gerlach + * Charles Harris * Chris Billington * Christian Clauss * Christoph Gohlke * Christopher Pezley + * Daniel B Allan + * Daniel Smith * Dawid Zych + * Derek Kim + * Dima Pasechnik + * Edgar Giovanni Lepe + * Elena Mokeeva + * Elliott Sales de Andrade + * Emil Hessman + * Eric Schles + * Eric Wieser * Giulio Benetti + * Guillaume Gautier + * Guo Ci * Heath Henley + * Isuru Fernando + * J. Lewis Muir + * Jack Vreeken + * Jaime Fernandez * James Bourbeau * Jeff VanOss * Jeffrey Yancey + * Jeremy Chen + * Jeremy Manning + * Jeroen Demeyer * John Darbyshire + * John Kirkham * John Zwinck * Jonas Jensen + * Joscha Reimer + * Juan Azcarreta + * Julian Taylor * Kevin Sheppard * Krzysztof Chomski + * Kyle Sunden * Lars Gr?ter * Lilian Besson + * MSeifert04 * Mark Harfouche * Marten van Kerkwijk * Martin Thoma * Matt Harrigan + * Matthew Bowden + * Matthew Brett * Matthias Bussonnier * Matti Picus * Max Aifer + * Michael Hirsch, Ph.D + * Michael James Jamie Schnaitter + * MichaelSaah + * Mike Toews * Minkyu Lee + * Mircea Akos Bruma + * Mircea-Akos Brum? + * Moshe Looks + * Muhammad Kasim + * Nathaniel J. Smith * Nikita Titov + * Paul M?ller + * Paul van Mulbregt * Pauli Virtanen * Pierre Glaser + * Pim de Haan * Ralf Gommers * Robert Kern * Robin Aggleton + * Rohit Pandey + * Roman Yurchak + * Ryan Soklaski * Sebastian Berg * Sho Nakamura + * Simon Gibbons * Stan Seibert + * Stefan Otte * Stefan van der Walt * Stephan Hoyer * Stuart Archibald * Taylor Smith + * Tim Felgentreff + * Tim Swast + * Tim Teichmann + * Toshiki Kataoka * Travis Oliphant * Tyler Reddy * Uddeshya Singh + * Warren Weckesser * Weitang Li + * Wenjamin Petrenko + * William D. Irons * Yannick Jadoul + * Yaroslav Halchenko * Yug Khanna + * Yuji Kanagawa + * Yukun Guo + * ankokumoyashi + * lerbuke + Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Sun Jan 6 15:44:58 2019 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sun, 6 Jan 2019 20:44:58 +0000 Subject: [Numpy-discussion] Issue with setup_requires and 1.16 release candidates Message-ID: <05126218-AC50-4647-9702-8F8E407B560B@gmail.com> Hi all, Back in December I started getting failures in continuous integration as well as reports of failures from users of installation issues for a couple of packages. The problem can be easily reproduced in a Docker container with: FROM ubuntu:16.04 RUN apt-get update RUN apt-get install -y python3 python3-dev python3-pip python3-wheel RUN pip3 install fast-histogram RUN python3 -c 'import fast_histogram' Doing this results in the following traceback: ImportError: No module named 'numpy.core._multiarray_umath' Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", line 1, in from .histogram import * File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", line 7, in from ._histogram_core import (_histogram1d, ImportError: numpy.core.multiarray failed to import I've seen similar issues with other packages too. The key is that the fast-histogram package defines: setup_requires=['numpy'] in the setup.py (since the package has a C extension that uses the Numpy C API) and numpy is needed before the install_requires dependencies are installed: https://github.com/astrofrog/fast-histogram/blob/master/setup.py Now this normally works fine, but the issues I saw started when the first 1.16 RC was made available, when installing into an environment in which numpy is not already installed. My understanding is that setup_requires is honored by easy_install (even if installing the main package with pip), which doesn't ignore pre-releases. Thus, the package is built against the 1.16 RC but then 1.15 is installed due to: install_requires=['numpy'] which is honored by pip. I think that the correct solution is to make sure that: [build-system] requires = ["setuptools", "wheel", "numpy"] is added to the pyproject.toml file (as per PEP 518). This then works properly with recent versions of pip (>10 I think). I think removing setup_requires then makes sense because it'd be better to have an error with older versions of pip that numpy is not installed rather than having the package built against the wrong version. I know this is a temporary issue in the sense that it will go away once the final version of 1.16 is out, but I just wanted to share this as a heads-up in case you get reports of issues from other people, and also to check whether there are any other solutions/workarounds to be aware of? (to avoid a similar situation in future). Thanks, Tom From ralf.gommers at gmail.com Sun Jan 6 17:05:32 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 6 Jan 2019 14:05:32 -0800 Subject: [Numpy-discussion] Issue with setup_requires and 1.16 release candidates In-Reply-To: <05126218-AC50-4647-9702-8F8E407B560B@gmail.com> References: <05126218-AC50-4647-9702-8F8E407B560B@gmail.com> Message-ID: On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > Hi all, > > Back in December I started getting failures in continuous integration > as well as reports of failures from users of installation issues for a > couple of packages. The problem can be easily reproduced in a Docker > container with: > > FROM ubuntu:16.04 > RUN apt-get update > RUN apt-get install -y python3 python3-dev python3-pip python3-wheel > RUN pip3 install fast-histogram > RUN python3 -c 'import fast_histogram' > > Doing this results in the following traceback: > > ImportError: No module named 'numpy.core._multiarray_umath' > Traceback (most recent call last): > File "", line 1, in > File > "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", > line 1, in > from .histogram import * > File > "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", > line 7, in > from ._histogram_core import (_histogram1d, > ImportError: numpy.core.multiarray failed to import > > I've seen similar issues with other packages too. The key is that the > fast-histogram package defines: > > setup_requires=['numpy'] > > in the setup.py (since the package has a C extension that uses the > Numpy C API) and numpy is needed before the install_requires > dependencies are installed: > > https://github.com/astrofrog/fast-histogram/blob/master/setup.py > > Now this normally works fine, but the issues I saw started when the > first 1.16 RC was made available, when installing into an environment > in which numpy is not already installed. > > My understanding is that setup_requires is honored by easy_install > (even if installing the main package with pip), which doesn't ignore > pre-releases. Thus, the package is built against the 1.16 RC but then > 1.15 is installed due to: > > install_requires=['numpy'] > > which is honored by pip. > Oh fun. Thanks for explaining this! > I think that the correct solution is to make sure that: > > [build-system] > requires = ["setuptools", "wheel", "numpy"] > > is added to the pyproject.toml file (as per PEP 518). To clarify: you're talking about fast-histogram's pyproject.toml file right? This then works > properly with recent versions of pip (>10 I think). I think removing > setup_requires then makes sense because it'd be better to have an > error with older versions of pip that numpy is not installed rather > than having the package built against the wrong version. > This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way. Cheers, Ralf > I know this is a temporary issue in the sense that it will go away > once the final version of 1.16 is out, but I just wanted to share this > as a heads-up in case you get reports of issues from other people, and > also to check whether there are any other solutions/workarounds to be > aware of? (to avoid a similar situation in future). > > Thanks, > Tom > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Sun Jan 6 19:28:13 2019 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 7 Jan 2019 00:28:13 +0000 Subject: [Numpy-discussion] Issue with setup_requires and 1.16 release candidates In-Reply-To: References: <05126218-AC50-4647-9702-8F8E407B560B@gmail.com> Message-ID: Hi Ralf, On Sun, 6 Jan 2019 at 22:06, Ralf Gommers wrote: > > > > On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille wrote: >> >> Hi all, >> >> Back in December I started getting failures in continuous integration >> as well as reports of failures from users of installation issues for a >> couple of packages. The problem can be easily reproduced in a Docker >> container with: >> >> FROM ubuntu:16.04 >> RUN apt-get update >> RUN apt-get install -y python3 python3-dev python3-pip python3-wheel >> RUN pip3 install fast-histogram >> RUN python3 -c 'import fast_histogram' >> >> Doing this results in the following traceback: >> >> ImportError: No module named 'numpy.core._multiarray_umath' >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", >> line 1, in >> from .histogram import * >> File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", >> line 7, in >> from ._histogram_core import (_histogram1d, >> ImportError: numpy.core.multiarray failed to import >> >> I've seen similar issues with other packages too. The key is that the >> fast-histogram package defines: >> >> setup_requires=['numpy'] >> >> in the setup.py (since the package has a C extension that uses the >> Numpy C API) and numpy is needed before the install_requires >> dependencies are installed: >> >> https://github.com/astrofrog/fast-histogram/blob/master/setup.py >> >> Now this normally works fine, but the issues I saw started when the >> first 1.16 RC was made available, when installing into an environment >> in which numpy is not already installed. >> >> My understanding is that setup_requires is honored by easy_install >> (even if installing the main package with pip), which doesn't ignore >> pre-releases. Thus, the package is built against the 1.16 RC but then >> 1.15 is installed due to: >> >> install_requires=['numpy'] >> >> which is honored by pip. > > > Oh fun. Thanks for explaining this! > >> >> I think that the correct solution is to make sure that: >> >> [build-system] >> requires = ["setuptools", "wheel", "numpy"] >> >> is added to the pyproject.toml file (as per PEP 518). > > > To clarify: you're talking about fast-histogram's pyproject.toml file right? Yes that's right. Just for the record, here's the pull request to fast-histogram to implement the fix: https://github.com/astrofrog/fast-histogram/pull/33 Cheers, Tom > >> This then works >> properly with recent versions of pip (>10 I think). I think removing >> setup_requires then makes sense because it'd be better to have an >> error with older versions of pip that numpy is not installed rather >> than having the package built against the wrong version. > > > This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way. > > Cheers, > Ralf > > >> >> I know this is a temporary issue in the sense that it will go away >> once the final version of 1.16 is out, but I just wanted to share this >> as a heads-up in case you get reports of issues from other people, and >> also to check whether there are any other solutions/workarounds to be >> aware of? (to avoid a similar situation in future). >> >> Thanks, >> Tom >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From matthias.geier at gmail.com Mon Jan 7 14:04:58 2019 From: matthias.geier at gmail.com (Matthias Geier) Date: Mon, 7 Jan 2019 20:04:58 +0100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: On Wed, Jan 2, 2019 at 2:24 PM Sebastian Berg wrote: > > On Wed, 2019-01-02 at 11:27 +0100, Matthias Geier wrote: > > Hi Sebastian. > > > > Thanks for the clarification. > > > > > > print(arr.shape) # also (5, 2) > > > > > > so the arr container (shape, dtype) is changed/muted. I think we > > > expect > > > that for content here, but not for the shape. > > > > Thanks for the clarification, I think I now understand your example. > > > > However, the behavior you are describing is just like the normal > > reference semantics of Python itself. > > > > If you have multiple identifiers bound to the same (mutable) object, > > you'll always have this "problem". > > > > I think every Python user should be aware of this behavior, but I > > don't think it is reason to discourage assigning to arr.shape. > > Well, I doubt I will convince you. I think we actually have quite little disagreement. I agree with you on what should be done *most of the time*, but I wouldn't totally discourage mutating NumPy array shapes, because I think in the right circumstances it can be very useful. > But want to point out that a numpy > array is: > > * underlying data > * shape/strides (pointing to the exact data) > * data type (interpret the data) > > Arrays are mutable, but this is only half true from my perspective. > Everyone using numpy should be aware of "views", i.e. that the content > of the underlying data can change. I agree, everyone should be aware of that. > However, if I have a read-only array, and pass it around, I would not > expect it to change. That is because while the underlying data is > muted, how this data is accessed and interpreted is not. > > In other words, I see array objects as having two sides to them [0]: > > * Underlying data -> normally mutable and often muted > * container: -> not muted by almost all code > * shape/strides > * data type Exactly: "almost all code". Most of the time I would not assign to arr.shape, but in some rare occasions I find it very useful. And one of those rare occasions is when you want guaranteed no-copy behavior. There are also some (most likely significantly rarer) cases where I would modify arr.strides. > I realize that in some cases muting the container metadata happens. But > I do believe it should be as minimal as possible. And frankly, probably > one could do away with it completely. I guess that's the only point where we disagree. I wouldn't completely discourage it and I would definitely not remove the functionality. > Another example for where it is bad would be a threaded environment. If > a python function temporarily changes the shape of an array to read > from it without creating a view first, this will break multi-threaded > access to that array. Sure, let's not use it while multi-threading then. I still think that's not at all a reason to remove the feature. There are some things that are problematic when multi-threading, but that's typically not reason enough to completely disallow them. cheers, Matthias > > - Sebastian > > > [0] I tried to find other examples for such a split. Maybe a > categorical/state object which is allowed change value/state. But the > list of possible states cannot change. > > > > Coming back to the original suggestion of this thread: > > Since assigning to arr.shape makes sure no copy of the array data is > > made, I don't think it's necessary to add a new no-copy argument to > > reshape(). > > > > But the bug you mentioned ("on error the `arr.shape = ...` code > > currently creates the copy temporarily") should probably be fixed at > > some point ... > > > > cheers, > > Matthias > > > > > - Sebastian > > > > > > > > > > > There may be some corner cases, but a lot of the > > > > > "than why is it allowed" questions are answered with: for > > > > > history > > > > > reasons. > > > > > > > > OK, that's a good point. > > > > > > > > > By the way, on error the `arr.shape = ...` code currently > > > > > creates > > > > > the > > > > > copy temporarily. > > > > > > > > That's interesting and it should probably be fixed. > > > > > > > > But it is not reason enough for me not to use it. > > > > I find it important that is doesn't make a copy in the success > > > > case, > > > > I > > > > don't care very much for the error case. > > > > > > > > Would you mind elaborating on the real reasons why I shouldn't > > > > use > > > > it? > > > > > > > > cheers, > > > > Matthias > > > > > > > > > - Sebastian > > > > > > > > > > > > > > > > cheers, > > > > > > Matthias From rainwoodman at gmail.com Mon Jan 7 14:21:51 2019 From: rainwoodman at gmail.com (Feng Yu) Date: Mon, 7 Jan 2019 11:21:51 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: Hi, Was it ever brought up the possibility of a new array class (ndrefonly, ndview) that is strictly no copy? All operations on ndrefonly will return ndrefonly and if the operation cannot be completed without making a copy, it shall throw an error. On the implementation there are two choices if we use subclasses: - ndrefonly can be a subclass of ndarray. The pattern would be subclass limiting functionality of super, but ndrefonly is a ndarray. - ndarray as a subclass of ndarray. Subclass supplements functionality of super. : ndarray will not throw an error when a copy is necessary. However ndarray is not a ndarray. If we want to be wild they do not even need to be subclasses of each other, or maybe they shall both be subclasses of something more fundamental. - Yu On Fri, Dec 28, 2018 at 5:45 AM Sebastian Berg wrote: > On Thu, 2018-12-27 at 17:45 -0800, Nathaniel Smith wrote: > > On Thu, Dec 27, 2018 at 3:27 PM Juan Nunez-Iglesias < > > jni.soma at gmail.com> wrote: > > > On Fri, Dec 28, 2018, at 3:40 AM, Sebastian Berg wrote: > > > > > > > It?s consistent with np.newaxis in spelling (modulo the _) > > > > If mistyped, it can be caught easily by IDEs. > > > > It?s typeable with mypy, unlike constant string literals which > > > > currently aren?t > > > > If code written against new numpy is run on old numpy, it will > > > > error > > > > rather than doing the wrong thing > > > > > > I am not sure I think that the above are too strong arguments, > > > since it > > > is not what is typically done for keywords (nobody suggests np.CLIP > > > instead of "clip"). Unless you feel this is different because it is > > > a > > > mix of strings and bools here? > > > > > > > > > :+1: to Eric's list. I don't think it's different because of the > > > mix, I think it's different because deprecating things is painful. > > > But now that we have good typing in Python, I think of string > > > literals as an anti-pattern going forward. > > > > I've certainly seen people argue that it's better to use proper > > enum's > > in this kind of case instead of strings. The 'enum' package is even > > included in the stdlib on all our supported versions now: > > https://docs.python.org/3/library/enum.html > > > > I am sympathetic with that, but it is something we (or scipy, etc.) > currently simply do not use, so I am not sure that I think it has much > validity at this time. That is least unless we agree to aim to use this > more generally in the future. > > > I guess another possibility to throw out there would be a second > > kwarg, require_view=False/True. > > > > My first gut feeling was that it is clumsy at least for `reshape`, but > one will always only use one of the two arguments at a time. > The more I look at it, the better the suggestion seems. Plus it reduces > the possible `copy=False` not meaning "never" confusion. > > I think with a bit more pondering, that will become my favorite > solution. > > - Sebastian > > > > -n > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Mon Jan 7 14:30:19 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Mon, 7 Jan 2019 11:30:19 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: @Matthias: Most of the time I would not assign to arr.shape, but in some rare occasions I find it very useful. And one of those rare occasions is when you want guaranteed no-copy behavior. Can you come up with any other example? The only real argument you seem to have here is ?my code uses arr.shape = ...? and I don?t want it to break. That?s a fair argument, but all it really means is we should start emitting DeprecationWarning("Use arr = arr.reshape(..., copy=np.never_copy) instead of arr.shape = ..."), and consider having a long deprecation. If necessary we could compromise on just putting a warning in the docs, and not notifying the user at all. @Ralf np.newaxis is not relevant here - it?s a simple alias for None, is just there for code readability, and is much more widely applicable than np.never_copy would be. Is there any particular reason we chose to use None? If I were designing it again, I?d consider a singleton object with a better __repr__ @Nathaniel I guess another possibility to throw out there would be a second kwarg, require_view=False/True. The downside of this approach is that array-likes will definitely need updating to support this new behavior, whereas many may work out of the box if we extend the copy argument (like, say, maskedarray). This also ties into the __bool__ override - that will ensure that subclasses which don?t have a trivial reshape crash. @Sebastian: Unless we replace the string when dispatching, which seems strange on first sight. I?m envisaging cases where we don?t have a dispatcher at all: - Duck arrays implementing methods matching ndarray - Something like my_custom_function(arr, copy=...) that forwards its copy argument to reshape Eric ? On Mon, 7 Jan 2019 at 11:05 Matthias Geier wrote: > On Wed, Jan 2, 2019 at 2:24 PM Sebastian Berg wrote: > > > > On Wed, 2019-01-02 at 11:27 +0100, Matthias Geier wrote: > > > Hi Sebastian. > > > > > > Thanks for the clarification. > > > > > > > > > print(arr.shape) # also (5, 2) > > > > > > > > so the arr container (shape, dtype) is changed/muted. I think we > > > > expect > > > > that for content here, but not for the shape. > > > > > > Thanks for the clarification, I think I now understand your example. > > > > > > However, the behavior you are describing is just like the normal > > > reference semantics of Python itself. > > > > > > If you have multiple identifiers bound to the same (mutable) object, > > > you'll always have this "problem". > > > > > > I think every Python user should be aware of this behavior, but I > > > don't think it is reason to discourage assigning to arr.shape. > > > > Well, I doubt I will convince you. > > I think we actually have quite little disagreement. > > I agree with you on what should be done *most of the time*, but I > wouldn't totally discourage mutating NumPy array shapes, because I > think in the right circumstances it can be very useful. > > > But want to point out that a numpy > > array is: > > > > * underlying data > > * shape/strides (pointing to the exact data) > > * data type (interpret the data) > > > > Arrays are mutable, but this is only half true from my perspective. > > Everyone using numpy should be aware of "views", i.e. that the content > > of the underlying data can change. > > I agree, everyone should be aware of that. > > > However, if I have a read-only array, and pass it around, I would not > > expect it to change. That is because while the underlying data is > > muted, how this data is accessed and interpreted is not. > > > > In other words, I see array objects as having two sides to them [0]: > > > > * Underlying data -> normally mutable and often muted > > * container: -> not muted by almost all code > > * shape/strides > > * data type > > Exactly: "almost all code". > > Most of the time I would not assign to arr.shape, but in some rare > occasions I find it very useful. > > And one of those rare occasions is when you want guaranteed no-copy > behavior. > > There are also some (most likely significantly rarer) cases where I > would modify arr.strides. > > > I realize that in some cases muting the container metadata happens. But > > I do believe it should be as minimal as possible. And frankly, probably > > one could do away with it completely. > > I guess that's the only point where we disagree. > > I wouldn't completely discourage it and I would definitely not remove > the functionality. > > > Another example for where it is bad would be a threaded environment. If > > a python function temporarily changes the shape of an array to read > > from it without creating a view first, this will break multi-threaded > > access to that array. > > Sure, let's not use it while multi-threading then. > > I still think that's not at all a reason to remove the feature. > > There are some things that are problematic when multi-threading, but > that's typically not reason enough to completely disallow them. > > cheers, > Matthias > > > > > - Sebastian > > > > > > [0] I tried to find other examples for such a split. Maybe a > > categorical/state object which is allowed change value/state. But the > > list of possible states cannot change. > > > > > > > Coming back to the original suggestion of this thread: > > > Since assigning to arr.shape makes sure no copy of the array data is > > > made, I don't think it's necessary to add a new no-copy argument to > > > reshape(). > > > > > > But the bug you mentioned ("on error the `arr.shape = ...` code > > > currently creates the copy temporarily") should probably be fixed at > > > some point ... > > > > > > cheers, > > > Matthias > > > > > > > - Sebastian > > > > > > > > > > > > > > There may be some corner cases, but a lot of the > > > > > > "than why is it allowed" questions are answered with: for > > > > > > history > > > > > > reasons. > > > > > > > > > > OK, that's a good point. > > > > > > > > > > > By the way, on error the `arr.shape = ...` code currently > > > > > > creates > > > > > > the > > > > > > copy temporarily. > > > > > > > > > > That's interesting and it should probably be fixed. > > > > > > > > > > But it is not reason enough for me not to use it. > > > > > I find it important that is doesn't make a copy in the success > > > > > case, > > > > > I > > > > > don't care very much for the error case. > > > > > > > > > > Would you mind elaborating on the real reasons why I shouldn't > > > > > use > > > > > it? > > > > > > > > > > cheers, > > > > > Matthias > > > > > > > > > > > - Sebastian > > > > > > > > > > > > > > > > > > > cheers, > > > > > > > Matthias > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Jan 7 14:52:10 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 07 Jan 2019 20:52:10 +0100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: <31fb599bf09377f1eed37aad0f31901e52facf6d.camel@sipsolutions.net> On Mon, 2019-01-07 at 20:04 +0100, Matthias Geier wrote: > On Wed, Jan 2, 2019 at 2:24 PM Sebastian Berg wrote: > > On Wed, 2019-01-02 at 11:27 +0100, Matthias Geier wrote: > > > Hi Sebastian. > > > > > > Thanks for the clarification. > > > > > > > > > print(arr.shape) # also (5, 2) > > > > > > > > so the arr container (shape, dtype) is changed/muted. I think > > > > we > > > > expect > > > > that for content here, but not for the shape. > > > > > > Thanks for the clarification, I think I now understand your > > > example. > > > > > > However, the behavior you are describing is just like the normal > > > reference semantics of Python itself. > > > > > > If you have multiple identifiers bound to the same (mutable) > > > object, > > > you'll always have this "problem". > > > > > > I think every Python user should be aware of this behavior, but I > > > don't think it is reason to discourage assigning to arr.shape. > > > > Well, I doubt I will convince you. > > I think we actually have quite little disagreement. > I am very sorry, that was very badly phrased. What I meant is just that saying that arrays simply are mutable objects does not seem all that wrong to me. However, I would very much prefer to move towards changing it for the container part. > I agree with you on what should be done *most of the time*, but I > wouldn't totally discourage mutating NumPy array shapes, because I > think in the right circumstances it can be very useful. > > > But want to point out that a numpy > > array is: > > > > * underlying data > > * shape/strides (pointing to the exact data) > > * data type (interpret the data) > > > > Arrays are mutable, but this is only half true from my perspective. > > In other words, I see array objects as having two sides to them > > [0]: > > > > * Underlying data -> normally mutable and often muted > > * container: -> not muted by almost all code > > * shape/strides > > * data type > > Exactly: "almost all code". > > Most of the time I would not assign to arr.shape, but in some rare > occasions I find it very useful. > > And one of those rare occasions is when you want guaranteed no-copy > behavior. `arr.shape = ...` is somewhat easier on the eye when compared to `arr = arr.reshape(..., ensure_view=True)` (or whatever we would end up doing). Other than that, do you have a technical reason for it (aside from ensuring that no copy will occur)? The thing is that I have seen the suggestion to use `arr.shape` as a "best practices". And I really disagree that it is very good practice. > There are also some (most likely significantly rarer) cases where I > would modify arr.strides. > The same again, there should be no reason why you should have to do this. In fact, this will cause hard crashes for object arrays, so even technically this is not a "harmless" convenience, it is a bug to allow for object arrays ? which own their data ? at all: arr = np.arange(100000, dtype=object) arr.strides = (0,) del arr free(): invalid pointer zsh: abort (core dumped) Note that using the alternatives is completely fine here (as long as you take care all point to valid objects). > > I realize that in some cases muting the container metadata happens. > > But > > I do believe it should be as minimal as possible. And frankly, > > probably > > one could do away with it completely. > > I guess that's the only point where we disagree. > > I wouldn't completely discourage it and I would definitely not remove > the functionality. > > > Another example for where it is bad would be a threaded > > environment. If > > a python function temporarily changes the shape of an array to read > > from it without creating a view first, this will break multi- > > threaded > > access to that array. > > Sure, let's not use it while multi-threading then. > > I still think that's not at all a reason to remove the feature. > While I wouldn't mind mind moving to deprecate it fully, that is not my intention right now. I would discourage it in the documentation with a pointer to a safe alternative. Maybe at some point we realize that nobody really has any good reason to keep using it, then we may move ahead slowly. - Sebastian > There are some things that are problematic when multi-threading, but > that's typically not reason enough to completely disallow them. > > cheers, > Matthias > > > - Sebastian > > > > > > [0] I tried to find other examples for such a split. Maybe a > > categorical/state object which is allowed change value/state. But > > the > > list of possible states cannot change. > > > > > > > Coming back to the original suggestion of this thread: > > > Since assigning to arr.shape makes sure no copy of the array data > > > is > > > made, I don't think it's necessary to add a new no-copy argument > > > to > > > reshape(). > > > > > > But the bug you mentioned ("on error the `arr.shape = ...` code > > > currently creates the copy temporarily") should probably be fixed > > > at > > > some point ... > > > > > > cheers, > > > Matthias > > > > > > > - Sebastian > > > > > > > > > > > > > > There may be some corner cases, but a lot of the > > > > > > "than why is it allowed" questions are answered with: for > > > > > > history > > > > > > reasons. > > > > > > > > > > OK, that's a good point. > > > > > > > > > > > By the way, on error the `arr.shape = ...` code currently > > > > > > creates > > > > > > the > > > > > > copy temporarily. > > > > > > > > > > That's interesting and it should probably be fixed. > > > > > > > > > > But it is not reason enough for me not to use it. > > > > > I find it important that is doesn't make a copy in the > > > > > success > > > > > case, > > > > > I > > > > > don't care very much for the error case. > > > > > > > > > > Would you mind elaborating on the real reasons why I > > > > > shouldn't > > > > > use > > > > > it? > > > > > > > > > > cheers, > > > > > Matthias > > > > > > > > > > > - Sebastian > > > > > > > > > > > > > > > > > > > cheers, > > > > > > > Matthias > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From kwgoodman at gmail.com Mon Jan 7 15:15:42 2019 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 7 Jan 2019 12:15:42 -0800 Subject: [Numpy-discussion] Pairwise summation Message-ID: Numpy uses pairwise summation along the fast axis if that axis contains no more than 8192 elements. How was 8192 chosen? Doubling to 16384 would result in a lot more function call overhead due to the recursion. Is it a speed issue? Memory? Or something else entirely? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Jan 7 15:32:28 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 07 Jan 2019 21:32:28 +0100 Subject: [Numpy-discussion] Pairwise summation In-Reply-To: References: Message-ID: On Mon, 2019-01-07 at 12:15 -0800, Keith Goodman wrote: > Numpy uses pairwise summation along the fast axis if that axis > contains no more than 8192 elements. How was 8192 chosen? > It is simply a constant used throughout the ufunc machinery (and iteration) for cache friendliness. However, that iteration should not always chunk to 8192 elements, it should often just the whole array. And I do not think the inner loop has anything chunking itself, so given a contiguous fast axis and no casting, you likely already get a single outer iteration. In any case 8192 chosen to be small enough to be cache friendly and is exposed as `np.BUFSIZE`, you can actually the buffer that is being used with `numpy.setbufsize(size)`, although I can't say I ever tried it. Note that it has to fit also larger datatypes and multiple buffers. - Sebastian > Doubling to 16384 would result in a lot more function call overhead > due to the recursion. Is it a speed issue? Memory? Or something else > entirely? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From tyler.je.reddy at gmail.com Tue Jan 8 12:57:03 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 8 Jan 2019 09:57:03 -0800 Subject: [Numpy-discussion] timedelta64 remainder behavior with div by 0 Message-ID: We are now at the stage of implementing the timedelta64 divmod inner loop given very recent additions of floordiv and remainder inner loops for this data type. However, there is some contention about a previous decision regarding modulus behavior that we'd like to resolve before we bake it in to divmod. Currently, a modulus operation with two timedelta64 operands with a 0 denominator returns 0. For example: np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) In contrast, np.float64(1) % np.float64(0) -> nan There's a suggestion that we should switch to returning NaT for the timedelta64 case for consistency, and that this probably isn't too harmful given how recent these additions are. Do we have consensus on this? Ref: https://github.com/numpy/numpy/pull/12683 Thanks! Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Tue Jan 8 13:02:13 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Tue, 8 Jan 2019 19:02:13 +0100 Subject: [Numpy-discussion] timedelta64 remainder behavior with div by 0 In-Reply-To: References: Message-ID: I would say this is desirable behaviour, but I?m still +0.8 on this for backward compatibility reasons. I doubt anyone would build code that relies on this though? They would almost certainly check for the zero in the denominator rather than the return value. Best Regards, Hameer Abbasi > On Tuesday, Jan 08, 2019 at 6:57 PM, Tyler Reddy wrote: > We are now at the stage of implementing the timedelta64 divmod inner loop given very recent additions of floordiv and remainder inner loops for this data type. However, there is some contention about a previous decision regarding modulus behavior that we'd like to resolve before we bake it in to divmod. > > Currently, a modulus operation with two timedelta64 operands with a 0 denominator returns 0. For example: > > np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) > > In contrast, np.float64(1) % np.float64(0) -> nan > > There's a suggestion that we should switch to returning NaT for the timedelta64 case for consistency, and that this probably isn't too harmful given how recent these additions are. > > Do we have consensus on this? > > Ref: https://github.com/numpy/numpy/pull/12683 > > Thanks! > Tyler _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Jan 8 13:05:22 2019 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 8 Jan 2019 10:05:22 -0800 Subject: [Numpy-discussion] timedelta64 remainder behavior with div by 0 In-Reply-To: References: Message-ID: <20190108180522.65v6ghyrglhtl7lb@carbo> On Tue, 08 Jan 2019 09:57:03 -0800, Tyler Reddy wrote: > np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) > > In contrast, np.float64(1) % np.float64(0) -> nan > > There's a suggestion that we should switch to returning NaT for the > timedelta64 case for consistency, and that this probably isn't too harmful > given how recent these additions are. That seems like a reasonable change to me; one could probably consider the previous behavior a bug? St?fan From wieser.eric+numpy at gmail.com Tue Jan 8 13:27:57 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Tue, 8 Jan 2019 10:27:57 -0800 Subject: [Numpy-discussion] timedelta64 remainder behavior with div by 0 In-Reply-To: <20190108180522.65v6ghyrglhtl7lb@carbo> References: <20190108180522.65v6ghyrglhtl7lb@carbo> Message-ID: If we consider it a bug, we could patch it in 1.16.1 (or are we still waiting on 1.16.0?), which would minimize the backwards compatibility cost. Eric On Tue, 8 Jan 2019 at 10:05 Stefan van der Walt wrote: > On Tue, 08 Jan 2019 09:57:03 -0800, Tyler Reddy wrote: > > np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) > > > > In contrast, np.float64(1) % np.float64(0) -> nan > > > > There's a suggestion that we should switch to returning NaT for the > > timedelta64 case for consistency, and that this probably isn't too > harmful > > given how recent these additions are. > > That seems like a reasonable change to me; one could probably consider the > previous behavior a bug? > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Tue Jan 8 13:35:53 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Tue, 8 Jan 2019 10:35:53 -0800 Subject: [Numpy-discussion] timedelta64 remainder behavior with div by 0 In-Reply-To: References: <20190108180522.65v6ghyrglhtl7lb@carbo> Message-ID: Looks like we're still on 1.16.0rc2 -- released 4 days ago. On Tue, 8 Jan 2019 at 10:28, Eric Wieser wrote: > If we consider it a bug, we could patch it in 1.16.1 (or are we still > waiting on 1.16.0?), which would minimize the backwards compatibility cost. > > Eric > > On Tue, 8 Jan 2019 at 10:05 Stefan van der Walt > wrote: > >> On Tue, 08 Jan 2019 09:57:03 -0800, Tyler Reddy wrote: >> > np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) >> > >> > In contrast, np.float64(1) % np.float64(0) -> nan >> > >> > There's a suggestion that we should switch to returning NaT for the >> > timedelta64 case for consistency, and that this probably isn't too >> harmful >> > given how recent these additions are. >> >> That seems like a reasonable change to me; one could probably consider the >> previous behavior a bug? >> >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Jan 8 17:57:36 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 9 Jan 2019 00:57:36 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Jan 9 at 12:00 pacific time Message-ID: <0194f3fb-b285-8194-2c00-8a7fdc62c971@gmail.com> The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. The BIDS team From nathan12343 at gmail.com Tue Jan 8 18:17:26 2019 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Tue, 8 Jan 2019 17:17:26 -0600 Subject: [Numpy-discussion] [ANN] 2019 Scipy Conference: Call for Proposals Message-ID: SciPy 2019, the 18th annual Scientific Computing with Python conference, will be held July 8-14, 2019 in Austin, Texas. The annual SciPy Conference brings together over 800 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for abstracts for SciPy 2019 for talks, posters and tutorials is now open. The deadline for submissions is February 10, 2019. Conference Website: https://www.scipy2019.scipy.org/ Submission Website: https://easychair.org/conferences/?conf=scipy2019 Talks and Posters (July 10-12, 2019) In addition to the general track, this year will have specialized tracks focused on: - Data Driven Discoveries (including Machine Learning and Data Science) - Open Source Communities (Sustainability) Mini Symposia - Science Communication through Visualization - Neuroscience and Cognitive Science - Image Processing - Earth, Ocean, Geo and Atmospheric Science There will also be a SciPy Tools Plenary Session each day with 2 to 5 minute updates on tools and libraries. Tutorials (July 8-9, 2019) Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for useful techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 10 01:25:05 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 9 Jan 2019 22:25:05 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: On Mon, Jan 7, 2019 at 11:30 AM Eric Wieser wrote: > > @Ralf > > np.newaxis is not relevant here - it?s a simple alias for None, is just > there for code readability, and is much more widely applicable than > np.never_copy would be. > > Is there any particular reason we chose to use None? If I were designing > it again, I?d consider a singleton object with a better __repr__ > It stems from Numeric: https://mail.python.org/pipermail/python-list/2009-September/552203.html. Note that the Python builtin slice also uses None, but that's probably due to Numeric using it first. Agree that a singleton with a nice repr could be a better choice than None. The more important part of my comment was "widely applicable" though. Slicing is a lot more important than some keyword. And design-wise, filling the numpy namespace with singletons for keyword to other things in that same namespace just makes no sense to me. Cheers, Ralf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Thu Jan 10 01:54:29 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Wed, 9 Jan 2019 22:54:29 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: Slicing is a lot more important than some keyword. And design-wise, filling the numpy namespace with singletons for keyword to other things in that same namespace just makes no sense to me. At least from the perspective of discoverability, you could argue that string constants form a namespace of their won, and so growing the ?string? namespace is not inherently better than growing any other. The main flaw in that comparison is that picking np.never_copy to be a singleton forever prevents us reusing that name to become a function. Perhaps the solution is to use np.NEVER_COPY instead - that?s never going to clash with a function name we want to add in future, and using upper attributes as arguments in that way is pretty typical for python ( subprocess.PIPE, socket.SOCK_STREAM, etc?) You could fairly argue that this approach is outdated in the face of enum.Enum - in which case we could go for the more heavy-handed np.CopyMode.NEVER, which still has a unique enough case for name clashes with functions never to be an issue. Eric ? On Wed, 9 Jan 2019 at 22:25 Ralf Gommers wrote: > On Mon, Jan 7, 2019 at 11:30 AM Eric Wieser > wrote: > >> >> @Ralf >> >> np.newaxis is not relevant here - it?s a simple alias for None, is just >> there for code readability, and is much more widely applicable than >> np.never_copy would be. >> >> Is there any particular reason we chose to use None? If I were designing >> it again, I?d consider a singleton object with a better __repr__ >> > It stems from Numeric: > https://mail.python.org/pipermail/python-list/2009-September/552203.html. > Note that the Python builtin slice also uses None, but that's probably due > to Numeric using it first. > > Agree that a singleton with a nice repr could be a better choice than > None. The more important part of my comment was "widely applicable" though. > Slicing is a lot more important than some keyword. And design-wise, filling > the numpy namespace with singletons for keyword to other things in that > same namespace just makes no sense to me. > > Cheers, > Ralf > > > >> _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Jan 10 02:33:05 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Jan 2019 02:33:05 -0500 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: On Mon, Jan 7, 2019, 14:22 Feng Yu Hi, > > Was it ever brought up the possibility of a new array class (ndrefonly, > ndview) that is strictly no copy? > > All operations on ndrefonly will return ndrefonly and if the operation > cannot be completed without making a copy, it shall throw an error. > > On the implementation there are two choices if we use subclasses: > > - ndrefonly can be a subclass of ndarray. The pattern would be subclass > limiting functionality of super, but ndrefonly is a ndarray. > - ndarray as a subclass of ndarray. Subclass supplements functionality of > super. : ndarray will not throw an error when a copy is necessary. However > ndarray is not a ndarray. > > If we want to be wild they do not even need to be subclasses of each > other, or maybe they shall both be subclasses of something more > fundamental. > > - Yu > I would prefer a flag for this. Someone can make an array read-only by setting `arr.flags.writable=False`. So along those lines, we could have a `arr.flags.copyable` flag that if set to `False` would result in an error of any operation tried to copy the data. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Jan 10 02:35:25 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Jan 2019 02:35:25 -0500 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> Message-ID: On Wed, Dec 26, 2018, 18:29 Sebastian Berg Hi all, > > In https://github.com/numpy/numpy/pull/11897 I am looking into the > addition of a `copy=np.never_copy` argument to: > * np.array > * arr.reshape/np.reshape > * arr.astype > > Which would cause an error to be raised when numpy cannot guarantee > that the returned array is a view of the input array. > The motivation is to easier avoid accidental copies of large data, or > ensure that in-place manipulation will be meaningful. > > The copy flag API would be: > * `copy=True` forces a copy > * `copy=False` allows numpy to copy if necessary > * `copy=np.never_copy` will error if a copy would be necessary > * (almost) all other input will be deprecated. > > Unfortunately using `copy="never"` is tricky, because currently > `np.array(..., copy="never")` behaves exactly the same as > `np.array(..., copy=bool("never"))`. So that the wrong result would be > given on old numpy versions and it would be a behaviour change. > > Some things that are a not so nice maybe: > * adding/using `np.never_copy` is not very nice > * Scalars need a copy and so will not be allowed > * For rare array-likes numpy may not be able to guarantee no-copy, > although it could happen (but should not). > > > The history is that a long while ago I considered adding a copy flag to > `reshape` so that it is possible to do `copy=np.never_copy` (or > similar) to ensure that no copy is made. In these, you may want > something like an assertion: > > ``` > new_arr = arr.reshape(new_shape) > assert np.may_share_memory(arr, new_arr) > > # Which is sometimes -- but should not be -- written as: > arr.shape = new_shape # unnecessary container modification > > # Or: > view = np.array(arr, order="F") > assert np.may_share_memory(arr, new_arr) > ``` > > but is more readable and will not cause an intermediate copy on error. > > > So what do you think? Other variants would be to not expose this for > `np.array` and probably limit `copy="never"` to the reshape method. Or > just to not do it at all. Or to also accept "never" for `reshape`, > although I think I would prefer to keep it in sync and wait for a few > years to consider that. > > Best, > > Sebastian > Could this approach be used to deprecate `ravel` and let us just use `flatten`? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Jan 10 02:49:14 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Jan 2019 02:49:14 -0500 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: On Thu, Jan 10, 2019, 01:55 Eric Wieser Slicing is a lot more important than some keyword. And design-wise, > filling the numpy namespace with singletons for keyword to other things in > that same namespace just makes no sense to me. > > At least from the perspective of discoverability, you could argue that > string constants form a namespace of their won, and so growing the ?string? > namespace is not inherently better than growing any other. The main flaw in > that comparison is that picking np.never_copy to be a singleton forever > prevents us reusing that name to become a function. > > Perhaps the solution is to use np.NEVER_COPY instead - that?s never going > to clash with a function name we want to add in future, and using upper > attributes as arguments in that way is pretty typical for python ( > subprocess.PIPE, socket.SOCK_STREAM, etc?) > What about a namespace as someone mentioned earlier. Perhaps `np.flags.NO_COPY` You could fairly argue that this approach is outdated in the face of > enum.Enum - in which case we could go for the more heavy-handed > np.CopyMode.NEVER, which still has a unique enough case for name clashes > with functions never to be an issue. > > Eric > Would all three conditions be supported this way or only `NEVER`? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rainwoodman at gmail.com Thu Jan 10 14:21:17 2019 From: rainwoodman at gmail.com (Feng Yu) Date: Thu, 10 Jan 2019 11:21:17 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: Hi Todd, I agree a flag is more suitable than classes. I would add another bonus of a flag than a function argument is to avoid massive contamination of function signatures for a global variation of behavior that affects many functions. Yu On Wed, Jan 9, 2019 at 11:34 PM Todd wrote: > On Mon, Jan 7, 2019, 14:22 Feng Yu >> Hi, >> >> Was it ever brought up the possibility of a new array class (ndrefonly, >> ndview) that is strictly no copy? >> >> All operations on ndrefonly will return ndrefonly and if the operation >> cannot be completed without making a copy, it shall throw an error. >> >> On the implementation there are two choices if we use subclasses: >> >> - ndrefonly can be a subclass of ndarray. The pattern would be subclass >> limiting functionality of super, but ndrefonly is a ndarray. >> - ndarray as a subclass of ndarray. Subclass supplements functionality of >> super. : ndarray will not throw an error when a copy is necessary. However >> ndarray is not a ndarray. >> >> If we want to be wild they do not even need to be subclasses of each >> other, or maybe they shall both be subclasses of something more >> fundamental. >> >> - Yu >> > > I would prefer a flag for this. Someone can make an array read-only by > setting `arr.flags.writable=False`. So along those lines, we could have a > `arr.flags.copyable` flag that if set to `False` would result in an error > of any operation tried to copy the data. > >> _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Jan 10 14:38:23 2019 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 10 Jan 2019 14:38:23 -0500 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: constants are easier to support for autocompletion than strings. My current env (emacs) will (usually) autocomplete the former, but not the latter. On Thu, Jan 10, 2019 at 2:21 PM Feng Yu wrote: > Hi Todd, > > I agree a flag is more suitable than classes. > > I would add another bonus of a flag than a function argument is to avoid > massive contamination of function signatures for a global variation of > behavior that affects many functions. > > Yu > > On Wed, Jan 9, 2019 at 11:34 PM Todd wrote: > >> On Mon, Jan 7, 2019, 14:22 Feng Yu > >>> Hi, >>> >>> Was it ever brought up the possibility of a new array class (ndrefonly, >>> ndview) that is strictly no copy? >>> >>> All operations on ndrefonly will return ndrefonly and if the operation >>> cannot be completed without making a copy, it shall throw an error. >>> >>> On the implementation there are two choices if we use subclasses: >>> >>> - ndrefonly can be a subclass of ndarray. The pattern would be subclass >>> limiting functionality of super, but ndrefonly is a ndarray. >>> - ndarray as a subclass of ndarray. Subclass supplements functionality >>> of super. : ndarray will not throw an error when a copy is necessary. >>> However ndarray is not a ndarray. >>> >>> If we want to be wild they do not even need to be subclasses of each >>> other, or maybe they shall both be subclasses of something more >>> fundamental. >>> >>> - Yu >>> >> >> I would prefer a flag for this. Someone can make an array read-only by >> setting `arr.flags.writable=False`. So along those lines, we could have a >> `arr.flags.copyable` flag that if set to `False` would result in an error >> of any operation tried to copy the data. >> >>> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Thu Jan 10 21:57:47 2019 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Fri, 11 Jan 2019 13:57:47 +1100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> Message-ID: > On 10 Jan 2019, at 6:35 pm, Todd wrote: > > Could this approach be used to deprecate `ravel` and let us just use `flatten`? Could we not? `.ravel()` is everywhere and it matches `ravel_multi_index` and `unravel_index`. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 10 23:25:26 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 10 Jan 2019 20:25:26 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <429011baa0cb7f19f20db84dfc65276c76eaccf9.camel@sipsolutions.net> <65f922ff87816cc2195b2ce2f59ea3671b29099a.camel@sipsolutions.net> Message-ID: On Wed, Jan 9, 2019 at 10:55 PM Eric Wieser wrote: > Slicing is a lot more important than some keyword. And design-wise, > filling the numpy namespace with singletons for keyword to other things in > that same namespace just makes no sense to me. > > At least from the perspective of discoverability, you could argue that > string constants form a namespace of their won, and so growing the ?string? > namespace is not inherently better than growing any other. > I don't really think those are valid arguments. Strings are not a namespace, and if you want to make the analogy then it's at best a namespace within the functions/methods that the strings apply to. Discoverability: what is valid input for a keyword is discovered pretty much exclusively by reading the docstring or html doc page of the function, and not the docstring of some random object in the numpy namespace. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 10 23:27:47 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 10 Jan 2019 20:27:47 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: On Thu, Jan 10, 2019 at 11:21 AM Feng Yu wrote: > Hi Todd, > > I agree a flag is more suitable than classes. > > I would add another bonus of a flag than a function argument is to avoid > massive contamination of function signatures for a global variation of > behavior that affects many functions. > I like this suggestion. Copy behavior fits very nicely with existing flags (e.g. UPDATEIFCOPY, WRITEABLE) and avoids both namespace pollution and complication docstrings. Ralf > Yu > > On Wed, Jan 9, 2019 at 11:34 PM Todd wrote: > >> On Mon, Jan 7, 2019, 14:22 Feng Yu > >>> Hi, >>> >>> Was it ever brought up the possibility of a new array class (ndrefonly, >>> ndview) that is strictly no copy? >>> >>> All operations on ndrefonly will return ndrefonly and if the operation >>> cannot be completed without making a copy, it shall throw an error. >>> >>> On the implementation there are two choices if we use subclasses: >>> >>> - ndrefonly can be a subclass of ndarray. The pattern would be subclass >>> limiting functionality of super, but ndrefonly is a ndarray. >>> - ndarray as a subclass of ndarray. Subclass supplements functionality >>> of super. : ndarray will not throw an error when a copy is necessary. >>> However ndarray is not a ndarray. >>> >>> If we want to be wild they do not even need to be subclasses of each >>> other, or maybe they shall both be subclasses of something more >>> fundamental. >>> >>> - Yu >>> >> >> I would prefer a flag for this. Someone can make an array read-only by >> setting `arr.flags.writable=False`. So along those lines, we could have a >> `arr.flags.copyable` flag that if set to `False` would result in an error >> of any operation tried to copy the data. >> >>> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 11 08:58:21 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Jan 2019 14:58:21 +0100 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> Message-ID: <6c67069775c35d6d45127b268822c5ca70c5c6f1.camel@sipsolutions.net> On Fri, 2019-01-11 at 13:57 +1100, Juan Nunez-Iglesias wrote: > > > > On 10 Jan 2019, at 6:35 pm, Todd wrote: > > > > Could this approach be used to deprecate `ravel` and let us just > > use `flatten`? > > > Could we not? `.ravel()` is everywhere and it matches > `ravel_multi_index` and `unravel_index`. True, I suppose either of those functions could get such an argument as well. Just side note `arr.ravel()` is also not quite equivalent to `arr.reshape(-1)`, since it actually provides some additional assurances on the result contiguity. - Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From kwgoodman at gmail.com Fri Jan 11 15:32:47 2019 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 11 Jan 2019 12:32:47 -0800 Subject: [Numpy-discussion] summation along a non-fast axis Message-ID: I remember back when a.sum(axis=0) was much slower than a.sum(axis=1) for something like a=np.ones((1000, 1000)). But now it runs in about the same time. How does numpy do it? Does numpy do something like for i in range(a.shape[0]): for j in range(x.shape[1]): result[j] += a[i, j] -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jan 11 16:24:48 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Jan 2019 22:24:48 +0100 Subject: [Numpy-discussion] summation along a non-fast axis In-Reply-To: References: Message-ID: <31947aade6d8399c2028fe859f930f6b3cb6958f.camel@sipsolutions.net> On Fri, 2019-01-11 at 12:32 -0800, Keith Goodman wrote: > I remember back when a.sum(axis=0) was much slower than a.sum(axis=1) > for something like a=np.ones((1000, 1000)). But now it runs in about > the same time. How does numpy do it? > "now" is since numpy 1.7 or so :). > Does numpy do something like > > for i in range(a.shape[0]): > for j in range(x.shape[1]): > result[j] += a[i, j] Yeah, numpy reorders the operation. Maybe a bit closer to what happens is to write it down with the result as a 2D array (as happens with keepdims), since internally that is how numpy operates on it (although it may optimize the `i*0` away): for i in range(a.shape[0]): for j in range(a.shape[1]): # If sum is along axis 0: result[i*0, j] += a[i, j] Since it doesn't matter which of the loop is the innermost one, the machinery is capable of reordering them. I think it learned it with 1.7 (because that added a lot), but maybe it was even earlier. - Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From wieser.eric+numpy at gmail.com Sun Jan 13 02:21:08 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Sat, 12 Jan 2019 23:21:08 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: I don?t think a NEVERCOPY entry in arr.flags would make much sense. Is this really a sensible limitation to put on how data gets used? Isn?t it up to the algorithm to decide whether to copy its data, not the original owner of the data? It also leads to some tricky questions of precedence - would np.array(arr, copy=True) respect the flag or the argument? How about np.array(arr)? Is arr + 0 considered a copy? By keeping it as a value passed in via a copy= kwarg, we don?t need to answer any of those questions. Eric On Thu, 10 Jan 2019 at 20:28 Ralf Gommers ralf.gommers at gmail.com wrote: On Thu, Jan 10, 2019 at 11:21 AM Feng Yu wrote: > >> Hi Todd, >> >> I agree a flag is more suitable than classes. >> >> I would add another bonus of a flag than a function argument is to avoid >> massive contamination of function signatures for a global variation of >> behavior that affects many functions. >> > > I like this suggestion. Copy behavior fits very nicely with existing flags > (e.g. UPDATEIFCOPY, WRITEABLE) and avoids both namespace pollution and > complication docstrings. > > Ralf > > >> Yu >> >> On Wed, Jan 9, 2019 at 11:34 PM Todd wrote: >> >>> On Mon, Jan 7, 2019, 14:22 Feng Yu >> >>>> Hi, >>>> >>>> Was it ever brought up the possibility of a new array class (ndrefonly, >>>> ndview) that is strictly no copy? >>>> >>>> All operations on ndrefonly will return ndrefonly and if the operation >>>> cannot be completed without making a copy, it shall throw an error. >>>> >>>> On the implementation there are two choices if we use subclasses: >>>> >>>> - ndrefonly can be a subclass of ndarray. The pattern would be subclass >>>> limiting functionality of super, but ndrefonly is a ndarray. >>>> - ndarray as a subclass of ndarray. Subclass supplements functionality >>>> of super. : ndarray will not throw an error when a copy is necessary. >>>> However ndarray is not a ndarray. >>>> >>>> If we want to be wild they do not even need to be subclasses of each >>>> other, or maybe they shall both be subclasses of something more >>>> fundamental. >>>> >>>> - Yu >>>> >>> >>> I would prefer a flag for this. Someone can make an array read-only by >>> setting `arr.flags.writable=False`. So along those lines, we could have a >>> `arr.flags.copyable` flag that if set to `False` would result in an error >>> of any operation tried to copy the data. >>> >>>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagru at mailbox.org Sun Jan 13 08:33:35 2019 From: lagru at mailbox.org (Lars Grueter) Date: Sun, 13 Jan 2019 14:33:35 +0100 Subject: [Numpy-discussion] Debugging NumPy development build in conda environment In-Reply-To: References: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> Message-ID: On 04/01/2019 14:55, Matti Picus wrote: > On 4/1/19 3:34 pm, Lars Grueter wrote: >> Unfortunately the last two commands >> stopped working out of the blue with my existing environment and fail >> with >> >>> RuntimeError: Broken toolchain: cannot link a simple C program > > > The compiler error is a few lines above this (copied below). When I have > gotten that kind of output, it is because I am using a 32 bit compiler > on 64 bit or the opposite. > > [...] > > Can you compile anything with that gcc? This problem even persists on a new system. I noticed that conda includes libgcc-ng [1] in every environment with Python. Is it possible that this library (or another one) is incompatible with the GCC version installed on my system? That would also explain why changing the available LA libraries has not effect and why the setup works fine using plain virtual environments created with venv. Is someone else successfully maintaining a development setup with conda on an up-to-date Arch-based system or even Manjaro? Best regards, Lars [1] http://docs.continuum.io/anaconda/packages/py3.7_linux-64/ From lagru at mailbox.org Sun Jan 13 08:44:59 2019 From: lagru at mailbox.org (Lars Grueter) Date: Sun, 13 Jan 2019 14:44:59 +0100 Subject: [Numpy-discussion] Debugging NumPy development build in conda environment In-Reply-To: <1dfdf9da-0917-46e5-ad52-0d85a81f761c@Canary> References: <091949a2-47a5-8ea9-c6f7-21a36136024e@mailbox.org> <1dfdf9da-0917-46e5-ad52-0d85a81f761c@Canary> Message-ID: <76f8cec2-6630-0532-684d-11f0537f4f13@mailbox.org> Sorry, I just noticed that I didn't send the message below to the mailing list but replied directly to Hameer. I've resent it for the archives below. @Hameer, sorry for the spam. :) Best regards, Lars On 04/01/2019 20:06, Lars Grueter wrote:> Hi Hameer, > > thanks for the fast answer! I'm even more confused now. If any of my > assumptions below are wrong please point them out. > > On 04.01.19 14:53, Hameer Abbasi wrote: >> Hi Lars, >> >> What I?m used to doing is simply the following (inside the NumPy root): >> >> conda create -n numpy-dev python=3[.x] pytest >> conda activate numpy-dev >> python setup.py build_ext --inplace -j 4 > > This fails right here telling me that it doesn't find any LA library > such as ATLAS, BLIS, MKL, etc. (assuming I read the output correctly). > The error message is however different from the workflow I described in > my first mail. It's missing the sections ? la > > lapack_mkl_info: > customize UnixCCompiler > FOUND: > libraries = ['mkl_rt', 'pthread'] > > I checked my OS's package manager and found blas, cblas and lapack. > Aren't these LA libraries? Am I still missing an OS version of either > ATLAS, BLIS or MKL? > > https://docs.scipy.org/doc/numpy/user/building.html#disabling-atlas-and-other-accelerated-libraries > tells me it should be possible to build NumPy without these libraries. > This has no effect and won't build. > > But if that's the case why is the build working inside an environment > created with virtualenv? Is it building without those libraries? > >> Often, when I run into issues with the build, I simply do this: >> >> [backup site.cfg] >> git clean -xfd >> [restore site.cfg] >> python setup.py build_ext --inplace -j 4 >> pip install -e . > >> In your case, it might be the second that helps. If it?s not finding any >> LA library, you can try compiling with the site.cfg that points to MKL. > > I didn't yet need my own site.cfg. I tried creating one myself right now > but regardless of how I try to point it to the conda envs lib and > include directories it has now effect on the output of the setup.py > command. The build script already finds these paths by default. E.g.: > > [mkl] > library_dirs = /home/lg/.miniconda3/envs/dev-numpy/lib > include_dirs = /home/lg/.miniconda3/envs/dev-numpy/include > mkl_libs = mkl_rt > lapack_libs = > > I'm not sure what else to provide in this file. > > Thanks again and best regards, > Lars From rainwoodman at gmail.com Sun Jan 13 15:05:17 2019 From: rainwoodman at gmail.com (Feng Yu) Date: Sun, 13 Jan 2019 12:05:17 -0800 Subject: [Numpy-discussion] Add guaranteed no-copy to array creation and reshape? In-Reply-To: References: <629842683a77aed9acfe7ca2f00a55e164abf6ac.camel@sipsolutions.net> <58cb2bf7b0b058590687bbbba14b526c1c276a58.camel@sipsolutions.net> <9ad06acc-93f3-436c-8df6-20a7eb2b2643@www.fastmail.com> <1496454a35d93bed6dece86fc38526148f48be98.camel@sipsolutions.net> Message-ID: Eric, I agree these are questions that shall be answered. I think issues you raised are toward functions always make a copy -- combining them with NEVERCOPY does sound sane. Your argument is that If the atom of the new behavior is per method, then there is no need to worry about those functions that does not sound sane when combined with NEVERCOPY. Here is a proposal that may address your concerns: 1. The name of the flag does not need to be NEVERCOPY -- especially it sounds insane combined with many methods. I think something like KEEPBASE may be easier to reason. The name KEEPBASE alwsy suggests it shall not be set on an object where base is None; this feature may simplify the matter. 2. Methods that creates copy shall always create a copy, regardless of a flag -- that's the definition of copy. (np.copy) KEEPBASE shall only affect methods that creates a new object object, which may reference the provided base or create a new base. 3. Functions that are not a ndarray method, shall ignore the flag, unless specified otherwise. (np.array) 4. arr + 0 when arr is KEEPBASE. It depends on if we think this triggered np.add(), or ndarray.__add__. I think ndarray.__add__ is a shortcut to call the ufunc np.add(). Thus the result of arr + 0 shall be the behavior on the ufunc np.add, which shall ignore the flag. Notice that in memory tight situations, users can already use inplace operations to avoid copies. This verbose but highly controlled syntax may be preferable than inferring an automated behavior that relies on KEEPBASE of arguments. 5. I think on the operation level, the difference between subclassing(flags) and function arguments would be: array.view(keepbase=True).reshape(-1) vs array.reshape(-1, keepbase=True) The gain is that no array method needs to be modified. The control can still be at the level of the algorithm. On Sat, Jan 12, 2019 at 11:22 PM Eric Wieser wrote: > I don?t think a NEVERCOPY entry in arr.flags would make much sense. > Is this really a sensible limitation to put on how data gets used? Isn?t > it up to the algorithm to decide whether to copy its data, not the original > owner of the data? > > It also leads to some tricky questions of precedence - would np.array(arr, > copy=True) respect the flag or the argument? How about np.array(arr)? Is arr > + 0 considered a copy? > By keeping it as a value passed in via a copy= kwarg, we don?t need to > answer any of those questions. > > Eric > > On Thu, 10 Jan 2019 at 20:28 Ralf Gommers ralf.gommers at gmail.com > wrote: > > On Thu, Jan 10, 2019 at 11:21 AM Feng Yu wrote: >> >>> Hi Todd, >>> >>> I agree a flag is more suitable than classes. >>> >>> I would add another bonus of a flag than a function argument is to avoid >>> massive contamination of function signatures for a global variation of >>> behavior that affects many functions. >>> >> >> I like this suggestion. Copy behavior fits very nicely with existing >> flags (e.g. UPDATEIFCOPY, WRITEABLE) and avoids both namespace pollution >> and complication docstrings. >> >> Ralf >> >> >>> Yu >>> >>> On Wed, Jan 9, 2019 at 11:34 PM Todd wrote: >>> >>>> On Mon, Jan 7, 2019, 14:22 Feng Yu >>> >>>>> Hi, >>>>> >>>>> Was it ever brought up the possibility of a new array class >>>>> (ndrefonly, ndview) that is strictly no copy? >>>>> >>>>> All operations on ndrefonly will return ndrefonly and if the operation >>>>> cannot be completed without making a copy, it shall throw an error. >>>>> >>>>> On the implementation there are two choices if we use subclasses: >>>>> >>>>> - ndrefonly can be a subclass of ndarray. The pattern would be >>>>> subclass limiting functionality of super, but ndrefonly is a ndarray. >>>>> - ndarray as a subclass of ndarray. Subclass supplements functionality >>>>> of super. : ndarray will not throw an error when a copy is necessary. >>>>> However ndarray is not a ndarray. >>>>> >>>>> If we want to be wild they do not even need to be subclasses of each >>>>> other, or maybe they shall both be subclasses of something more >>>>> fundamental. >>>>> >>>>> - Yu >>>>> >>>> >>>> I would prefer a flag for this. Someone can make an array read-only by >>>> setting `arr.flags.writable=False`. So along those lines, we could have a >>>> `arr.flags.copyable` flag that if set to `False` would result in an error >>>> of any operation tried to copy the data. >>>> >>>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > ? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 13 22:29:54 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Jan 2019 20:29:54 -0700 Subject: [Numpy-discussion] NumPy 1.16.0 released. Message-ID: Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.0. This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are Experimental support for overriding numpy functions in downstream projects. - The matmul function is now a ufunc and can be overridden using __array_ufunc__. - Improved support for the ARM, POWER, and SPARC architectures. - Improved support for AIX and PyPy. - Improved interoperation with ctypes. - Improved support for PEP 3118. The supported Python versions are 2.7 and 3.5-3.7, support for 3.4 has been dropped. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4. Wheels for this release can be downloaded from PyPI , source archives are available from Github . *Contributors* A total of 113 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Alan Fontenot + * Allan Haldane * Alon Hershenhorn + * Alyssa Quek + * Andreas Nussbaumer + * Anner + * Anthony Sottile + * Antony Lee * Ayappan P + * Bas van Schaik + * C.A.M. Gerlach + * Charles Harris * Chris Billington * Christian Clauss * Christoph Gohlke * Christopher Pezley + * Daniel B Allan + * Daniel Smith * Dawid Zych + * Derek Kim + * Dima Pasechnik + * Edgar Giovanni Lepe + * Elena Mokeeva + * Elliott Sales de Andrade + * Emil Hessman + * Eric Larson * Eric Schles + * Eric Wieser * Giulio Benetti + * Guillaume Gautier + * Guo Ci * Heath Henley + * Isuru Fernando + * J. Lewis Muir + * Jack Vreeken + * Jaime Fernandez * James Bourbeau * Jeff VanOss * Jeffrey Yancey + * Jeremy Chen + * Jeremy Manning + * Jeroen Demeyer * John Darbyshire + * John Kirkham * John Zwinck * Jonas Jensen + * Joscha Reimer + * Juan Azcarreta + * Julian Taylor * Kevin Sheppard * Krzysztof Chomski + * Kyle Sunden * Lars Gr?ter * Lilian Besson + * MSeifert04 * Mark Harfouche * Marten van Kerkwijk * Martin Thoma * Matt Harrigan + * Matthew Bowden + * Matthew Brett * Matthias Bussonnier * Matti Picus * Max Aifer + * Michael Hirsch, Ph.D + * Michael James Jamie Schnaitter + * MichaelSaah + * Mike Toews * Minkyu Lee + * Mircea Akos Bruma + * Mircea-Akos Brum? + * Moshe Looks + * Muhammad Kasim + * Nathaniel J. Smith * Nikita Titov + * Paul M?ller + * Paul van Mulbregt * Pauli Virtanen * Pierre Glaser + * Pim de Haan * Ralf Gommers * Robert Kern * Robin Aggleton + * Rohit Pandey + * Roman Yurchak + * Ryan Soklaski * Sebastian Berg * Sho Nakamura + * Simon Gibbons * Stan Seibert + * Stefan Otte * Stefan van der Walt * Stephan Hoyer * Stuart Archibald * Taylor Smith + * Tim Felgentreff + * Tim Swast + * Tim Teichmann + * Toshiki Kataoka * Travis Oliphant * Tyler Reddy * Uddeshya Singh + * Warren Weckesser * Weitang Li + * Wenjamin Petrenko + * William D. Irons * Yannick Jadoul + * Yaroslav Halchenko * Yug Khanna + * Yuji Kanagawa + * Yukun Guo + * @ankokumoyashi + * @lerbuke + Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at stuartreynolds.net Mon Jan 14 19:28:08 2019 From: stuart at stuartreynolds.net (Stuart Reynolds) Date: Mon, 14 Jan 2019 16:28:08 -0800 Subject: [Numpy-discussion] py3k.os_fspath enforces inconsist rules with python3 os.fspath Message-ID: After a recent upgrade of numpy (1.13.1 -> 1.6.0), my code is failing where I provide unicode objects as filenames. Previously they were allowed. Now that are not, and I *must* provide a (py2) str or bytearray only. # str is OK $ python2.7 -c "from numpy.compat import py3k; print py3k.os_fspath('123')" 123 # unicode is not $ python -c "from numpy.compat import py3k; print py3k.os_fspath(u'123')" Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/numpy/compat/py3k.py", line 237, in os_fspath "not " + path_type.__name__) TypeError: expected str, bytes or os.PathLike object, not unicode But this enforcement of "str, bytes or os.PathLike" comes from: https://docs.python.org/3/library/os.html where in Python 3 str is a unicode, and moreover, os.fspath allows $ python3 -c "import os; print(os.fspath(u'123'))" # unicode str 123 $ python3 -c "import os; print(os.fspath('123'))" # also unicode str 123 .... so... shouldn't py3k.os_fspath allow py2 unicode obejcts. - Stu -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Mon Jan 14 22:09:40 2019 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Mon, 14 Jan 2019 19:09:40 -0800 Subject: [Numpy-discussion] py3k.os_fspath enforces inconsist rules with python3 os.fspath In-Reply-To: References: Message-ID: This looks like a bug to me - can you file it on the issue tracker. Evidently I did not consider python 2 behavior when backporting `os.fspath` from python 3. Eric On Mon, 14 Jan 2019 at 16:28 Stuart Reynolds wrote: > After a recent upgrade of numpy (1.13.1 -> 1.6.0), my code is failing > where I provide unicode objects as filenames. > Previously they were allowed. Now that are not, and I *must* provide a > (py2) str or bytearray only. > > # str is OK > $ python2.7 -c "from numpy.compat import py3k; print py3k.os_fspath('123')" > 123 > > # unicode is not > $ python -c "from numpy.compat import py3k; print py3k.os_fspath(u'123')" > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/numpy/compat/py3k.py", line > 237, in os_fspath > "not " + path_type.__name__) > TypeError: expected str, bytes or os.PathLike object, not unicode > > But this enforcement of "str, bytes or os.PathLike" comes from: > https://docs.python.org/3/library/os.html > where in Python 3 str is a unicode, and moreover, os.fspath allows > > $ python3 -c "import os; print(os.fspath(u'123'))" # unicode str > 123 > $ python3 -c "import os; print(os.fspath('123'))" # also unicode str > 123 > > .... so... shouldn't py3k.os_fspath allow py2 unicode obejcts. > > - Stu > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at stuartreynolds.net Tue Jan 15 11:08:09 2019 From: stuart at stuartreynolds.net (Stuart Reynolds) Date: Tue, 15 Jan 2019 08:08:09 -0800 Subject: [Numpy-discussion] py3k.os_fspath enforces inconsist rules with python3 os.fspath In-Reply-To: References: Message-ID: Will do. On Mon, Jan 14, 2019 at 7:10 PM Eric Wieser wrote: > This looks like a bug to me - can you file it on the issue tracker. > > Evidently I did not consider python 2 behavior when backporting > `os.fspath` from python 3. > > Eric > > On Mon, 14 Jan 2019 at 16:28 Stuart Reynolds > wrote: > >> After a recent upgrade of numpy (1.13.1 -> 1.6.0), my code is failing >> where I provide unicode objects as filenames. >> Previously they were allowed. Now that are not, and I *must* provide a >> (py2) str or bytearray only. >> >> # str is OK >> $ python2.7 -c "from numpy.compat import py3k; print >> py3k.os_fspath('123')" >> 123 >> >> # unicode is not >> $ python -c "from numpy.compat import py3k; print py3k.os_fspath(u'123')" >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/local/lib/python2.7/dist-packages/numpy/compat/py3k.py", >> line 237, in os_fspath >> "not " + path_type.__name__) >> TypeError: expected str, bytes or os.PathLike object, not unicode >> >> But this enforcement of "str, bytes or os.PathLike" comes from: >> https://docs.python.org/3/library/os.html >> where in Python 3 str is a unicode, and moreover, os.fspath allows >> >> $ python3 -c "import os; print(os.fspath(u'123'))" # unicode str >> 123 >> $ python3 -c "import os; print(os.fspath('123'))" # also unicode str >> 123 >> >> .... so... shouldn't py3k.os_fspath allow py2 unicode obejcts. >> >> - Stu >> > _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at stuartreynolds.net Tue Jan 15 12:03:45 2019 From: stuart at stuartreynolds.net (Stuart Reynolds) Date: Tue, 15 Jan 2019 09:03:45 -0800 Subject: [Numpy-discussion] py3k.os_fspath enforces inconsist rules with python3 os.fspath In-Reply-To: References: Message-ID: Was recently already submitted: https://github.com/numpy/numpy/issues/12749 On Tue, Jan 15, 2019 at 8:08 AM Stuart Reynolds wrote: > Will do. > > On Mon, Jan 14, 2019 at 7:10 PM Eric Wieser > wrote: > >> This looks like a bug to me - can you file it on the issue tracker. >> >> Evidently I did not consider python 2 behavior when backporting >> `os.fspath` from python 3. >> >> Eric >> >> On Mon, 14 Jan 2019 at 16:28 Stuart Reynolds >> wrote: >> >>> After a recent upgrade of numpy (1.13.1 -> 1.6.0), my code is failing >>> where I provide unicode objects as filenames. >>> Previously they were allowed. Now that are not, and I *must* provide a >>> (py2) str or bytearray only. >>> >>> # str is OK >>> $ python2.7 -c "from numpy.compat import py3k; print >>> py3k.os_fspath('123')" >>> 123 >>> >>> # unicode is not >>> $ python -c "from numpy.compat import py3k; print py3k.os_fspath(u'123')" >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "/usr/local/lib/python2.7/dist-packages/numpy/compat/py3k.py", >>> line 237, in os_fspath >>> "not " + path_type.__name__) >>> TypeError: expected str, bytes or os.PathLike object, not unicode >>> >>> But this enforcement of "str, bytes or os.PathLike" comes from: >>> https://docs.python.org/3/library/os.html >>> where in Python 3 str is a unicode, and moreover, os.fspath allows >>> >>> $ python3 -c "import os; print(os.fspath(u'123'))" # unicode str >>> 123 >>> $ python3 -c "import os; print(os.fspath('123'))" # also unicode str >>> 123 >>> >>> .... so... shouldn't py3k.os_fspath allow py2 unicode obejcts. >>> >>> - Stu >>> >> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Jan 15 18:12:32 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 16 Jan 2019 01:12:32 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Jan 16 at 12:00 pacific time Message-ID: The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. The BIDS team From raghuveer.devulapalli at intel.com Wed Jan 16 11:42:17 2019 From: raghuveer.devulapalli at intel.com (Devulapalli, Raghuveer) Date: Wed, 16 Jan 2019 16:42:17 +0000 Subject: [Numpy-discussion] Transcendental Functions Message-ID: Hello, Are Transcendental Functions SIMD vectorized in NumPy? Raghuveer -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at stuartreynolds.net Thu Jan 17 14:54:22 2019 From: stuart at stuartreynolds.net (Stuart Reynolds) Date: Thu, 17 Jan 2019 11:54:22 -0800 Subject: [Numpy-discussion] numba: maps, and list of vectors Message-ID: I'm having great luck using with numba, but there are two problems I find difficult to solve with it. Often I want to algorithms that performs a lookup of a list/array of numbers. And this operation often comes in two flavors: (1) => # datastructure.get(0) => array (2) => # datastructure.get(999999999999) => array In numba, how to people deal with indexed lookup where the range of the data structure is a memory managed object (array/list) (1)? In numba, how to people deal with problems needing hashed lookups (2)? Are there extension libraries / ways of calling out to C/C++ libraries that can easily help. (I know Cython is a route, but I am asking about numba specifically). Thanks, - Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jan 17 15:04:27 2019 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Jan 2019 14:04:27 -0600 Subject: [Numpy-discussion] numba: maps, and list of vectors In-Reply-To: References: Message-ID: The numba mailing list is here: https://groups.google.com/a/continuum.io/forum/#!forum/numba-users On Thu, Jan 17, 2019 at 1:55 PM Stuart Reynolds wrote: > I'm having great luck using with numba, but there are two problems I find > difficult to solve with it. > > Often I want to algorithms that performs a lookup of a list/array of > numbers. And this operation often comes in two flavors: > (1) => # datastructure.get(0) => array > (2) => # datastructure.get(999999999999) => > array > > In numba, how to people deal with indexed lookup where the range of the > data structure is a memory managed object (array/list) (1)? > > In numba, how to people deal with problems needing hashed lookups (2)? > > Are there extension libraries / ways of calling out to C/C++ libraries > that can easily help. (I know Cython is a route, but I am asking about > numba specifically). > > Thanks, > - Stuart > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Tue Jan 22 15:13:20 2019 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Tue, 22 Jan 2019 15:13:20 -0500 Subject: [Numpy-discussion] DOC: Updates to nditer usage instructions PR#12828 Message-ID: Hi, I have just added PR #12828, based off issue #12764 to clarify some of the documentation of `nditer`. While it contains most of the necessary material, it's still a bit of a rough draft, and I'd be happy to have some comments/advice on it. In particular, I didn't know how much to emphasize the fact that context management only became a thing for `nditer` in version 1.15.0. I'm also not sure about how much need there is to reiterate the C-style iteration methods vs the normal Python-style iteration. Regards, - Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoxrabinovitz at gmail.com Tue Jan 22 17:12:47 2019 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Tue, 22 Jan 2019 17:12:47 -0500 Subject: [Numpy-discussion] ENH: Added option to suppress stout/err capture in tests PR #12829 Message-ID: Hi, I recently had some issues setting up gdb to fix some self-inflicted segfaults, so instead I ended up adding an option to suppress stdout/stderr capture by pytest in PR#12829. I think this is a useful feature to have in general, so I made this PR. The only problem with it is that there are no formal tests for it (I did verify that all the possible options work manually though). Regards, - Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Jan 23 02:25:44 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 23 Jan 2019 09:25:44 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Dec 19 at 12:00 pacific time Message-ID: <8558c405-d6d2-1f5a-537a-91ba14272eaf@gmail.com> The draft agenda is at https://hackmd.io/6N3r7yUtSHqUsijC-CEbJA?both Everyone is invited to join. Matti, Tyler and Stefan From wyupinghhu at 163.com Tue Jan 29 02:22:00 2019 From: wyupinghhu at 163.com (Yuping Wang) Date: Tue, 29 Jan 2019 15:22:00 +0800 (CST) Subject: [Numpy-discussion] Numpy-discussion Message-ID: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> Dear Nmupy developers and users: I am a new user of Numpy ,I have encountered a question about Numpy recently, which need your help. The question is below: I don know why there are negative numbers can somebody explain to me why these happens Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ??.JPG Type: image/jpeg Size: 34523 bytes Desc: not available URL: From kirit.thadaka at gmail.com Tue Jan 29 03:00:50 2019 From: kirit.thadaka at gmail.com (Kirit Thadaka) Date: Tue, 29 Jan 2019 00:00:50 -0800 Subject: [Numpy-discussion] Numpy-discussion In-Reply-To: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> References: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> Message-ID: You are seeing negative numbers because of an overflow beyond 32 bits. Change the type of your np array to int64 using a = (np.arange(2000) ** 3).astype(np.int64) and you won?t see negative numbers. I assume you are running this code on a 32 bit machine which is why Numpy is defaulting to int32. On a 64 bit machine it defaults to int64. Hope this helps. > On Jan 28, 2019, at 11:22 PM, Yuping Wang wrote: > > Dear Nmupy developers and users: > I am a new user of Numpy ,I have encountered a question about Numpy recently, which need your help. The question is below: > > > > > I don know why there are negative numbers > can somebody explain to me why these happens > Thanks in advance! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Jan 29 03:15:00 2019 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 29 Jan 2019 10:15:00 +0200 Subject: [Numpy-discussion] Numpy-discussion In-Reply-To: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> References: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> Message-ID: <0ba0d689-9074-4ee8-ac44-e90e0e71124e@gmail.com> On 29/1/19 9:22 am, Yuping Wang wrote: > Dear Nmupy developers and users: > ? ? ? ? I am a new user of Numpy ,I have encountered a question about > Numpy recently, which need your help. The question is below: > > > > ? ?? I don? know why there are negative numbers > ? ?? can somebody explain to me why these happens > Thanks in advance! > NumPy ndarrays are typed. Since you do not specify a type in arange, numpy assumes you want `a` to be the type of 2000. I assume you are on a system where numpy converts 2000 to an `int32`, so if you ask what is the type of a: (`a.dtype`) it will show `int32`, meaning the values in a are interpreted as if they are 32 bit integers. 2000*2000*2000 overflows a 32 bit signed integer, so it wraps around to a negative value. The way around this is to specify what type you wish to use: `a=np.arange(2000, dtype=float)**3` or `a=np.arange(2000, dtype='int64`)**3 will not overflow. For floats, most CPUs/compilers provide an indication when float values overflow, so we can raise a warning appropriately. Unfortunately, detecting overflows with ints has no hardware support, so adding a warning would mean checking each value before a calculation, causing an unacceptable slow down. Where did you start learning about NumPy? Perhaps you could suggest they add this explanation (or if it is in our documentation point out where you think would be appropriate) since it seems many people have problems with dtypes and overflow. Matti From stefanv at berkeley.edu Tue Jan 29 03:29:18 2019 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 29 Jan 2019 00:29:18 -0800 Subject: [Numpy-discussion] Numpy-discussion In-Reply-To: <0ba0d689-9074-4ee8-ac44-e90e0e71124e@gmail.com> References: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> <0ba0d689-9074-4ee8-ac44-e90e0e71124e@gmail.com> Message-ID: <20190129082918.swcxlkjucwj6w6we@carbo> On Tue, 29 Jan 2019 10:15:00 +0200, Matti Picus wrote: > 2000*2000*2000 overflows a 32 bit signed integer, so it wraps around to a > negative value. I was curious about what happens, bit-wise, in this case. We are dealing with a signed integer, so I presume of the 33 bits in 2000**3, only 32 are kept: In [60]: len("{0:b}".format(1999**3)) Out [60]: 33 In [61]: "{0:b}".format(1999**3) Out [61]: '111011100000111110100110001101111' In [62]: "{0:b}".format(1999**3)[-32:] Out [62]: '11011100000111110100110001101111' The first of the 32 bits indicates sign, so converting back to integer would give: 1 * np.iinfo(np.int32).min + int('1011100000111110100110001101111', 2) == -601928593 Best regards, St?fan From jslavin at cfa.harvard.edu Tue Jan 29 08:38:07 2019 From: jslavin at cfa.harvard.edu (Slavin, Jonathan) Date: Tue, 29 Jan 2019 08:38:07 -0500 Subject: [Numpy-discussion] negative numbers In-Reply-To: References: Message-ID: I'm not sure which numpy version you're using (I don't get that using numpy 1.14.3), but the issue is that for some reason the data type (dtype) of the array is being assumed to be a 4 byte integer or ' wrote: > From: Yuping Wang > To: numpy-discussion at python.org > Cc: > Bcc: > Date: Tue, 29 Jan 2019 15:22:00 +0800 (CST) > Subject: [Numpy-discussion] Numpy-discussion > Dear Nmupy developers and users: > I am a new user of Numpy ,I have encountered a question about > Numpy recently, which need your help. The question is below: > > > > > I don know why there are negative numbers > can somebody explain to me why these happens > Thanks in advance! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Jonathan D. Slavin Astrophysicist - High Energy Astrophysics Division Center for Astrophysics | Harvard & Smithsonian Office: (617) 496-7981 | Cell: (781) 363-0035 60 Garden Street | MS 83 | Cambridge, MA 02138 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ??.JPG Type: image/jpeg Size: 34523 bytes Desc: not available URL: From chris.barker at noaa.gov Tue Jan 29 20:45:54 2019 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 29 Jan 2019 17:45:54 -0800 Subject: [Numpy-discussion] Numpy-discussion In-Reply-To: <0ba0d689-9074-4ee8-ac44-e90e0e71124e@gmail.com> References: <38754356.657e.168987c7655.Coremail.wyupinghhu@163.com> <0ba0d689-9074-4ee8-ac44-e90e0e71124e@gmail.com> Message-ID: > On Jan 29, 2019, at 12:15 AM, Matti Picus wrote: >> Perhaps you could suggest they add this explanation (or if it is in our documentation point out where you think would be appropriate) since it seems many people have problems with dtypes and overflow. arange() is particularly problematic, as it defaults(in the common case) to integers, which are less familiar to new users than float64, which numpy uses as default dtype in most places. And many (most) use of arange() would be better served by lib space anyway. -CHB > > > Matti > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From wyupinghhu at 163.com Wed Jan 30 01:06:31 2019 From: wyupinghhu at 163.com (Yuping Wang) Date: Wed, 30 Jan 2019 14:06:31 +0800 (CST) Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 148, Issue 27 In-Reply-To: References: Message-ID: <27b9ae0e.3b35.1689d5db9cc.Coremail.wyupinghhu@163.com> I start to learn python from ?Numpy Beginner's Guide?second edition written by Ivan Idris now I know what the problem is ,the reason is that Numpy converts 2000 to an `int32`.When I use code `a=np.arange(2000, dtype='int64')**3 that you taught me, the calculation is correct thank you all ! Yuping Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Jan 30 07:30:52 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 30 Jan 2019 14:30:52 +0200 Subject: [Numpy-discussion] Reminder: weekly status meeting Jan 30 at 12:00 pacific time Message-ID: <3cebcd92-c0d8-1362-f0d0-dce57f10dd90@gmail.com> The draft agenda is at https://hackmd.io/UnnvbPNpSm-gRpbfTK7cmg?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. The BIDS team From koushik.naskar9 at gmail.com Wed Jan 30 12:10:46 2019 From: koushik.naskar9 at gmail.com (koushik naskar) Date: Wed, 30 Jan 2019 22:40:46 +0530 Subject: [Numpy-discussion] Compiling openmp implemented fortran code by f2py using ifort compiler Message-ID: Hi, I have a FORTRAN90 code, with openmp implemented for some parts. Now to compile the fortran code using f2py with openmp support I have to pass the f90 compiler flags. Now the code should compile with openmp support only if I provide the openmp relavant flag, unless it must be compiled as a normal serial code. This works as expected for gfortran but for ifort it doesn't. Lets explain this, if I run, (gfortran serial mode) *f2py -c adt.f90 -m adt_module* Gives output (last few lines) *Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops* *Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops* *Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops* (gfortran parallel mode) *f2py -c adt.f90 -m adt_module --f90flags='-fopenmp' -lgomp * output *Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops* *Fortran f90 compiler: /usr/bin/gfortran -fopenmp -fPIC -O3 -funroll-loops* *Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fopenmp -fPIC -O3 -funroll-loops* As you can clearly see there is no -fopenmp flag during the compilation in serial mode, as expected as I haven't pass the required flag *Now for ifort* (ifort parallel mode) *f2py -c adt.f90 -m adt_module --fcompiler=intelem --f90flags='-qopenmp' -liomp5* Output *Fortran f77 compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FI -fPIC -fp-model strict -O1 -qopenmp * *Fortran f90 compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FR -qopenmp -fPIC -fp-model strict -O1 -qopenmp * *Fortran fix compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FI -qopenmp -fPIC -fp-model strict -O1 -qopenmp* (ifort serial mode) *f2py -c adt.f90 -m adt_module --fcompiler=intelem* Output *Fortran f77 compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FI -fPIC -fp-model strict -O1 -qopenmp * *Fortran f90 compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FR -fPIC -fp-model strict -O1 -qopenmp * *Fortran fix compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort -FI -fPIC -fp-model strict -O1 -qopenmp* Now, here's the problem, see here compiling is done with -qopenmp flag for serial mode but I haven't passed it in command line. Why is it happening when compiling with ifort but not with gfortran ? And how to resolve this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From paterno at fnal.gov Wed Jan 30 19:27:37 2019 From: paterno at fnal.gov (Marc F Paterno) Date: Thu, 31 Jan 2019 00:27:37 +0000 Subject: [Numpy-discussion] Does numpy depend upon a Fortran library? Message-ID: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> Hello, I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib. While investigating the source of this problem, I downloaded the numpy source (https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation. I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS. I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing? thanks Marc From charlesr.harris at gmail.com Wed Jan 30 20:18:41 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2019 18:18:41 -0700 Subject: [Numpy-discussion] Does numpy depend upon a Fortran library? In-Reply-To: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> References: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> Message-ID: On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno wrote: > Hello, > > I have encountered a problem with a binary incompatibility between the > Fortran runtime library installed with numpy when using 'pip install --user > numpy', and that used by the rest of my program, which is built using > gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, > and GCC 8.2 provides libgfortran.5.dylib. > > While investigating the source of this problem, I downloaded the numpy > source > ( > https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip > ), > and tried building it. The resulting libraries have no coupling to any > Fortran library that I can find. I can find no Fortran source code files > in the numpy source, > except in tests or documentation. > > I am working on a MacBook laptop, running macOS Mojave, and so am using > the Accelerate framework to supply BLAS. > > I do not understand why the pip installation of numpy includes a Fortran > runtime library. Can someone explain to me what I am missing? > > That's interesting, it looks like the wheel includes four libraries: -rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 libopenblasp-r0.3.5.dev.dylib -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib I thought we only needed the openblas, but that in turn probably depends on libgcc. But why we have the fortran library and quadmath escapes me. Perhaps someone else knows. Note that compiling from source is different and will generally use different libraries. We don't use Accelerate because it is buggy, not thread safe, and it appears Apple is not interested in doing anything about that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 30 20:31:05 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 30 Jan 2019 17:31:05 -0800 Subject: [Numpy-discussion] Does numpy depend upon a Fortran library? In-Reply-To: References: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> Message-ID: On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris wrote: > > > On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno wrote: > >> Hello, >> >> I have encountered a problem with a binary incompatibility between the >> Fortran runtime library installed with numpy when using 'pip install --user >> numpy', and that used by the rest of my program, which is built using >> gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, >> and GCC 8.2 provides libgfortran.5.dylib. >> >> While investigating the source of this problem, I downloaded the numpy >> source >> ( >> https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip >> ), >> and tried building it. The resulting libraries have no coupling to any >> Fortran library that I can find. I can find no Fortran source code files >> in the numpy source, >> except in tests or documentation. >> >> I am working on a MacBook laptop, running macOS Mojave, and so am using >> the Accelerate framework to supply BLAS. >> >> I do not understand why the pip installation of numpy includes a Fortran >> runtime library. Can someone explain to me what I am missing? >> >> > That's interesting, it looks like the wheel includes four libraries: > > -rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib > -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib > -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 > libopenblasp-r0.3.5.dev.dylib > -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib > > I thought we only needed the openblas, but that in turn probably depends > on libgcc. But why we have the fortran library and quadmath escapes me. > Perhaps someone else knows. > I suspect it's a leftover from when we were using ATLAS, we did need a Fortran runtime library at some point. The cause will be somewhere in the numpy-wheel build scripts, there is a gfortran-install git submodule: https://github.com/MacPython/numpy-wheels > > Note that compiling from source is different and will generally use > different libraries. We don't use Accelerate because it is buggy, not > thread safe, and it appears Apple is not interested in doing anything about > that. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 30 21:03:15 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Jan 2019 19:03:15 -0700 Subject: [Numpy-discussion] Does numpy depend upon a Fortran library? In-Reply-To: References: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> Message-ID: On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers wrote: > > > On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno wrote: >> >>> Hello, >>> >>> I have encountered a problem with a binary incompatibility between the >>> Fortran runtime library installed with numpy when using 'pip install --user >>> numpy', and that used by the rest of my program, which is built using >>> gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, >>> and GCC 8.2 provides libgfortran.5.dylib. >>> >>> While investigating the source of this problem, I downloaded the numpy >>> source >>> ( >>> https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip >>> ), >>> and tried building it. The resulting libraries have no coupling to any >>> Fortran library that I can find. I can find no Fortran source code files >>> in the numpy source, >>> except in tests or documentation. >>> >>> I am working on a MacBook laptop, running macOS Mojave, and so am using >>> the Accelerate framework to supply BLAS. >>> >>> I do not understand why the pip installation of numpy includes a Fortran >>> runtime library. Can someone explain to me what I am missing? >>> >>> >> That's interesting, it looks like the wheel includes four libraries: >> >> -rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib >> -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib >> -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 >> libopenblasp-r0.3.5.dev.dylib >> -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib >> >> I thought we only needed the openblas, but that in turn probably depends >> on libgcc. But why we have the fortran library and quadmath escapes me. >> Perhaps someone else knows. >> > > I suspect it's a leftover from when we were using ATLAS, we did need a > Fortran runtime library at some point. The cause will be somewhere in the > numpy-wheel build scripts, there is a gfortran-install git submodule: > https://github.com/MacPython/numpy-wheels > And fortran is probably why the quadmath is there. Hmm, if we fix it we will need to test it... Chuck > > >> >> Note that compiling from source is different and will generally use >> different libraries. We don't use Accelerate because it is buggy, not >> thread safe, and it appears Apple is not interested in doing anything about >> that. >> >> Chuck >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Thu Jan 31 13:27:54 2019 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Thu, 31 Jan 2019 10:27:54 -0800 Subject: [Numpy-discussion] [ANN] pvlib-python v0.6.1: predicting power from PV (renweable solar energy) Message-ID: pvlib-0.6.1 is now available PyPI: https://pypi.org/project/pvlib/ Read the Docs: https://pvlib-python.readthedocs.io/en/latest/ GitHub: https://github.com/pvlib/pvlib-python -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 31 18:57:46 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 31 Jan 2019 15:57:46 -0800 Subject: [Numpy-discussion] Does numpy depend upon a Fortran library? In-Reply-To: References: <0F4FE3FA-50F9-421B-B323-3EFBA4EFBF2D@fnal.gov> Message-ID: On Wed, Jan 30, 2019 at 6:03 PM Charles R Harris wrote: > > > On Wed, Jan 30, 2019 at 6:32 PM Ralf Gommers > wrote: > >> >> >> On Wed, Jan 30, 2019 at 5:19 PM Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Wed, Jan 30, 2019 at 5:28 PM Marc F Paterno wrote: >>> >>>> Hello, >>>> >>>> I have encountered a problem with a binary incompatibility between the >>>> Fortran runtime library installed with numpy when using 'pip install --user >>>> numpy', and that used by the rest of my program, which is built using >>>> gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, >>>> and GCC 8.2 provides libgfortran.5.dylib. >>>> >>>> While investigating the source of this problem, I downloaded the numpy >>>> source >>>> ( >>>> https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64ce378259677de516bd899dd9eb8f9b32/numpy-1.16.0.zip >>>> ), >>>> and tried building it. The resulting libraries have no coupling to any >>>> Fortran library that I can find. I can find no Fortran source code files >>>> in the numpy source, >>>> except in tests or documentation. >>>> >>>> I am working on a MacBook laptop, running macOS Mojave, and so am using >>>> the Accelerate framework to supply BLAS. >>>> >>>> I do not understand why the pip installation of numpy includes a >>>> Fortran runtime library. Can someone explain to me what I am missing? >>>> >>>> >>> That's interesting, it looks like the wheel includes four libraries: >>> >>> -rw-r--r--. 1 charris charris 273072 Jan 1 1980 libgcc_s.1.dylib >>> -rwxr-xr-x. 1 charris charris 1550456 Jan 1 1980 libgfortran.3.dylib >>> -rwxr-xr-x. 1 charris charris 63433364 Jan 1 1980 >>> libopenblasp-r0.3.5.dev.dylib >>> -rwxr-xr-x. 1 charris charris 279932 Jan 1 1980 libquadmath.0.dylib >>> >>> I thought we only needed the openblas, but that in turn probably depends >>> on libgcc. But why we have the fortran library and quadmath escapes me. >>> Perhaps someone else knows. >>> >> >> I suspect it's a leftover from when we were using ATLAS, we did need a >> Fortran runtime library at some point. The cause will be somewhere in the >> numpy-wheel build scripts, there is a gfortran-install git submodule: >> https://github.com/MacPython/numpy-wheels >> > > And fortran is probably why the quadmath is there. Hmm, if we fix it we > will need to test it... > I opened an issue: https://github.com/MacPython/numpy-wheels/issues/42 Ralf > Chuck > >> >> >>> >>> Note that compiling from source is different and will generally use >>> different libraries. We don't use Accelerate because it is buggy, not >>> thread safe, and it appears Apple is not interested in doing anything about >>> that. >>> >>> Chuck >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jan 31 19:07:21 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 31 Jan 2019 17:07:21 -0700 Subject: [Numpy-discussion] NumPy 1.16.1 release Message-ID: Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.1. This release fixes bugs reported against the 1.16.0 release, and also backports several enhancements from master that seem appropriate for a release series that is the last to support Python 2.7. The supported Python versions are 2.7 and 3.5-3.7. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.4. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an *ImportError*. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 for discussion of the issue. Note that previously this problem resulted in an *AttributeError*. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Enhancements* * #12767: ENH: add mm->q floordiv * #12768: ENH: port np.core.overrides to C for speed * #12769: ENH: Add np.ctypeslib.as_ctypes_type(dtype), improve `np.ctypeslib.as_ctypes` * #12773: ENH: add "max difference" messages to np.testing.assert_array_equal... * #12820: ENH: Add mm->qm divmod * #12890: ENH: add _dtype_ctype to namespace for freeze analysis *Contributors* A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Antoine Pitrou * Arcesio Castaneda Medina + * Charles Harris * Chris Markiewicz + * Christoph Gohlke * Christopher J. Markiewicz + * Daniel Hrisca + * EelcoPeacs + * Eric Wieser * Kevin Sheppard * Matti Picus * OBATA Akio + * Ralf Gommers * Sebastian Berg * Stephan Hoyer * Tyler Reddy Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: