From indiajoe at gmail.com Thu Jun 1 12:18:41 2017 From: indiajoe at gmail.com (Joe P Ninan) Date: Thu, 1 Jun 2017 12:18:41 -0400 Subject: [Numpy-discussion] Input on np.ma.polyfit default behaviour Message-ID: Hi numpy users, This email is regarding the discussion on how np.ma.polyfit should deal with non-uniform mask across columns in 2D data array. Github Issue#9193 [ https://github.com/numpy/numpy/issues/9193 ] Current behaviour of np.ma.polyfit is to union combine all the column masks across all the columns and do the polynomial fit for all columns using this same mask. This has the potentially undesired behaviour of masking out lots of data across the columns if just one column has a certain row masked out. Since the polynomial fit of data in each column is independent, it seems to be an undesired behaviour. We were thinking to change this behaviour by fitting columns with different mask, separately. What are the opinions on this in regards to backward compatibility? Was there any usecase for the previous behaviour? Also, any implementation ideas/suggestions on the plan currently being discussed in Issue#9193 page. Thanks, -cheers Joe -- /--------------------------------------------------------------- "GNU/Linux: because a PC is a terrible thing to waste" - GNU Generation ************************************************ Joe Philip Ninan Postdoctoral Researcher 406 Davey Lab, Dept. of Astronomy & Astrophysics The Pennsylvania State University University Park, PA-16802 ------------------------------------------------------------ Website: https://indiajoe.gitlab.io/ My GnuPG Public Key: https://indiajoe.gitlab.io/files/JPN_public.key -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Fri Jun 2 10:23:04 2017 From: mailinglists at xgm.de (Florian Lindner) Date: Fri, 2 Jun 2017 16:23:04 +0200 Subject: [Numpy-discussion] Evaluate function of pairs from two meshgrids Message-ID: <7b1e4e15-e787-e671-1793-39104b17a1a8@xgm.de> Hello, I have 2*N arrays (let's say 2 arrays of length 4) that I combine using np.meshgrid xxA, yyA = np.meshgrid(xA, yA) xxB, yyB = np.meshgrid(xB, yB) which gives me two meshes xx.shape = yy.shape = (4,4) wish represent N-dimensional mesh with 16 elements. Now I want to evaluate a function f on every possible pair of N-dimensional points in the grid, resulting in a 16 x 16 matrix: in a flattened notation, pA = (xxA, yyA) f(pA[1]-pB[1]) f(pA[1]-pB[2]) f(pA[1]-pB[3]) ... f(pA[2]-pB[1]) f(pA[2]-pB[2]) f(pA[2]-pB[3]) ... f(pA[3]-pB[1]) f(pA[3]-pB[2]) f(pA[3]-pB[3]) ... . . . Let's say xA = yA = [1,2,3] and xB = yB = [10,20,30] that gives me a mesh A: (1,3) (2,3) (3,3) (1,2) (2,2) (3,2) (1,1) (2,1) (3,1) and a mesh B alike. My result matrix now should be of size 9 x 9: f( (1,3), (10,30) ) f( (2,3), (20,30) ) f( (3,3), (30, 30) ) f( (1,2), (10,20) ) f( (2,2), (20,20) ) f( (3,2), (30, 20) ) ... f always takes two N-dimensional vectors and returns a scalar. I hope I was able to explain what I want to achieve. What is the best way to do that in numpy? Thanks, Florian From mikhailwas at gmail.com Sun Jun 4 14:04:15 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 4 Jun 2017 20:04:15 +0200 Subject: [Numpy-discussion] Array and string interoperability Message-ID: Array and string interoperability Just sharing my thoughts and few ideas about simplification of casting strings to arrays. In examples assume Numpy is in the namespace (from numpy import *) Initialize array from a string currently looks like: s= "012 abc" A= fromstring(s,"u1") print A -> [48 49 50 32 97 98 99] Perfect. Now when writing values it will not work as IMO it should, namley consider this example: B= zeros(7,"u1") B[0]=s[1] print B -> [1 0 0 0 0 0 0] Ugh? It tries to parse the s[1] character "1" as integer and writes 1 to B[0]. First thing I would expect is a value error and I'd never expect it does that high-level manipulations with parsing. IMO ideally it would do the following instead: B[0]=s[1] print B -> [49 0 0 0 0 0 0] So it should just write ord(s[1]) to B. Sounds logical? For me very much. Further, one could write like this: B[:] = s print B-> [48 49 50 32 97 98 99] Namely cast the string into byte array. IMO this would be the logical expected behavior. Currently it just throws the value error if met non-digits in a string, so IMO current casting hardly can be of practical use. Furthermore, I think this code: A= array(s,"u1") Could act exactly same as: A= fromstring(s,"u1") But this is just a side-idea for spelling simplicty/generality. Not really necessary. Further thoughts: If trying to create "u1" array from a Pyhton 3 string, question is, whether it should throw an error, I think yes, and in this case "u4" type should be explicitly specified by initialisation, I suppose. And e.g. translation from unicode to extended ascii (Latin1) or whatever should be done on Python side or with explicit translation. Python3 assumes 4-byte strings but in reality most of the time we deal with 1-byte strings, so there is huge waste of resources when dealing with 4-bytes. For many serious projects it is just not needed. Furthermore I think some of the methods from "chararray" submodule should be possible to use directly on normal integer arrays without conversions to other array types. So I personally don't realy get why the need of additional chararray type, Its all numbers anyway and it's up to the programmer to decide what size of translation tables/value ranges he wants to use. There can be some convinience methods for ascii operations, like eg char.toupper(), but currently they don't seem to work with integer arrays so why not make those potentially useful methots usable and make them work on normal integer arrays? Or even migrate them to the root namespace to e.g. introduce names with prefixes: A=ascii_toupper(A) A=ascii_tolower(A) Many things can be be achieved with general numeric methods, e.g. translate/reduce the array. Here obviosly I mean not dynamical arrays, just fixed-sized arrays. How to deal with dynamically changing array sizes is another problematic, and it depends on how the software is designed in the first place and what it does with the data. For my own text-editing software project I consider fixed allocated 1D and 2D "uint8" arrays only. And specifically I experiment with own encodings, so just as a side-note, I don't think that encoding should be assumed much for creating new array types, it is up to the programmer to decide what 'meanings' the bytes have. Kind regards, Mikhail From tjol at tjol.eu Sun Jun 4 17:59:50 2017 From: tjol at tjol.eu (Thomas Jollans) Date: Sun, 4 Jun 2017 23:59:50 +0200 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On 04/06/17 20:04, Mikhail V wrote: > Initialize array from a string currently looks like: > > s= "012 abc" > A= fromstring(s,"u1") > print A -> > [48 49 50 32 97 98 99] > > Perfect. > Now when writing values it will not work > as IMO it should, namley consider this example: > > B= zeros(7,"u1") > B[0]=s[1] > print B -> > [1 0 0 0 0 0 0] > > Ugh? It tries to parse the s[1] character "1" as integer and writes 1 to B[0]. > First thing I would expect is a value error and I'd never expect it does > that high-level manipulations with parsing. > IMO ideally it would do the following instead: > > B[0]=s[1] > print B -> > [49 0 0 0 0 0 0] > > So it should just write ord(s[1]) to B. > Sounds logical? For me very much. > Further, one could write like this: > > B[:] = s > print B-> > [48 49 50 32 97 98 99] > > Namely cast the string into byte array. IMO this would be > the logical expected behavior. I disagree. If numpy treated bytestrings as sequences of uint8s (which would, granted, be perfectly reasonable, at least in py3), you wouldn't have needed the fromstring function in the first place. Personally, I think I would prefer this, actually. However, numpy normally treats strings as objects that can sometimes be cast to numbers, so this behaviour is perfectly logical. For what it's worth, in Python 3 (which you probably should want to be using), everything behaves as you'd expect: >>> import numpy as np >>> s = b'012 abc' >>> a = np.fromstring(s, 'u1') >>> a array([48, 49, 50, 32, 97, 98, 99], dtype=uint8) >>> b = np.zeros(7, 'u1') >>> b[0] = s[1] >>> b array([49, 0, 0, 0, 0, 0, 0], dtype=uint8) >>> > Currently it just throws the value error if met non-digits in a string, > so IMO current casting hardly can be of practical use. > > Furthermore, I think this code: > > A= array(s,"u1") > > Could act exactly same as: > > A= fromstring(s,"u1") > > But this is just a side-idea for spelling simplicty/generality. > Not really necessary. There is also something to be said for the current behaviour: >>> np.array('100', 'u1') array(100, dtype=uint8) However, the fact that this works for bytestrings on Python 3 is, in my humble opinion, ridiculous: >>> np.array(b'100', 'u1') # b'100' IS NOT TEXT array(100, dtype=uint8) This is of course consistent with the fact that you can cast a bytestring to builtin python int or float (but not complex). Interestingly enough, numpy complex behaves differently from python complex: >>> complex(b'1') Traceback (most recent call last): File "", line 1, in TypeError: complex() argument must be a string or a number, not 'bytes' >>> complex('1') (1+0j) >>> np.complex128('1') Traceback (most recent call last): File "", line 1, in TypeError: a float is required >>> > Further thoughts: > If trying to create "u1" array from a Pyhton 3 string, question is, > whether it should throw an error, I think yes, and in this case > "u4" type should be explicitly specified by initialisation, I suppose. > And e.g. translation from unicode to extended ascii (Latin1) or whatever > should be done on Python side or with explicit translation. If you ask me, passing a unicode string to fromstring with sep='' (i.e. to parse binary data) should ALWAYS raise an error: the semantics only make sense for strings of bytes. Currently, there appears to be some UTF-8 conversion going on, which creates potentially unexpected results: >>> s = '????' >>> a = np.fromstring(s, 'u1') >>> a array([206, 177, 206, 178, 206, 179, 206, 180], dtype=uint8) >>> assert len(a) * a.dtype.itemsize == len(s) Traceback (most recent call last): File "", line 1, in AssertionError >>> This is, apparently (https://github.com/numpy/numpy/issues/2152), due to how the internals of Python deal with unicode strings in C code, and not due to anything numpy is doing. Speaking of unexpected results, I'm not sure you realize what fromstring does when you give it a multi-byte dtype: >>> s = '????' >>> a = np.fromstring(s, 'u4') >>> a array([2999890382, 3033445326], dtype=uint32) >>> Give fromstring() a numpy unicode string, and all is right with the world: >>> s = np.array('????') >>> s array('????', dtype='>> np.fromstring(s, 'u4') array([945, 946, 947, 948], dtype=uint32) >>> IMHO calling fromstring(..., sep='') with a unicode string should be deprecated and perhaps eventually forbidden. (Or fixed, but that would break backwards compatibility) > Python3 assumes 4-byte strings but in reality most of the time > we deal with 1-byte strings, so there is huge waste of resources > when dealing with 4-bytes. For many serious projects it is just not needed. That's quite enough anglo-centrism, thank you. For when you need byte strings, Python 3 has a type for that. For when your strings contain text, bytes with no information on encoding are not enough. > Furthermore I think some of the methods from "chararray" submodule > should be possible to use directly on normal integer arrays without > conversions to other array types. > So I personally don't realy get why the need of additional chararray type, > Its all numbers anyway and it's up to the programmer to > decide what size of translation tables/value ranges he wants to use. chararray is deprecated. > There can be some convinience methods for ascii operations, > like eg char.toupper(), but currently they don't seem to work with integer > arrays so why not make those potentially useful methots usable > and make them work on normal integer arrays? I don't know what you're doing, but I don't think numpy is normally the right tool for text manipulation... > [snip] > > as a side-note, I don't think that encoding should be assumed much for > creating new array types, it is up to the programmer > to decide what 'meanings' the bytes have. Agreed! -- Thomas From chris.barker at noaa.gov Mon Jun 5 13:40:33 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 5 Jun 2017 10:40:33 -0700 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: Just a few notes: However, the fact that this works for bytestrings on Python 3 is, in my > humble opinion, ridiculous: > > >>> np.array(b'100', 'u1') # b'100' IS NOT TEXT > array(100, dtype=uint8) > Yes, that is a mis-feature -- I think due to bytes and string being the same object in py2 -- so on py3, numpy continues to treat a bytes objects as also a 1-byte-per-char string, depending on context. And users want to be able to write numpy code that will run the same on py2 and py3, so we kinda need this kind of thing. Makes me think that an optional "pure-py-3" mode for numpy might be a good idea. If that flag is set, your code will only run on py3 (or at least might run differently). > > Further thoughts: > > If trying to create "u1" array from a Pyhton 3 string, question is, > > whether it should throw an error, I think yes, well, you can pass numbers > 255 into a u1 already: In [*96*]: np.array(456, dtype='u1') Out[*96*]: array(200, dtype=uint8) and it does the wrap-around overflow thing... so why not? > and in this case > > "u4" type should be explicitly specified by initialisation, I suppose. > > And e.g. translation from unicode to extended ascii (Latin1) or whatever > > should be done on Python side or with explicit translation. > absolutely! If you ask me, passing a unicode string to fromstring with sep='' (i.e. > to parse binary data) should ALWAYS raise an error: the semantics only > make sense for strings of bytes. > exactly -- we really should have a "frombytes()" alias for fromstring() and it should only work for atual bytes objects (strings on py2, naturally). and overloading fromstring() to mean both "binary dump of data" and "parse the text" due to whether the sep argument is set was always a bad idea :-( .. and fromstring(s, sep=a_sep_char) has been semi broken (or at least not robust) forever anyway. Currently, there appears to be some UTF-8 conversion going on, which > creates potentially unexpected results: > > >>> s = '????' > >>> a = np.fromstring(s, 'u1') > >>> a > array([206, 177, 206, 178, 206, 179, 206, 180], dtype=uint8) > >>> assert len(a) * a.dtype.itemsize == len(s) > Traceback (most recent call last): > File "", line 1, in > AssertionError > >>> > > This is, apparently (https://github.com/numpy/numpy/issues/2152), due to > how the internals of Python deal with unicode strings in C code, and not > due to anything numpy is doing. > exactly -- py3 strings are pretty nifty implementation of unicode text -- they have nothing to do with storing binary data, and should not be used that way. There is essentially no reason you would ever want to pass the actual binary representation to any other code. fromstring should be re-named frombytes, and it should raise an exception if you pass something other than a bytes object (or maybe a memoryview or other binary container?) we might want to keep fromstring() for parsing strings, but only if it were fixed... IMHO calling fromstring(..., sep='') with a unicode string should be > deprecated and perhaps eventually forbidden. (Or fixed, but that would > break backwards compatibility) agreed. > Python3 assumes 4-byte strings but in reality most of the time > > we deal with 1-byte strings, so there is huge waste of resources > > when dealing with 4-bytes. For many serious projects it is just not > needed. > > That's quite enough anglo-centrism, thank you. For when you need byte > strings, Python 3 has a type for that. For when your strings contain > text, bytes with no information on encoding are not enough. > There was a big thread about this recently -- it seems to have not quite come to a conclusion. But anglo-centrism aside, there is substantial demand for a "smaller" way to store mostly-ascii text. I _think_ the conversation was steering toward an encoding-specified string dtype, so us anglo-centric folks could use latin-1 or utf-8. But someone would need to write the code. -CHB > There can be some convenience methods for ascii operations, > > like eg char.toupper(), but currently they don't seem to work with > integer > > arrays so why not make those potentially useful methots usable > > and make them work on normal integer arrays? > I don't know what you're doing, but I don't think numpy is normally the > right tool for text manipulation... > I agree here. But if one were to add such a thing (vectorized string operations) -- I'd think the thing to do would be to wrap (or port) the python string methods. But it shoudl only work for actual string dtypes, of course. note that another part of the discussion previously suggested that we have a dtype that wraps a native python string object -- then you'd get all for free. This is essentially an object array with strings in it, which you can do now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjol at tjol.eu Mon Jun 5 16:51:52 2017 From: tjol at tjol.eu (Thomas Jollans) Date: Mon, 5 Jun 2017 22:51:52 +0200 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On 05/06/17 19:40, Chris Barker wrote: > > If you ask me, passing a unicode string to fromstring with sep='' > (i.e. > to parse binary data) should ALWAYS raise an error: the semantics only > make sense for strings of bytes. > > > exactly -- we really should have a "frombytes()" alias for > fromstring() and it should only work for atual bytes objects (strings > on py2, naturally). > > and overloading fromstring() to mean both "binary dump of data" and > "parse the text" due to whether the sep argument is set was always a > bad idea :-( > > .. and fromstring(s, sep=a_sep_char) As it happens, this is pretty much what stdlib bytearray does since 3.2 (http://bugs.python.org/issue8990) > > has been semi broken (or at least not robust) forever anyway. > > Currently, there appears to be some UTF-8 conversion going on, which > creates potentially unexpected results: > > >>> s = '????' > >>> a = np.fromstring(s, 'u1') > >>> a > array([206, 177, 206, 178, 206, 179, 206, 180], dtype=uint8) > >>> assert len(a) * a.dtype.itemsize == len(s) > Traceback (most recent call last): > File "", line 1, in > AssertionError > >>> > > This is, apparently (https://github.com/numpy/numpy/issues/2152 > ), due to > how the internals of Python deal with unicode strings in C code, > and not > due to anything numpy is doing. > > > exactly -- py3 strings are pretty nifty implementation of unicode text > -- they have nothing to do with storing binary data, and should not be > used that way. There is essentially no reason you would ever want to > pass the actual binary representation to any other code. > > fromstring should be re-named frombytes, and it should raise an > exception if you pass something other than a bytes object (or maybe a > memoryview or other binary container?) > > we might want to keep fromstring() for parsing strings, but only if it > were fixed... > > IMHO calling fromstring(..., sep='') with a unicode string should be > deprecated and perhaps eventually forbidden. (Or fixed, but that would > break backwards compatibility) > > > agreed. > > > Python3 assumes 4-byte strings but in reality most of the time > > we deal with 1-byte strings, so there is huge waste of resources > > when dealing with 4-bytes. For many serious projects it is just > not needed. > > That's quite enough anglo-centrism, thank you. For when you need byte > strings, Python 3 has a type for that. For when your strings contain > text, bytes with no information on encoding are not enough. > > > There was a big thread about this recently -- it seems to have not > quite come to a conclusion. But anglo-centrism aside, there is > substantial demand for a "smaller" way to store mostly-ascii text. > > I _think_ the conversation was steering toward an encoding-specified > string dtype, so us anglo-centric folks could use latin-1 or utf-8. > > But someone would need to write the code. > > -CHB > > > There can be some convenience methods for ascii operations, > > like eg char.toupper(), but currently they don't seem to work > with integer > > arrays so why not make those potentially useful methots usable > > and make them work on normal integer arrays? > I don't know what you're doing, but I don't think numpy is > normally the > right tool for text manipulation... > > > I agree here. But if one were to add such a thing (vectorized string > operations) -- I'd think the thing to do would be to wrap (or port) > the python string methods. But it shoudl only work for actual string > dtypes, of course. > > note that another part of the discussion previously suggested that we > have a dtype that wraps a native python string object -- then you'd > get all for free. This is essentially an object array with strings in > it, which you can do now. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -- Thomas Jollans m ? +31 6 42630259 e ? tjol at tjol.eu From chris.barker at noaa.gov Mon Jun 5 18:51:11 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 5 Jun 2017 15:51:11 -0700 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On Mon, Jun 5, 2017 at 1:51 PM, Thomas Jollans wrote: > > and overloading fromstring() to mean both "binary dump of data" and > > "parse the text" due to whether the sep argument is set was always a > > bad idea :-( > > > > .. and fromstring(s, sep=a_sep_char) > > As it happens, this is pretty much what stdlib bytearray does since 3.2 > (http://bugs.python.org/issue8990) I'm not sure that the array.array.fromstring() ever parsed the data string as text, did it? Anyway, This is what array.array now has: array.frombytes(s) Appends items from the string, interpreting the string as an array of machine values (as if it had been read from a file using the fromfile()method). New in version 3.2: fromstring() is renamed to frombytes() for clarity. array.fromfile(f, n) Read n items (as machine values) from the file object f and append them to the end of the array. If less than n items are available, EOFError is raised, but the items that were available are still inserted into the array. f must be a real built-in file object; something else with a read() method won?t do. array.fromstring() Deprecated alias for frombytes(). I think numpy should do the same.And frombytes() should remove the "sep" parameter. If someone wants to write a fast efficient simple text parser, then it should get a new name: fromtext() maybe???And the fromfile() sep argument should be deprecated as well, for the same reasons.array also has: array.fromunicode(s) Extends this array with data from the given unicode string. The array must be a type 'u' array; otherwise a ValueError is raised. Usearray.frombytes(unicodestring.encode(enc)) to append Unicode data to an array of some other type. which I think would be better supported by:np.frombytes(str.encode('UCS-4'), dtype=uint32) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Jun 5 18:59:27 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 6 Jun 2017 00:59:27 +0200 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On 4 June 2017 at 23:59, Thomas Jollans wrote: > > > For what it's worth, in Python 3 (which you probably should want to be > using), everything behaves as you'd expect: > >>>> import numpy as np >>>> s = b'012 abc' >>>> a = np.fromstring(s, 'u1') >>>> a > array([48, 49, 50, 32, 97, 98, 99], dtype=uint8) >>>> b = np.zeros(7, 'u1') >>>> b[0] = s[1] >>>> b > array([49, 0, 0, 0, 0, 0, 0], dtype=uint8) >>>> Ok, examples do best. I think we have to separate cases though. So I will do examples in recent Python 3 now to avoid confusion. Case divisions: -- classify by "forward/backward" conversion: For this time consider only forward, i.e. I copy data from string to numpy array -- classify by " bytes vs ordinals ": a) bytes: If I need raw bytes - in this case e.g. B = bytes(s.encode()) will do it. then I can copy data to array. So currently there are methods coverings this. If I understand correctly the data extracted corresponds to utf-?? byte feed, i.e. non-constant byte-length of chars (1 up to 4 bytes per char for the 'wide' unicode, correct me if I am wrong). b): I need *ordinals* Yes, I need ordinals, so for the bytes() method, if a Python 3 string contains only basic ascii, I can so or so convert to bytes then to integer array and the length will be the same 1byte for each char. Although syntactically seen, and with slicing, this will look e.g. like: s= "012 abc" B = bytes(s.encode()) # convert to bytes k = len(s) arr = np.zeros(k,"u1") # init empty array length k arr[0:2] = list(B[0:2]) print ("my array: ", arr) -> my array: [48 49 0 0 0 0 0] Result seems correct. Note that I also need to use list(B), otherwise the slicing does not work (fills both values with 1, no idea where 1 comes from). Or I can write e.g.: arr[0:2] = np.fromstring(B[0:2], "u1") But looks indeed like a 'hack' and not so sinple. Considering your other examples there is other (better?) way, see below. Note, I personally don't know best practices and many technical nuances here so I repeat it from your words. -- classify "what is maximal ordinal value in the string" Well, say, I don't know what is maximal ordinal, e.g. here I take 3 Cyrillic letters instead of 'abc': s= "012 ???" k = len(s) arr = np.zeros(k,"u4") # init empty 32 bit array length k arr[:] = np.fromstring(np.array(s),"u4") -> [ 48 49 50 32 1040 1041 1042] This gives correct results indeed. So I get my ordinals as expected. So this is better/preferred way, right? Ok... Just some further thoughts on the topic: I would want to do the above things, in simpler syntax. For example, if there would be methods taking Python strings: arr = np.ordinals(s) arr[0:2] = np.ordinals(s[0:2]) # with slicing or, e.g. in such format: arr = np.copystr(s) arr[0:2] = np.copystr(s[0:2]) Which would give me same result as your proposed : arr = np.fromstring(np.array(s),"u4") arr[0:2] = np.fromstring(np.array(s[0:2]),"u4") IOW omitting "u4" parameter seems to be OK. E.g. if on the left side of assignment is "u1" array the values would be silently wrapped(?) according to Numpy rules (as Chris pointed out). And in similar way backward conversion to Python string. Though for Python 2 could raise questions why need casting to "u4". Would be cool just to use = without any methods as I've originally supposed, but as I understand now this behaviour is already occupied and would cause backward compatibility issues if touched. So approximately are my ideas. For me it would cover many applicaton cases. Mikhail From mikhailwas at gmail.com Mon Jun 5 19:06:34 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 6 Jun 2017 01:06:34 +0200 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On 5 June 2017 at 19:40, Chris Barker wrote: > > >> > Python3 assumes 4-byte strings but in reality most of the time >> > we deal with 1-byte strings, so there is huge waste of resources >> > when dealing with 4-bytes. For many serious projects it is just not >> > needed. >> >> That's quite enough anglo-centrism, thank you. For when you need byte >> strings, Python 3 has a type for that. For when your strings contain >> text, bytes with no information on encoding are not enough. > > There was a big thread about this recently -- it seems to have not quite > come to a conclusion. I have started to read that thread, though I've lost in idea transitions. Likely it was about some new string array type... > But anglo-centrism aside, there is substantial demand > for a "smaller" way to store mostly-ascii text. > Obviously there is demand. Terror of unicode touches many aspects of programmers life. It is not Numpy's problem though. The realistic scenario for satisfaction for this demand is a hard and wide problem. Foremost, it comes down to the question of defining this "optimal 8-bit character table". And "Latin-1", (exactly as it is) is not that optimal table, at least because of huge amount of accented letters. But, granted, if define most accented letters as "optional", i.e . delete them then it is quite reasonable basic char table to start with. Further comes the question of popularizisng new table (which doesn't even exists yet). >> > There can be some convenience methods for ascii operations, >> > like eg char.toupper(), but currently they don't seem to work with >> > integer >> > arrays so why not make those potentially useful methots usable >> > and make them work on normal integer arrays? >> I don't know what you're doing, but I don't think numpy is normally the >> right tool for text manipulation... > > > I agree here. But if one were to add such a thing (vectorized string > operations) -- I'd think the thing to do would be to wrap (or port) the > python string methods. But it shoudl only work for actual string dtypes, of > course. > > note that another part of the discussion previously suggested that we have a > dtype that wraps a native python string object -- then you'd get all for > free. This is essentially an object array with strings in it, which you can > do now. > Well here I must admit I don't quite understand the whole idea of "numpy array of string type". How used? What is main bebefit/feature...? Example integer array usage in context of textual data in my case: - holding data in a text editor (mutability+indexing/slicing) - filtering, transformations (e.g. table translations, cryptography, etc.) String type array? Will this be a string array you describe: s= "012 abc" arr = np.array(s) print ("type ", arr.dtype) print ("shape ", arr.shape) print ("my array: ", arr) arr = np.roll(arr[0],2) print ("my array: ", arr) -> type References: Message-ID: On Mon, Jun 5, 2017 at 3:59 PM, Mikhail V wrote: > -- classify by "forward/backward" conversion: > For this time consider only forward, i.e. I copy data from string > to numpy array > > -- classify by " bytes vs ordinals ": > > a) bytes: If I need raw bytes - in this case e.g. > > B = bytes(s.encode()) > no need to call "bytes" -- encode() returns a bytes object: In [1]: s = "this is a simple ascii-only string" In [2]: b = s.encode() In [3]: type(b) Out[3]: bytes In [4]: b Out[4]: b'this is a simple ascii-only string' > > will do it. then I can copy data to array. So currently there are methods > coverings this. If I understand correctly the data extracted corresponds > to utf-?? byte feed, i.e. non-constant byte-length of chars (1 up to > 4 bytes per char for > the 'wide' unicode, correct me if I am wrong). > In [5]: s.encode? Docstring: S.encode(encoding='utf-8', errors='strict') -> bytes So the default is utf-8, but you can set any encoding you want (that python supports) In [6]: s.encode('utf-16') Out[6]: b'\xff\xfet\x00h\x00i\x00s\x00 \x00i\x00s\x00 \x00a\x00 \x00s\x00i\x00m\x00p\x00l\x00e\x00 \x00a\x00s\x00c\x00i\x00i\x00-\x00o\x00n\x00l\x00y\x00 \x00s\x00t\x00r\x00i\x00n\x00g\x00' > b): I need *ordinals* > Yes, I need ordinals, so for the bytes() method, if a Python 3 > string contains only > basic ascii, I can so or so convert to bytes then to integer array > and the length will > be the same 1byte for each char. > Although syntactically seen, and with slicing, this will look e.g. like: > > s= "012 abc" > B = bytes(s.encode()) # convert to bytes > k = len(s) > arr = np.zeros(k,"u1") # init empty array length k > arr[0:2] = list(B[0:2]) > print ("my array: ", arr) > -> > my array: [48 49 0 0 0 0 0] > This can be done more cleanly: In [15]: s= "012 abc" In [16]: b = s.encode('ascii') # you want to use the ascii encoding so you don't get utf-8 cruft if there are non-ascii characters # you could use latin-1 too (Or any other one-byte per char encoding In [17]: arr = np.fromstring(b, np.uint8) # this is using fromstring() to means it's old py definiton - treat teh contenst as bytes # -- it really should be called "frombytes()" # you could also use: In [22]: np.frombuffer(b, dtype=np.uint8) Out[22]: array([48, 49, 50, 32, 97, 98, 99], dtype=uint8)In [18]: print arr In [19]: print(arr) [48 49 50 32 97 98 99] # you got the ordinals In [20]: "".join([chr(i) for i in arr]) Out[20]: '012 abc' # yes, they are the right ones... > Result seems correct. Note that I also need to use list(B), otherwise > the slicing does not work (fills both values with 1, no idea where 1 > comes from). > that is odd -- I can't explain it right now either... > Or I can write e.g.: > arr[0:2] = np.fromstring(B[0:2], "u1") > > But looks indeed like a 'hack' and not so simple. > is the above OK? > -- classify "what is maximal ordinal value in the string" > Well, say, I don't know what is maximal ordinal, e.g. here I take > 3 Cyrillic letters instead of 'abc': > > s= "012 ???" > k = len(s) > arr = np.zeros(k,"u4") # init empty 32 bit array length k > arr[:] = np.fromstring(np.array(s),"u4") > -> > [ 48 49 50 32 1040 1041 1042] > so this is making a numpy string, which is a UCS-4 encoding unicode -- i.e. 4 bytes per charactor. Then you care converting that to an 4-byte unsigned int. but no need to do it with fromstring: In [52]: s Out[52]: '012 ???' In [53]: s_arr.reshape((1,)).view(np.uint32) Out[53]: array([ 48, 49, 50, 32, 1040, 1041, 1042], dtype=uint32) we need the reshape() because .view does not work with array scalars -- not sure why not? > This gives correct results indeed. So I get my ordinals as expected. > So this is better/preferred way, right? > I would maybe do it more "directly" -- i.e. use python's string to do the encoding: In [64]: s Out[64]: '012 ???' In [67]: np.fromstring(s.encode('U32'), dtype=np.uint32) Out[67]: array([65279, 48, 49, 50, 32, 1040, 1041, 1042], dtype=uint32) that first value is the byte-order mark (I think...), you can strip it off with: In [68]: np.fromstring(s.encode('U32')[4:], dtype=np.uint32) Out[68]: array([ 48, 49, 50, 32, 1040, 1041, 1042], dtype=uint32) or, probably better simply specify the byte order in the encoding: In [69]: np.fromstring(s.encode('UTF-32LE'), dtype=np.uint32) Out[69]: array([ 48, 49, 50, 32, 1040, 1041, 1042], dtype=uint32) arr = np.ordinals(s) > arr[0:2] = np.ordinals(s[0:2]) # with slicing > > or, e.g. in such format: > > arr = np.copystr(s) > arr[0:2] = np.copystr(s[0:2]) > I don't think any of this is necessary -- the UCS4 (Or UTF-32) "encoding" is pretty much the ordinals anyway. As you notices, if you make a numpy unicode string array, and change the dtype to unsigned int32, you get what you want. You really don't want to mess with any of this unless you understand unicode and encodings anyway.... Though it is a bit akward -- why is your actual use-case for working with ordinals??? BTW, you can use regular python to get the ordinals first: In [71]: np.array([ord(c) for c in s]) Out[71]: array([ 48, 49, 50, 32, 1040, 1041, 1042]) Though for Python 2 could raise questions why need casting to "u4". > this would all work the same with python 2 if you used unicode objects instead of strings. Maybe good to put: from __future__ import unicode_literals in your source.... > So approximately are my ideas. > For me it would cover many application cases. I'm still curious as to your use-cases -- when do you have a bunch of ordinal values?? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Jun 6 18:29:00 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 6 Jun 2017 15:29:00 -0700 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On Mon, Jun 5, 2017 at 4:06 PM, Mikhail V wrote: > Likely it was about some new string array type... yes, it was. > Obviously there is demand. Terror of unicode touches many aspects > of programmers life. I don't know that I'd call it Terror, but frankly, the fact that you need up to 4 bytes for a single character is really not the big issues. Given that computer memory has grown by literally orders of magnitude since Unicode was introduced, I don't know why there is such a hang up about it. But we're scientific programmers we like to be efficient ! > Foremost, it comes down to the question of defining this "optimal > 8-bit character table". > And "Latin-1", (exactly as it is) is not that optimal table, there is no such thing as a single "optimal" set of characters when you are limited to 255 of them... latin-1 is pretty darn good for the, well, latin-based languages.... > But, granted, if define most accented letters as > "optional", i.e . delete them > then it is quite reasonable basic char table to start with. > Then you are down to ASCII, no? but anyway, I don't think a new encoding is really the topic at hand here.... >> I don't know what you're doing, but I don't think numpy is normally the > >> right tool for text manipulation... > > > > > > I agree here. But if one were to add such a thing (vectorized string > > operations) -- I'd think the thing to do would be to wrap (or port) the > > python string methods. But it shoudl only work for actual string dtypes, > of > > course. > > > > note that another part of the discussion previously suggested that we > have a > > dtype that wraps a native python string object -- then you'd get all for > > free. This is essentially an object array with strings in it, which you > can > > do now. > > > > Well here I must admit I don't quite understand the whole idea of > "numpy array of string type". How used? What is main bebefit/feature...? > here you go -- you can do this now: In [74]: s_arr = np.array([s, "another string"], dtype=np.object) In [75]: In [75]: s_arr Out[75]: array(['012 ???', 'another string'], dtype=object) In [76]: s_arr.shape Out[76]: (2,) You now have an array with python string object in it -- thus access to all the string functionality: In [81]: s_arr[1] = s_arr[1].upper() In [82]: s_arr Out[82]: array(['012 ???', 'ANOTHER STRING'], dtype=object) and the ability to have each string be a different length. If numpy were to know that those were string objects, rather than arbitrary python objects, it could do vectorized operations on them, etc. You can do that now with numpy.vectorize, but it's pretty klunky. In [87]: np_upper = np.vectorize(str.upper) In [88]: np_upper(s_arr) Out[88]: array(['012 ???', 'ANOTHER STRING'], dtype=' Example integer array usage in context of textual data in my case: > - holding data in a text editor (mutability+indexing/slicing) > you really want to use regular old python data structures for that... > - filtering, transformations (e.g. table translations, cryptography, etc.) > that may be something to do with ordinals and numpy -- but then you need to work with ascii or latin-1 and uint8 dtypes, or full Unicode and uint32 dtype -- that's that. String type array? Will this be a string array you describe: > > s= "012 abc" > arr = np.array(s) > print ("type ", arr.dtype) > print ("shape ", arr.shape) > print ("my array: ", arr) > arr = np.roll(arr[0],2) > print ("my array: ", arr) > -> > type shape () > my array: 012 abc > my array: 012 abc > > > So what it does? What's up with shape? > shape is an empty tuple, meaning this is a numpy scalar, containing a single string type ' e.g. here I wanted to 'roll' the string. > How would I replace chars? or delete? > What is the general idea behind? > the numpy string type (unicode type) works with fixed length strings -- not characters, but you can reshape it and make a view: In [89]: s= "012 abc" In [90]: arr.shape = (1,) In [91]: arr.shape Out[91]: (1,) In [93]: c_arr = arr.view(dtype = ' From mikhailwas at gmail.com Tue Jun 6 23:47:17 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 7 Jun 2017 05:47:17 +0200 Subject: [Numpy-discussion] Array and string interoperability In-Reply-To: References: Message-ID: On 7 June 2017 at 00:05, Chris Barker wrote: > On Mon, Jun 5, 2017 at 3:59 PM, Mikhail V wrote: >> s= "012 abc" >> B = bytes(s.encode()) # convert to bytes >> k = len(s) >> arr = np.zeros(k,"u1") # init empty array length k >> arr[0:2] = list(B[0:2]) >> print ("my array: ", arr) >> -> >> my array: [48 49 0 0 0 0 0] > > > This can be done more cleanly: > > In [15]: s= "012 abc" > > In [16]: b = s.encode('ascii') > > # you want to use the ascii encoding so you don't get utf-8 cruft if there > are non-ascii characters > # you could use latin-1 too (Or any other one-byte per char encoding Thanks for clarifying, that makes sense. Also it's a good way to validate the string. > > or, probably better simply specify the byte order in the encoding: > > In [69]: np.fromstring(s.encode('UTF-32LE'), dtype=np.uint32) > Out[69]: array([ 48, 49, 50, 32, 1040, 1041, 1042], dtype=uint32) Ok, this gives what I want too. So now for unicode I am by two possible options (apart from possible "fromstring" spelling): with indexing (if I want to copy into already existing array on the fly): arr[0:3] = np.fromstring(np.array(s[0:3]),"u4") arr[0:3] = np.fromstring(s[0:3].encode('UTF-32LE'),"u4") > >> arr = np.ordinals(s) >> arr[0:2] = np.ordinals(s[0:2]) # with slicing > > > I don't think any of this is necessary -- the UCS4 (Or UTF-32) "encoding" is > pretty much the ordinals anyway. > > As you notices, if you make a numpy unicode string array, and change the > dtype to unsigned int32, you get what you want. No I am not implying anything is necessary, just seems to be sort of a pattern. And from Python 3 perspective where string indexing is by wide characters ... well I don't know. >> Example integer array usage in context of textual data in my case: >> - holding data in a text editor (mutability+indexing/slicing) >> >you really want to use regular old python data structures for that... >[...] >the numpy string type (unicode type) works with fixed length strings -- not >characters, but you can reshape it and make a view: >[...] I am intentionally choosing fixed size array for holding data and writing values using indexes. But wait a moment, characters *are* integers, identities, [put some other name here]. > In [93]: c_arr = arr.view(dtype = ' In [97]: np.roll(c_arr, 3) > Out[97]: > array(['a', 'b', 'c', '0', '1', '2', ' '], > dtype='> Foremost, it comes down to the question of defining this "optimal >> 8-bit character table". >> And "Latin-1", (exactly as it is) is not that optimal table, > >there is no such thing as a single "optimal" set of characters when you are >limited to 255 of them... Yeah, depends much on criteria of 'optimality' and many other things ;) >> But, granted, if define most accented letters as >> "optional", i.e . delete them >> then it is quite reasonable basic char table to start with. > >Then you are down to ASCII, no? No, then I am down to ASCII plus few vital characters, e.g.: - Dashes (which could solve the painful and old as world problem of "hyphen" vs "minus") - Multiplication sign, degree - Em dash, quotation marks, spaces (non-breaking, half) -- all vital for typesetting ... If you think about it, 255 units is more than enough to define perfect communication standards. >but anyway, I don't think a new encoding is really the topic at hand >here.... Yes I think this is off-opic on this list. But intersting indeed, where it is on-topic. Seems like those encodings are coming from some "mysterios castle in the clouds". Mikhail From marc.barbry at mailoo.org Wed Jun 7 04:52:08 2017 From: marc.barbry at mailoo.org (marc) Date: Wed, 7 Jun 2017 10:52:08 +0200 Subject: [Numpy-discussion] Variance without temporary array Message-ID: <48dc2a92-3e00-8b95-beb4-4930e8d2a704@mailoo.org> Dear Numpy developers, I'm trying to write a version of the variance function that does not use temporary array. I got something that start to be functional (the branch is located here https://github.com/mbarbry/numpy/tree/sum_square_shift) but when running the tests I get the following memory error, Running unit tests for numpy NumPy version 1.14.0.dev0+f9db20b NumPy relaxed strides checking option: True NumPy is installed in /home/marc/programs/python/my_numpy/local/lib/python2.7/site-packages/numpy-1.14.0.dev0+f9db20b-py2.7-linux-x86_64.egg/numpy Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609] nose version 1.3.7 Program received signal SIGSEGV, Segmentation fault. 0x00007ffff1ef1f6d in inittest_rational () at numpy/core/src/umath/test_rational.c.src:1262 1262 REGISTER_UFUNC_BINARY_RATIONAL(add) This is weird because I didn't touch the umath module, I'm working in the multiarray one. Do you have a idea how I can debug this? Thanks for your help, Marc Barbry From charlesr.harris at gmail.com Wed Jun 7 15:25:59 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 7 Jun 2017 13:25:59 -0600 Subject: [Numpy-discussion] NumPy 1.13.0 release Message-ID: Hi All, On behalf of the NumPy team, I amn pleased to announce the NumPy 1.13.0 release. This release supports Python 2.7 and 3.4-3.6 and contains many new features. It is one of the most ambitious releases in the last several years. Some of the highlights and new functions are *Highlights* - Operations like ``a + b + c`` will reuse temporaries on some platforms, resulting in less memory use and faster execution. - Inplace operations check if inputs overlap outputs and create temporaries to avoid problems. - New __array_ufunc__ attribute provides improved ability for classes to override default ufunc behavior. - New np.block function for creating blocked arrays. *New functions* - New ``np.positive`` ufunc. - New ``np.divmod`` ufunc provides more efficient divmod. - New ``np.isnat`` ufunc tests for NaT special values. - New ``np.heaviside`` ufunc computes the Heaviside function. - New ``np.isin`` function, improves on ``in1d``. - New ``np.block`` function for creating blocked arrays. - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API. Wheels for the pre-release are available on PyPI. Source tarballs, zipfiles, release notes, and the changelog are available on github . A total of 102 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - A. Jesse Jiryu Davis + - Alessandro Pietro Bardelli + - Alex Rothberg + - Alexander Shadchin - Allan Haldane - Andres Guzman-Ballen + - Antoine Pitrou - Antony Lee - B R S Recht + - Baurzhan Muftakhidinov + - Ben Rowland - Benda Xu + - Blake Griffith - Bradley Wogsland + - Brandon Carter + - CJ Carey - Charles Harris - Christoph Gohlke - Danny Hermes + - David Hagen + - David Nicholson + - Duke Vijitbenjaronk + - Egor Klenin + - Elliott Forney + - Elliott M Forney + - Endolith - Eric Wieser - Erik M. Bray - Eugene + - Evan Limanto + - Felix Berkenkamp + - Fran?ois Bissey + - Frederic Bastien - Greg Young - Gregory R. Lee - Importance of Being Ernest + - Jaime Fernandez - Jakub Wilk + - James Cowgill + - James Sanders - Jean Utke + - Jesse Thoren + - Jim Crist + - Joerg Behrmann + - John Kirkham - Jonathan Helmus - Jonathan L Long - Jonathan Tammo Siebert + - Joseph Fox-Rabinovitz - Joshua Loyal + - Juan Nunez-Iglesias + - Julian Taylor - Kirill Balunov + - Likhith Chitneni + - Lo?c Est?ve - Mads Ohm Larsen - Marein K?nings + - Marten van Kerkwijk - Martin Thoma - Martino Sorbaro + - Marvin Schmidt + - Matthew Brett - Matthias Bussonnier + - Matthias C. M. Troffaes + - Matti Picus - Michael Seifert - Mikhail Pak + - Mortada Mehyar - Nathaniel J. Smith - Nick Papior - Oscar Villellas + - Pauli Virtanen - Pavel Potocek - Pete Peeradej Tanruangporn + - Philipp A + - Ralf Gommers - Robert Kern - Roland Kaufmann + - Ronan Lamy - Sami Salonen + - Sanchez Gonzalez Alvaro - Sebastian Berg - Shota Kawabuchi - Simon Gibbons - Stefan Otte - Stefan Peterson + - Stephan Hoyer - S?ren Fuglede J?rgensen + - Takuya Akiba - Tom Boyd + - Ville Skytt? + - Warren Weckesser - Wendell Smith - Yu Feng - Zixu Zhao + - Z? Vin?cius + - aha66 + - drabach + - drlvk + - jsh9 + - solarjoe + - zengi + Cheers, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jun 7 16:42:30 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 7 Jun 2017 14:42:30 -0600 Subject: [Numpy-discussion] Add changelogs to repo Message-ID: Hi All, I've made a PR adding the NumPy `1.13.0-changelog.rst` to a new `doc/changelog` directory. The idea is to keep the lengthy changelogs separate from the release notes, but still easily accessible and permanently part to the NumPy history. If you hit the `rich diff` tag at the PR, you can see the rendered rst version. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.debuyl at kuleuven.be Mon Jun 12 15:30:43 2017 From: pierre.debuyl at kuleuven.be (Pierre de Buyl) Date: Mon, 12 Jun 2017 21:30:43 +0200 Subject: [Numpy-discussion] EuroSciPy 2017 call for contributions Message-ID: <20170612193043.GT1316@pi-x230> (Apologies if you receive multiple copies of this message) 10th European Conference on Python in Science August 28 - September 1, 2017 in Erlangen, Germany Description: The EuroSciPy meeting is a cross-disciplinary gathering focused on the use and development of the Python language in scientific research. This event strives to bring together both users and developers of scientific tools, as well as academic research and state of the art industry. Erlangen is one of Germany's major science hubs and located north of Munich (90 minutes by train). Topics of Interest Presentations of scientific tools and libraries using the Python language, including but not limited to: - Vector and array manipulation - Parallel computing - Scientific visualization - Scientific data flow and persistence - Algorithms implemented or exposed in Python - Web applications and portals for science and engineering- - Reports on the use of Python in scientific achievements or ongoing projects. - General-purpose Python tools that can be of special interest to the scientific community. Submission Details: You can send your proposal by submitting your application here: https://www.papercall.io/euroscipy-2017 The Call for Papers closes on June 25, 2017 00:00 CEST Regards, The EuroSciPy team https://www.euroscipy.org/2017/ From jaime.frio at gmail.com Thu Jun 15 03:08:12 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 15 Jun 2017 09:08:12 +0200 Subject: [Numpy-discussion] np.diff on boolean arrays now raises Message-ID: There is an ongoing discussion on github: https://github.com/numpy/numpy/issues/9251 In 1.13 np.diff has started raising on boolean arrays, since subtraction of boolean arrays is now deprecated. A decision has to be made whether: - raising an error is the correct thing to do, and only the docstring needs updating, or - backwards compatibility is more important and diff should still work on boolean arrays. If you have opinions on the subject, or enjoy arguing for the sake of it, please share your thoughts either here or on github. Thanks! -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Jun 15 03:38:08 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 15 Jun 2017 09:38:08 +0200 Subject: [Numpy-discussion] Taking NumPy in Stride workshop at PyData Barcelona Message-ID: I ended up going to the PyData event in Barcelona last month and running a workshop there. The notebook I used can be found here: https://github.com/jaimefrio/pydatabcn2017/tree/master/taking_numpy_in_stride I was afraid that the content would be too technical, but all the feedback I got from attendees was that they very much enjoyed it and want more of this deep technical dives into the belly of the beast. Of course IRL, contrary to the internet, it is easier to approach someone to tell them you like what they presented, than to tell them they suck. So maybe this is just a biased illusion, but until proven wrong I am encouraging anyone with the knowledge and the opportunity to go ahead and organize a similar workshop in any conference they go to. Jaime P.S. I also got a chance to catch up with Francesc Alted and Travis Oliphant, which is always great! -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Thu Jun 15 03:43:29 2017 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 15 Jun 2017 09:43:29 +0200 Subject: [Numpy-discussion] Taking NumPy in Stride workshop at PyData Barcelona In-Reply-To: References: Message-ID: <20170615074329.GA750213@phare.normalesup.org> +1 In my experience, there is a lack of deep understanding of numpy works, and people are happy to learn. Ga?l On Thu, Jun 15, 2017 at 09:38:08AM +0200, Jaime Fern?ndez del R?o wrote: > I ended up going to the PyData event in Barcelona last month and running a > workshop there. The notebook I used can be found here: > https://github.com/jaimefrio/pydatabcn2017/tree/master/taking_numpy_in_stride > I was afraid that the content would be too technical, but all the feedback I > got from attendees was that they very much enjoyed it and want more of this > deep technical dives into the belly of the beast. > Of course IRL, contrary to the internet, it is easier to approach someone to > tell them you like what they presented, than to tell them they suck. So maybe > this is just a biased illusion, but until proven wrong I am encouraging anyone > with the knowledge and the opportunity to go ahead and organize a similar > workshop in any conference they go to. > Jaime > P.S. I also got a chance to catch up with Francesc Alted and Travis Oliphant, > which is always great! -- Gael Varoquaux Researcher, INRIA Parietal NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From kikocorreoso at gmail.com Thu Jun 15 03:57:47 2017 From: kikocorreoso at gmail.com (Kiko) Date: Thu, 15 Jun 2017 09:57:47 +0200 Subject: [Numpy-discussion] Taking NumPy in Stride workshop at PyData Barcelona In-Reply-To: <20170615074329.GA750213@phare.normalesup.org> References: <20170615074329.GA750213@phare.normalesup.org> Message-ID: 2017-06-15 9:43 GMT+02:00 Gael Varoquaux : > +1 > > In my experience, there is a lack of deep understanding of numpy works, > and people are happy to learn. > > Ga?l > > On Thu, Jun 15, 2017 at 09:38:08AM +0200, Jaime Fern?ndez del R?o wrote: > > I ended up going to the PyData event in Barcelona last month and running > a > > workshop there. The notebook I used can be found here: > > > https://github.com/jaimefrio/pydatabcn2017/tree/master/ > taking_numpy_in_stride > > > I was afraid that the content would be too technical, but all the > feedback I > > got from attendees was that they very much enjoyed it and want more of > this > > deep technical dives into the belly of the beast. > > > Of course IRL, contrary to the internet, it is easier to approach > someone to > > tell them you like what they presented, than to tell them they suck. So > maybe > > this is just a biased illusion, but until proven wrong I am encouraging > anyone > > with the knowledge and the opportunity to go ahead and organize a similar > > workshop in any conference they go to. > Jaime, as you know I was there and I really enjoyed the workshop (and the talk). I've also received positive comments from others there. Maybe it is not and ad-hoc workshop for a PyData these days where all the buzzwords are Spark, TensorFlow, DeepWhatever,..., but, again, I think it was great and it was one of my reasons to go to PyData Barcelona. Thanks for the workshop. > > > Jaime > > > P.S. I also got a chance to catch up with Francesc Alted and Travis > Oliphant, > > which is always great! > -- > Gael Varoquaux > Researcher, INRIA Parietal > NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France > Phone: ++ 33-1-69-08-79-68 > http://gael-varoquaux.info http://twitter.com/GaelVaroquaux > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Thu Jun 15 06:10:36 2017 From: faltet at gmail.com (Francesc Alted) Date: Thu, 15 Jun 2017 12:10:36 +0200 Subject: [Numpy-discussion] Taking NumPy in Stride workshop at PyData Barcelona In-Reply-To: References: <20170615074329.GA750213@phare.normalesup.org> Message-ID: 2017-06-15 9:57 GMT+02:00 Kiko : > > > 2017-06-15 9:43 GMT+02:00 Gael Varoquaux : > >> +1 >> >> In my experience, there is a lack of deep understanding of numpy works, >> and people are happy to learn. >> >> Ga?l >> >> On Thu, Jun 15, 2017 at 09:38:08AM +0200, Jaime Fern?ndez del R?o wrote: >> > I ended up going to the PyData event in Barcelona last month and >> running a >> > workshop there. The notebook I used can be found here: >> >> > https://github.com/jaimefrio/pydatabcn2017/tree/master/takin >> g_numpy_in_stride >> >> > I was afraid that the content would be too technical, but all the >> feedback I >> > got from attendees was that they very much enjoyed it and want more of >> this >> > deep technical dives into the belly of the beast. >> > ?When the content is deeply technical and the teacher can really convey its more hidden details, that says a lot about the teacher :)? We routinely have very good feedback from a similar tutorial run by St?fan van der Walt at our ASPP annual event: https://github.com/ASPP/2016_numpy ?but again, this is just a proof of another top-notch teacher in action.? > >> > Of course IRL, contrary to the internet, it is easier to approach >> someone to >> > tell them you like what they presented, than to tell them they suck. So >> maybe >> > this is just a biased illusion, but until proven wrong I am encouraging >> anyone >> > with the knowledge and the opportunity to go ahead and organize a >> similar >> > workshop in any conference they go to. >> > > Jaime, as you know I was there and I really enjoyed the workshop (and the > talk). I've also received positive comments from others there. > > Maybe it is not and ad-hoc workshop for a PyData these days where all the > buzzwords are Spark, TensorFlow, DeepWhatever,..., but, again, I think it > was great and it was one of my reasons to go to PyData Barcelona. > ?Exactly, and IMO it is precisely this balance between teaching fundamentals and new technologies what makes a conference to ?stand out and fullfill different expectations. And I must say that PyData Barcelona was pretty good at that. ?Francesc? > > Thanks for the workshop. > > >> >> > Jaime >> >> > P.S. I also got a chance to catch up with Francesc Alted and Travis >> Oliphant, >> > which is always great! >> -- >> Gael Varoquaux >> Researcher, INRIA Parietal >> NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France >> Phone: ++ 33-1-69-08-79-68 >> http://gael-varoquaux.info http://twitter.com/GaelVaroqua >> ux >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jun 15 06:35:36 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 15 Jun 2017 22:35:36 +1200 Subject: [Numpy-discussion] np.diff on boolean arrays now raises In-Reply-To: References: Message-ID: On Thu, Jun 15, 2017 at 7:08 PM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > There is an ongoing discussion on github: > > https://github.com/numpy/numpy/issues/9251 > > In 1.13 np.diff has started raising on boolean arrays, since subtraction > of boolean arrays is now deprecated. > > A decision has to be made whether: > > - raising an error is the correct thing to do, and only the docstring > needs updating, or > - backwards compatibility is more important and diff should still work > on boolean arrays. > > The issue is bigger than np.diff. For example, there's a problem with the scipy.ndimage morphology functions ( https://github.com/scipy/scipy/issues/7493) that looks pretty serious. All ndimage.binary_* functions (7 of them) for example return boolean arrays, and chaining those is now broken. Unfortunately it seems that that wasn't covered by the ndimage test suite. Possibly reverting the breaking change in 1.13.1 is the best way to fix this. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Jun 15 07:15:20 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 15 Jun 2017 13:15:20 +0200 Subject: [Numpy-discussion] np.diff on boolean arrays now raises In-Reply-To: References: Message-ID: <1497525320.17526.7.camel@sipsolutions.net> On Thu, 2017-06-15 at 22:35 +1200, Ralf Gommers wrote: > > > On Thu, Jun 15, 2017 at 7:08 PM, Jaime Fern?ndez del R?o gmail.com> wrote: > > There is an ongoing discussion on github: > > > > https://github.com/numpy/numpy/issues/9251 > > > > In 1.13?np.diff?has started raising on boolean arrays, since > > subtraction of ?boolean?arrays is now deprecated. > > > > A decision has to be made whether: > > raising an error is the correct thing to do, and only the docstring > > needs updating, or > > backwards compatibility is more important and diff should still > > work on boolean arrays. > > > > The issue is bigger than np.diff. For example, there's a problem with > the scipy.ndimage morphology functions (https://github.com/scipy/scip > y/issues/7493) that looks pretty serious. All ndimage.binary_* > functions (7 of them) for example return boolean arrays, and chaining > those is now broken. Unfortunately it seems that that wasn't covered > by the ndimage test suite. > > Possibly reverting the breaking change in 1.13.1 is the best way to > fix this. > Sure, I would say there is nothing wrong with reverting for now (and it simply is the safe and easy way). Though it would be good to address the issue of what should happen in the future with diff (and possibly the subtract deprecation itself). If we stick to it, but its necessary, we could delay the deprecation and make it a VisibleDeprecationWarning. - Sebastian > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Thu Jun 15 08:16:16 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Jun 2017 06:16:16 -0600 Subject: [Numpy-discussion] np.diff on boolean arrays now raises In-Reply-To: References: Message-ID: On Thu, Jun 15, 2017 at 4:35 AM, Ralf Gommers wrote: > > > On Thu, Jun 15, 2017 at 7:08 PM, Jaime Fern?ndez del R?o < > jaime.frio at gmail.com> wrote: > >> There is an ongoing discussion on github: >> >> https://github.com/numpy/numpy/issues/9251 >> >> In 1.13 np.diff has started raising on boolean arrays, since subtraction >> of boolean arrays is now deprecated. >> >> A decision has to be made whether: >> >> - raising an error is the correct thing to do, and only the docstring >> needs updating, or >> - backwards compatibility is more important and diff should still >> work on boolean arrays. >> >> > The issue is bigger than np.diff. For example, there's a problem with the > scipy.ndimage morphology functions (https://github.com/scipy/ > scipy/issues/7493) that looks pretty serious. All ndimage.binary_* > functions (7 of them) for example return boolean arrays, and chaining those > is now broken. Unfortunately it seems that that wasn't covered by the > ndimage test suite. > > Possibly reverting the breaking change in 1.13.1 is the best way to fix > this. > There were two related deprecations of boolean operators: subtraction and negative. I haven't heard any complaints about the second, which I think is less controversial, but I think reversion is the way to go for subtraction, at least for 1.13.1. The main virtue of the deprecations is clarity, xor is more informative when applied to boolean values than `-`, although the latter is certainly correct in F_2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jun 15 08:49:18 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Jun 2017 06:49:18 -0600 Subject: [Numpy-discussion] np.diff on boolean arrays now raises In-Reply-To: References: Message-ID: On Thu, Jun 15, 2017 at 6:16 AM, Charles R Harris wrote: > > > On Thu, Jun 15, 2017 at 4:35 AM, Ralf Gommers > wrote: > >> >> >> On Thu, Jun 15, 2017 at 7:08 PM, Jaime Fern?ndez del R?o < >> jaime.frio at gmail.com> wrote: >> >>> There is an ongoing discussion on github: >>> >>> https://github.com/numpy/numpy/issues/9251 >>> >>> In 1.13 np.diff has started raising on boolean arrays, since >>> subtraction of boolean arrays is now deprecated. >>> >>> A decision has to be made whether: >>> >>> - raising an error is the correct thing to do, and only the >>> docstring needs updating, or >>> - backwards compatibility is more important and diff should still >>> work on boolean arrays. >>> >>> >> The issue is bigger than np.diff. For example, there's a problem with the >> scipy.ndimage morphology functions (https://github.com/scipy/scip >> y/issues/7493) that looks pretty serious. All ndimage.binary_* functions >> (7 of them) for example return boolean arrays, and chaining those is now >> broken. Unfortunately it seems that that wasn't covered by the ndimage test >> suite. >> >> Possibly reverting the breaking change in 1.13.1 is the best way to fix >> this. >> > > There were two related deprecations of boolean operators: subtraction and > negative. I haven't heard any complaints about the second, which I think is > less controversial, but I think reversion is the way to go for subtraction, > at least for 1.13.1. The main virtue of the deprecations is clarity, xor is > more informative when applied to boolean values than `-`, although the > latter is certainly correct in F_2. > > See reversion at https://github.com/numpy/numpy/pull/9255. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon Jun 19 08:27:49 2017 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 19 Jun 2017 14:27:49 +0200 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" Message-ID: I will assume some simple linear systems knowledge but the question can be generalized to any operator that implements __mul__ and __rmul__ methods. Motivation: I am trying to implement a gain matrix, say 3x3 identity matrix, for time being with a single input single output (SISO) system that I have implemented as a class modeling a Transfer or a state space representation. In the typical usecase, suppose you would like to create an n-many parallel connections with the same LTI system sitting at each branch. MATLAB implements this as an elementwise multiplication and returning a multi input multi output(MIMO) system. G = tf(1,[1,1]); eye(3)*G produces (manually compactified) ans = From input 1 to output... [ 1 ] [ ------ , 0 , 0 ] [ s + 1 ] [ 1 ] [ 0 , ------ , 0 ] [ s + 1 ] [ 1 ] [ 0 , 0 , ------ ] [ s + 1 ] Notice that the result type is of LTI system but, in our context, not a NumPy array with "object" dtype. In order to achieve a similar behavior, I would like to let the __rmul__ of G take care of the multiplication. In fact, when I do G.__rmul__(np.eye(3)) I can control what the behavior should be and I receive the exception/result I've put in. However the array never looks for this method and carries out the default array __mul__ behavior. The situation is similar if we go about it as left multiplication G*eye(3) has no problems since this uses directly the __mul__ of G. Therefore we get a different result depending on the direction of multiplication. Is there anything I can do about this without forcing users subclassing or just letting them know about this particular quirk in the documentation? What I have in mind is to force the users to create static LTI objects and then multiply and reject this possibility. But then I still need to stop NumPy returning "object" dtyped array to be able to let the user know about this. Relevant links just in case the library : https://github.com/ilayn/harold/ the issue discussion (monologue actually) : https://github.com/ilayn/harold/issues/7 The question I've asked on SO (but with a rather offtopic answer): https://stackoverflow.com/q/40694380/4950339 ilhan -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Mon Jun 19 11:36:53 2017 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 19 Jun 2017 08:36:53 -0700 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" In-Reply-To: References: Message-ID: I answered your question on StackOverflow: https://stackoverflow.com/questions/40694380/forcing-multiplication-to-use-rmul-instead-of-numpy-array-mul-or-byp/44634634#44634634 In brief, you need to set __array_priority__ or __array_ufunc__ on your object. On Mon, Jun 19, 2017 at 5:27 AM, Ilhan Polat wrote: > I will assume some simple linear systems knowledge but the question can be > generalized to any operator that implements __mul__ and __rmul__ methods. > > Motivation: > > I am trying to implement a gain matrix, say 3x3 identity matrix, for time > being with a single input single output (SISO) system that I have > implemented as a class modeling a Transfer or a state space representation. > > In the typical usecase, suppose you would like to create an n-many > parallel connections with the same LTI system sitting at each branch. > MATLAB implements this as an elementwise multiplication and returning a > multi input multi output(MIMO) system. > > G = tf(1,[1,1]); > eye(3)*G > > produces (manually compactified) > > ans = > > From input 1 to output... > [ 1 ] > [ ------ , 0 , 0 ] > [ s + 1 ] > [ 1 ] > [ 0 , ------ , 0 ] > [ s + 1 ] > [ 1 ] > [ 0 , 0 , ------ ] > [ s + 1 ] > > Notice that the result type is of LTI system but, in our context, not a > NumPy array with "object" dtype. > > In order to achieve a similar behavior, I would like to let the __rmul__ > of G take care of the multiplication. In fact, when I do > G.__rmul__(np.eye(3)) I can control what the behavior should be and I > receive the exception/result I've put in. However the array never looks for > this method and carries out the default array __mul__ behavior. > > The situation is similar if we go about it as left multiplication G*eye(3) > has no problems since this uses directly the __mul__ of G. Therefore we get > a different result depending on the direction of multiplication. > > Is there anything I can do about this without forcing users subclassing or > just letting them know about this particular quirk in the documentation? > > What I have in mind is to force the users to create static LTI objects and > then multiply and reject this possibility. But then I still need to stop > NumPy returning "object" dtyped array to be able to let the user know about > this. > > > Relevant links just in case > > the library : https://github.com/ilayn/harold/ > > the issue discussion (monologue actually) : https://github.com/ilayn/ > harold/issues/7 > > The question I've asked on SO (but with a rather offtopic answer): > https://stackoverflow.com/q/40694380/4950339 > > > ilhan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon Jun 19 12:07:46 2017 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 19 Jun 2017 18:07:46 +0200 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" In-Reply-To: References: Message-ID: Thank you. I didn't know that it existed. Is there any place where I can get a feeling for a sane priority number compared to what's being done in production? Just to make sure I'm not stepping on any toes. On Mon, Jun 19, 2017 at 5:36 PM, Stephan Hoyer wrote: > I answered your question on StackOverflow: > https://stackoverflow.com/questions/40694380/forcing- > multiplication-to-use-rmul-instead-of-numpy-array-mul-or- > byp/44634634#44634634 > > In brief, you need to set __array_priority__ or __array_ufunc__ on your > object. > > On Mon, Jun 19, 2017 at 5:27 AM, Ilhan Polat wrote: > >> I will assume some simple linear systems knowledge but the question can >> be generalized to any operator that implements __mul__ and __rmul__ >> methods. >> >> Motivation: >> >> I am trying to implement a gain matrix, say 3x3 identity matrix, for time >> being with a single input single output (SISO) system that I have >> implemented as a class modeling a Transfer or a state space representation. >> >> In the typical usecase, suppose you would like to create an n-many >> parallel connections with the same LTI system sitting at each branch. >> MATLAB implements this as an elementwise multiplication and returning a >> multi input multi output(MIMO) system. >> >> G = tf(1,[1,1]); >> eye(3)*G >> >> produces (manually compactified) >> >> ans = >> >> From input 1 to output... >> [ 1 ] >> [ ------ , 0 , 0 ] >> [ s + 1 ] >> [ 1 ] >> [ 0 , ------ , 0 ] >> [ s + 1 ] >> [ 1 ] >> [ 0 , 0 , ------ ] >> [ s + 1 ] >> >> Notice that the result type is of LTI system but, in our context, not a >> NumPy array with "object" dtype. >> >> In order to achieve a similar behavior, I would like to let the __rmul__ >> of G take care of the multiplication. In fact, when I do >> G.__rmul__(np.eye(3)) I can control what the behavior should be and I >> receive the exception/result I've put in. However the array never looks for >> this method and carries out the default array __mul__ behavior. >> >> The situation is similar if we go about it as left multiplication >> G*eye(3) has no problems since this uses directly the __mul__ of G. >> Therefore we get a different result depending on the direction of >> multiplication. >> >> Is there anything I can do about this without forcing users subclassing >> or just letting them know about this particular quirk in the documentation? >> >> What I have in mind is to force the users to create static LTI objects >> and then multiply and reject this possibility. But then I still need to >> stop NumPy returning "object" dtyped array to be able to let the user know >> about this. >> >> >> Relevant links just in case >> >> the library : https://github.com/ilayn/harold/ >> >> the issue discussion (monologue actually) : >> https://github.com/ilayn/harold/issues/7 >> >> The question I've asked on SO (but with a rather offtopic answer): >> https://stackoverflow.com/q/40694380/4950339 >> >> >> ilhan >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Mon Jun 19 12:14:07 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Mon, 19 Jun 2017 11:14:07 -0500 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" In-Reply-To: References: Message-ID: I don't think there's any real standard here. Just doing a github search reveals many different choices people have used: https://github.com/search?l=Python&q=__array_priority__&type=Code&utf8=%E2%9C%93 On Mon, Jun 19, 2017 at 11:07 AM, Ilhan Polat wrote: > Thank you. I didn't know that it existed. Is there any place where I can > get a feeling for a sane priority number compared to what's being done in > production? Just to make sure I'm not stepping on any toes. > > On Mon, Jun 19, 2017 at 5:36 PM, Stephan Hoyer wrote: > >> I answered your question on StackOverflow: >> https://stackoverflow.com/questions/40694380/forcing-multipl >> ication-to-use-rmul-instead-of-numpy-array-mul-or-byp/44634634#44634634 >> >> In brief, you need to set __array_priority__ or __array_ufunc__ on your >> object. >> >> On Mon, Jun 19, 2017 at 5:27 AM, Ilhan Polat >> wrote: >> >>> I will assume some simple linear systems knowledge but the question can >>> be generalized to any operator that implements __mul__ and __rmul__ >>> methods. >>> >>> Motivation: >>> >>> I am trying to implement a gain matrix, say 3x3 identity matrix, for >>> time being with a single input single output (SISO) system that I have >>> implemented as a class modeling a Transfer or a state space representation. >>> >>> In the typical usecase, suppose you would like to create an n-many >>> parallel connections with the same LTI system sitting at each branch. >>> MATLAB implements this as an elementwise multiplication and returning a >>> multi input multi output(MIMO) system. >>> >>> G = tf(1,[1,1]); >>> eye(3)*G >>> >>> produces (manually compactified) >>> >>> ans = >>> >>> From input 1 to output... >>> [ 1 ] >>> [ ------ , 0 , 0 ] >>> [ s + 1 ] >>> [ 1 ] >>> [ 0 , ------ , 0 ] >>> [ s + 1 ] >>> [ 1 ] >>> [ 0 , 0 , ------ ] >>> [ s + 1 ] >>> >>> Notice that the result type is of LTI system but, in our context, not a >>> NumPy array with "object" dtype. >>> >>> In order to achieve a similar behavior, I would like to let the __rmul__ >>> of G take care of the multiplication. In fact, when I do >>> G.__rmul__(np.eye(3)) I can control what the behavior should be and I >>> receive the exception/result I've put in. However the array never looks for >>> this method and carries out the default array __mul__ behavior. >>> >>> The situation is similar if we go about it as left multiplication >>> G*eye(3) has no problems since this uses directly the __mul__ of G. >>> Therefore we get a different result depending on the direction of >>> multiplication. >>> >>> Is there anything I can do about this without forcing users subclassing >>> or just letting them know about this particular quirk in the documentation? >>> >>> What I have in mind is to force the users to create static LTI objects >>> and then multiply and reject this possibility. But then I still need to >>> stop NumPy returning "object" dtyped array to be able to let the user know >>> about this. >>> >>> >>> Relevant links just in case >>> >>> the library : https://github.com/ilayn/harold/ >>> >>> the issue discussion (monologue actually) : >>> https://github.com/ilayn/harold/issues/7 >>> >>> The question I've asked on SO (but with a rather offtopic answer): >>> https://stackoverflow.com/q/40694380/4950339 >>> >>> >>> ilhan >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Mon Jun 19 12:58:32 2017 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 19 Jun 2017 09:58:32 -0700 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" In-Reply-To: References: Message-ID: Coming up with a single number for a sane "array priority" is basically an impossible task :). If you only need compatibility with the latest version of NumPy, this is one good reason to set __array_ufunc__ instead, even if only to write __array_ufunc__ = None. On Mon, Jun 19, 2017 at 9:14 AM, Nathan Goldbaum wrote: > I don't think there's any real standard here. Just doing a github search > reveals many different choices people have used: > > https://github.com/search?l=Python&q=__array_priority__& > type=Code&utf8=%E2%9C%93 > > On Mon, Jun 19, 2017 at 11:07 AM, Ilhan Polat > wrote: > >> Thank you. I didn't know that it existed. Is there any place where I can >> get a feeling for a sane priority number compared to what's being done in >> production? Just to make sure I'm not stepping on any toes. >> >> On Mon, Jun 19, 2017 at 5:36 PM, Stephan Hoyer wrote: >> >>> I answered your question on StackOverflow: >>> https://stackoverflow.com/questions/40694380/forcing-multipl >>> ication-to-use-rmul-instead-of-numpy-array-mul-or-byp/44634634#44634634 >>> >>> In brief, you need to set __array_priority__ or __array_ufunc__ on your >>> object. >>> >>> On Mon, Jun 19, 2017 at 5:27 AM, Ilhan Polat >>> wrote: >>> >>>> I will assume some simple linear systems knowledge but the question can >>>> be generalized to any operator that implements __mul__ and __rmul__ >>>> methods. >>>> >>>> Motivation: >>>> >>>> I am trying to implement a gain matrix, say 3x3 identity matrix, for >>>> time being with a single input single output (SISO) system that I have >>>> implemented as a class modeling a Transfer or a state space representation. >>>> >>>> In the typical usecase, suppose you would like to create an n-many >>>> parallel connections with the same LTI system sitting at each branch. >>>> MATLAB implements this as an elementwise multiplication and returning a >>>> multi input multi output(MIMO) system. >>>> >>>> G = tf(1,[1,1]); >>>> eye(3)*G >>>> >>>> produces (manually compactified) >>>> >>>> ans = >>>> >>>> From input 1 to output... >>>> [ 1 ] >>>> [ ------ , 0 , 0 ] >>>> [ s + 1 ] >>>> [ 1 ] >>>> [ 0 , ------ , 0 ] >>>> [ s + 1 ] >>>> [ 1 ] >>>> [ 0 , 0 , ------ ] >>>> [ s + 1 ] >>>> >>>> Notice that the result type is of LTI system but, in our context, not a >>>> NumPy array with "object" dtype. >>>> >>>> In order to achieve a similar behavior, I would like to let the >>>> __rmul__ of G take care of the multiplication. In fact, when I do >>>> G.__rmul__(np.eye(3)) I can control what the behavior should be and I >>>> receive the exception/result I've put in. However the array never looks for >>>> this method and carries out the default array __mul__ behavior. >>>> >>>> The situation is similar if we go about it as left multiplication >>>> G*eye(3) has no problems since this uses directly the __mul__ of G. >>>> Therefore we get a different result depending on the direction of >>>> multiplication. >>>> >>>> Is there anything I can do about this without forcing users subclassing >>>> or just letting them know about this particular quirk in the documentation? >>>> >>>> What I have in mind is to force the users to create static LTI objects >>>> and then multiply and reject this possibility. But then I still need to >>>> stop NumPy returning "object" dtyped array to be able to let the user know >>>> about this. >>>> >>>> >>>> Relevant links just in case >>>> >>>> the library : https://github.com/ilayn/harold/ >>>> >>>> the issue discussion (monologue actually) : >>>> https://github.com/ilayn/harold/issues/7 >>>> >>>> The question I've asked on SO (but with a rather offtopic answer): >>>> https://stackoverflow.com/q/40694380/4950339 >>>> >>>> >>>> ilhan >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon Jun 19 13:07:04 2017 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 19 Jun 2017 19:07:04 +0200 Subject: [Numpy-discussion] Controlling NumPy __mul__ method or forcing it to use __rmul__ of the "other" In-Reply-To: References: Message-ID: Ah OK. I was just wondering if there are recommended values to start with in case below some values are reserved for NumPy/SciPy internals. I'll just go with the ufunc path just in case. This really looks like TeX overful/underful badness value adjustment. As long as the journal accepts don't mention it. :) On Mon, Jun 19, 2017 at 6:58 PM, Stephan Hoyer wrote: > Coming up with a single number for a sane "array priority" is basically an > impossible task :). If you only need compatibility with the latest version > of NumPy, this is one good reason to set __array_ufunc__ instead, even if > only to write __array_ufunc__ = None. > > On Mon, Jun 19, 2017 at 9:14 AM, Nathan Goldbaum > wrote: > >> I don't think there's any real standard here. Just doing a github search >> reveals many different choices people have used: >> >> https://github.com/search?l=Python&q=__array_priority__&type >> =Code&utf8=%E2%9C%93 >> >> On Mon, Jun 19, 2017 at 11:07 AM, Ilhan Polat >> wrote: >> >>> Thank you. I didn't know that it existed. Is there any place where I can >>> get a feeling for a sane priority number compared to what's being done in >>> production? Just to make sure I'm not stepping on any toes. >>> >>> On Mon, Jun 19, 2017 at 5:36 PM, Stephan Hoyer wrote: >>> >>>> I answered your question on StackOverflow: >>>> https://stackoverflow.com/questions/40694380/forcing-multipl >>>> ication-to-use-rmul-instead-of-numpy-array-mul-or-byp/44634634#44634634 >>>> >>>> In brief, you need to set __array_priority__ or __array_ufunc__ on your >>>> object. >>>> >>>> On Mon, Jun 19, 2017 at 5:27 AM, Ilhan Polat >>>> wrote: >>>> >>>>> I will assume some simple linear systems knowledge but the question >>>>> can be generalized to any operator that implements __mul__ and __rmul__ >>>>> methods. >>>>> >>>>> Motivation: >>>>> >>>>> I am trying to implement a gain matrix, say 3x3 identity matrix, for >>>>> time being with a single input single output (SISO) system that I have >>>>> implemented as a class modeling a Transfer or a state space representation. >>>>> >>>>> In the typical usecase, suppose you would like to create an n-many >>>>> parallel connections with the same LTI system sitting at each branch. >>>>> MATLAB implements this as an elementwise multiplication and returning a >>>>> multi input multi output(MIMO) system. >>>>> >>>>> G = tf(1,[1,1]); >>>>> eye(3)*G >>>>> >>>>> produces (manually compactified) >>>>> >>>>> ans = >>>>> >>>>> From input 1 to output... >>>>> [ 1 ] >>>>> [ ------ , 0 , 0 ] >>>>> [ s + 1 ] >>>>> [ 1 ] >>>>> [ 0 , ------ , 0 ] >>>>> [ s + 1 ] >>>>> [ 1 ] >>>>> [ 0 , 0 , ------ ] >>>>> [ s + 1 ] >>>>> >>>>> Notice that the result type is of LTI system but, in our context, not >>>>> a NumPy array with "object" dtype. >>>>> >>>>> In order to achieve a similar behavior, I would like to let the >>>>> __rmul__ of G take care of the multiplication. In fact, when I do >>>>> G.__rmul__(np.eye(3)) I can control what the behavior should be and I >>>>> receive the exception/result I've put in. However the array never looks for >>>>> this method and carries out the default array __mul__ behavior. >>>>> >>>>> The situation is similar if we go about it as left multiplication >>>>> G*eye(3) has no problems since this uses directly the __mul__ of G. >>>>> Therefore we get a different result depending on the direction of >>>>> multiplication. >>>>> >>>>> Is there anything I can do about this without forcing users >>>>> subclassing or just letting them know about this particular quirk in the >>>>> documentation? >>>>> >>>>> What I have in mind is to force the users to create static LTI objects >>>>> and then multiply and reject this possibility. But then I still need to >>>>> stop NumPy returning "object" dtyped array to be able to let the user know >>>>> about this. >>>>> >>>>> >>>>> Relevant links just in case >>>>> >>>>> the library : https://github.com/ilayn/harold/ >>>>> >>>>> the issue discussion (monologue actually) : >>>>> https://github.com/ilayn/harold/issues/7 >>>>> >>>>> The question I've asked on SO (but with a rather offtopic answer): >>>>> https://stackoverflow.com/q/40694380/4950339 >>>>> >>>>> >>>>> ilhan >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at python.org >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michelemartone at users.sourceforge.net Wed Jun 21 05:51:55 2017 From: michelemartone at users.sourceforge.net (Michele Martone) Date: Wed, 21 Jun 2017 11:51:55 +0200 Subject: [Numpy-discussion] PyRSB: Python interface to librsb sparse matrices library Message-ID: <20170621095155.GE19876@localhost> Hi. I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs. With this email I am looking for prospective: - users/testers - developers (any interest to collaborate/adopt/include the project?) Looking forward for your feedback, Michele -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From michelemartone at users.sourceforge.net Wed Jun 21 05:51:55 2017 From: michelemartone at users.sourceforge.net (Michele Martone) Date: Wed, 21 Jun 2017 11:51:55 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library Message-ID: Hi. I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs. With this email I am looking for prospective: - users/testers - developers (any interest to collaborate/adopt/include the project?) Looking forward for your feedback, Michele -------------- next part -------------- A non-text attachment was scrubbed... Name: Unsupported File Types Alert.txt Type: application/octet-stream Size: 253 bytes Desc: Unsupported File Types Alert.txt URL: From ralf.gommers at gmail.com Fri Jun 23 11:27:12 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 24 Jun 2017 03:27:12 +1200 Subject: [Numpy-discussion] ANN: SciPy 0.19.1 release Message-ID: On behalf of the Scipy development team I am pleased to announce the availability of Scipy 0.19.1. This is a bugfix-only release, no new features are included. This release requires Python 2.7 or 3.4-3.6 and NumPy 1.8.2 or greater. Source tarballs and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.19.1. OS X and Linux wheels are available from PyPI: https://pypi.python.org/pypi/scipy/0.19.1 Thanks to everyone who contributed! Cheers, Ralf -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ========================== SciPy 0.19.1 Release Notes ========================== SciPy 0.19.1 is a bug-fix release with no new features compared to 0.19.0. The most important change is a fix for a severe memory leak in ``integrate.quad``. Authors ======= * Evgeni Burovski * Patrick Callier + * Yu Feng * Ralf Gommers * Ilhan Polat * Eric Quintero * Scott Sievert * Pauli Virtanen * Warren Weckesser A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.19.1 - ------------------------ - - `#7214 `__: Memory use in integrate.quad in scipy-0.19.0 - - `#7258 `__: ``linalg.matrix_balance`` gives wrong transformation matrix - - `#7262 `__: Segfault in daily testing - - `#7273 `__: ``scipy.interpolate._bspl.evaluate_spline`` gets wrong type - - `#7335 `__: scipy.signal.dlti(A,B,C,D).freqresp() fails Pull requests for 0.19.1 - ------------------------ - - `#7211 `__: BUG: convolve may yield inconsistent dtypes with method changed - - `#7216 `__: BUG: integrate: fix refcounting bug in quad() - - `#7229 `__: MAINT: special: Rewrite a test of wrightomega - - `#7261 `__: FIX: Corrected the transformation matrix permutation - - `#7265 `__: BUG: Fix broken axis handling in spectral functions - - `#7266 `__: FIX 7262: ckdtree crashes in query_knn. - - `#7279 `__: Upcast half- and single-precision floats to doubles in BSpline... - - `#7336 `__: BUG: Fix signal.dfreqresp for StateSpace systems - - `#7419 `__: Fix several issues in ``sparse.load_npz``, ``save_npz`` - - `#7420 `__: BUG: stats: allow integers as kappa4 shape parameters Checksums ========= MD5 ~~~ 72415e8da753eea97eb9820602931cb5 scipy-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl e0022540df2735eb0475071b266d5d71 scipy-0.19.1-cp27-cp27m-manylinux1_i686.whl f513eb4ea2086de169a502df7efb91c7 scipy-0.19.1-cp27-cp27m-manylinux1_x86_64.whl 906c3c59209d6249b5d8ce14cfa01382 scipy-0.19.1-cp27-cp27mu-manylinux1_i686.whl afbf8ffb4a4fe7c18e34cb8a313c18ee scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl 5ba945b3404644244ab469883a1723f0 scipy-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 9c02cdd79e4ffadddcce7b2212039816 scipy-0.19.1-cp34-cp34m-manylinux1_i686.whl 79c0ba3618466614744de9a2f5362bbc scipy-0.19.1-cp34-cp34m-manylinux1_x86_64.whl 602a741a54190e16698ff8b2fe9fd27c scipy-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d6c2ecadd4df36eb61870227fae42d3a scipy-0.19.1-cp35-cp35m-manylinux1_i686.whl e7167c0a9cf270f89437e2fd09731636 scipy-0.19.1-cp35-cp35m-manylinux1_x86_64.whl fc2e4679e83590ff19c1a5c5b1aa4786 scipy-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 458615e9a56429a72038531dd5dcb3cb scipy-0.19.1-cp36-cp36m-manylinux1_i686.whl 65b1667ac56861da4cbc609960ed735b scipy-0.19.1-cp36-cp36m-manylinux1_x86_64.whl b704ebe9a28b8fe83d9f238d40031266 scipy-0.19.1.tar.gz cad6bac0638b176f72c00fe81ed54d19 scipy-0.19.1.tar.xz eb69261e5026ef2f3b9ae827caa7e5b8 scipy-0.19.1.zip SHA256 ~~~~~~ 1e8fedf602859b541ebae78667ccfc53158edef58d9ee19ee659309004565952 scipy-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 023ee29faa76c184a607e21076f097dc32f3abba7c71ece374588f95920aa993 scipy-0.19.1-cp27-cp27m-manylinux1_i686.whl 2a26d06a642e3c9107ca06df125f5dc5507abe2b87fd7865415d03ab654b0b43 scipy-0.19.1-cp27-cp27m-manylinux1_x86_64.whl b3e97be2cd9f052d984fc5ba2d441897971b744c64d658617944c47bc366f8ff scipy-0.19.1-cp27-cp27mu-manylinux1_i686.whl 78713101b32af384c564837fd7ae665a2a72cb6d49edbd8f32148d74724a65a8 scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl 2c9e87d556b83a8e11de7a064856c3997bbaff7f9cf62bf63ff0623751549e4c scipy-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl b87a923ed390ba0aafcbbcc6f0e023d495f19d2bd22ae59bdef3b0aec6485b39 scipy-0.19.1-cp34-cp34m-manylinux1_i686.whl 2dc9bcfaa585d9d941fb1add0d160556fe8587c3800264a77643695565a2d279 scipy-0.19.1-cp34-cp34m-manylinux1_x86_64.whl 93825cc80c638d901099f657dfff852ad2421beb51cb7d1d3f91157741ebe287 scipy-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 4118647273c907ed984da52b71fa2bbb30229f1820fb79b1ed087c8bc9a20cca scipy-0.19.1-cp35-cp35m-manylinux1_i686.whl 4c26ef44e8bb2cd2aef11c60d163caa04670d6f42996789b209526677310ded2 scipy-0.19.1-cp35-cp35m-manylinux1_x86_64.whl 2a3b6ceadbb58d8b8d4a329f8219f9e6f17757ec6c85baf03987bbd2b728c263 scipy-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 909be30bdd1d8cd57609760ea72a1499543169ed6ea5c66fad50946582682cf8 scipy-0.19.1-cp36-cp36m-manylinux1_i686.whl 4de84e3870228e8afd55a6e63e762aa7c9d1e3bd9a9ef5ab716835a69c77d257 scipy-0.19.1-cp36-cp36m-manylinux1_x86_64.whl 9147f7b8b5ed4b80fefe82057a6d0d4bf3a50375a1d573407f84686d3b82c3bb scipy-0.19.1.tar.gz 0dca04c4860afdcb066cab4fd520fcffa8c85e9a7b5aa37a445308e899d728b3 scipy-0.19.1.tar.xz 2cc1baed85f7f702d958ad19e5d11c9dad917e981d3d0c299a7f0fd15027516a scipy-0.19.1.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJZTTG2AAoJEO2+o3i/Gl69V8gH/3S199hh1Fcl0a05VOCgt7aZ zRKvdZhYkqGvWoWX+YwJBaZn0aXThJfIDE2lAGt4S9PyxObBKzsujI8OLIZF4zU5 PtAARrooOqDEH1CC39k2gscF9GBhxAlxwntM2Hd8h+4xsklAhFVxn4UT0M8igBoX txpaIqghzBJnenkMq6lqrj7vhupuulz0zrMnJU4LetpdSxX1lb5sGvWmZDTKd/nr xmthQ4TmHPbf9jAYCtLI4V6OUGFGLQ+d7IqiYvU+DVKNZFEgyPMnvN9QvSdu6diR JBEIBcijxM0BqjPwRoavCQjCHk37kR/G3UsgcEyHO2tr5zuOXogjjBo3Bei8jnI= =lFXD -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Sat Jun 24 10:29:35 2017 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Sat, 24 Jun 2017 16:29:35 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: <20170621095155.GE19876@localhost> References: <20170621095155.GE19876@localhost> Message-ID: Hi Michele, This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference... Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Congratulations on the release! Sylvain On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone wrote: > Hi. > > I'm the author of the high performance multithreaded sparse matrix > library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ > > I'm *not* a user of SciPy/NumPy/Python, but using Cython I have > written a proof-of-concept interface to librsb, named `PyRSB': > https://github.com/michelemartone/pyrsb > > PyRSB is in a prototypal state; e.g. still lacks good error handling. > Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. > Advantages over csr_matrix are in fast multithreaded multiplication > of huge sparse matrices. > Intended application area is iterative solution of linear systems; > particularly fast if with symmetric matrices and many rhs. > > With this email I am looking for prospective: > - users/testers > - developers (any interest to collaborate/adopt/include the project?) > > Looking forward for your feedback, > Michele > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Jun 24 15:16:39 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 24 Jun 2017 12:16:39 -0700 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> Message-ID: On Jun 24, 2017 7:29 AM, "Sylvain Corlay" wrote: Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Jun 24 15:47:35 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 24 Jun 2017 15:47:35 -0400 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> Message-ID: On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith wrote: > On Jun 24, 2017 7:29 AM, "Sylvain Corlay" > wrote: > > > Also, one quick question: is the LGPL license a deliberate choice or is it > not important to you? Most projects in the Python scientific stack are BSD > licensed. So the LGPL choice makes it unlikely that a higher-level project > adopts it as a dependency. If you are the only copyright holder, you would > still have the possibility to license it under a more permissive license > such as BSD or MIT... > > > Why would LGPL be a problem in a dependency? That doesn't stop you making > your code BSD, and it's less restrictive license-wise than depending on MKL > or the windows C runtime... > Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid) Josef > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Jun 24 16:07:55 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 24 Jun 2017 22:07:55 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> Message-ID: <1498334875.2430.1.camel@sipsolutions.net> On Sat, 2017-06-24 at 15:47 -0400, josef.pktd at gmail.com wrote: > > > On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith > wrote: > > On Jun 24, 2017 7:29 AM, "Sylvain Corlay" > > wrote: > > > > Also, one quick question: is the LGPL license a deliberate choice > > or is it not important to you? Most projects in the Python > > scientific stack are BSD licensed. So the LGPL choice makes it > > unlikely that a higher-level project adopts it as a dependency. If > > you are the only copyright holder, you would still have the > > possibility to license it under a more permissive license such as > > BSD or MIT... > > > > Why would LGPL be a problem in a dependency? That doesn't stop you > > making your code BSD, and it's less restrictive license-wise than > > depending on MKL or the windows C runtime... > > > > Is scipy still including any LGPL code, I thought not. > There might still be some optional dependencies that not many users > are using by default. ? > Julia packages are mostly MIT, AFAIK. (except for the GPL parts > because of cholmod, which we (?) avoid) > Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc. Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license. - Sebastian > Josef > ? > > -n > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From cmkleffner at gmail.com Sat Jun 24 16:58:55 2017 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sat, 24 Jun 2017 22:58:55 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: <1498334875.2430.1.camel@sipsolutions.net> References: <20170621095155.GE19876@localhost> <1498334875.2430.1.camel@sipsolutions.net> Message-ID: Does this still apply: https://scipy.github.io/old-wiki/pages/License_Compatibility.html Carl 2017-06-24 22:07 GMT+02:00 Sebastian Berg : > On Sat, 2017-06-24 at 15:47 -0400, josef.pktd at gmail.com wrote: > > > > > > On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith > > wrote: > > > On Jun 24, 2017 7:29 AM, "Sylvain Corlay" > > > wrote: > > > > > > Also, one quick question: is the LGPL license a deliberate choice > > > or is it not important to you? Most projects in the Python > > > scientific stack are BSD licensed. So the LGPL choice makes it > > > unlikely that a higher-level project adopts it as a dependency. If > > > you are the only copyright holder, you would still have the > > > possibility to license it under a more permissive license such as > > > BSD or MIT... > > > > > > Why would LGPL be a problem in a dependency? That doesn't stop you > > > making your code BSD, and it's less restrictive license-wise than > > > depending on MKL or the windows C runtime... > > > > > > > Is scipy still including any LGPL code, I thought not. > > There might still be some optional dependencies that not many users > > are using by default. ? > > Julia packages are mostly MIT, AFAIK. (except for the GPL parts > > because of cholmod, which we (?) avoid) > > > > > Well, I don't think scipy has many dependencies (but I would not be > surprised if those are LGPL). Not a specialist, but as a dependency it > should be fine (that is the point of the L in LGPL after all as far as > I understand, it is much less viral). > If you package it with your own stuff, you have to make sure to point > out that parts are LGPL of course (just like there is a reason you get > the GPL printed out with some devices) and if you modify it provide > these modifications, etc. > > Of course you cannot include it into the scipy codebase itself, but > there is probably no aim of doing so here, so without a specific reason > I would think that LGPL is a great license. > > - Sebastian > > > > Josef > > > > > -n > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Jun 24 17:26:02 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 24 Jun 2017 23:26:02 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> <1498334875.2430.1.camel@sipsolutions.net> Message-ID: <1498339562.2430.3.camel@sipsolutions.net> On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote: > Does this still apply: https://scipy.github.io/old-wiki/pages/License > _Compatibility.html > Of course, but it talks about putting it into the code base of scipy not about being able to use the package in any way in a dependency (i.e. `import package`). - Sebastian > Carl > > 2017-06-24 22:07 GMT+02:00 Sebastian Berg >: > > On Sat, 2017-06-24 at 15:47 -0400, josef.pktd at gmail.com wrote: > > > > > > > > > On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith > > > wrote: > > > > On Jun 24, 2017 7:29 AM, "Sylvain Corlay" > .com > > > > > wrote: > > > > > > > > Also, one quick question: is the LGPL license a deliberate > > choice > > > > or is it not important to you? Most projects in the Python > > > > scientific stack are BSD licensed. So the LGPL choice makes it > > > > unlikely that a higher-level project adopts it as a dependency. > > If > > > > you are the only copyright holder, you would still have the > > > > possibility to license it under a more permissive license such > > as > > > > BSD or MIT... > > > > > > > > Why would LGPL be a problem in a dependency? That doesn't stop > > you > > > > making your code BSD, and it's less restrictive license-wise > > than > > > > depending on MKL or the windows C runtime... > > > > > > > > > > Is scipy still including any LGPL code, I thought not. > > > There might still be some optional dependencies that not many > > users > > > are using by default. ? > > > Julia packages are mostly MIT, AFAIK. (except for the GPL parts > > > because of cholmod, which we (?) avoid) > > > > > > > > > Well, I don't think scipy has many dependencies (but I would not be > > surprised if those are LGPL). Not a specialist, but as a dependency > > it > > should be fine (that is the point of the L in LGPL after all as far > > as > > I understand, it is much less viral). > > If you package it with your own stuff, you have to make sure to > > point > > out that parts are LGPL of course (just like there is a reason you > > get > > the GPL printed out with some devices) and if you modify it provide > > these modifications, etc. > > > > Of course you cannot include it into the scipy codebase itself, but > > there is probably no aim of doing so here, so without a specific > > reason > > ?I would think that LGPL is a great license. > > > > - Sebastian > > > > > > > Josef > > > ? > > > > -n > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From josef.pktd at gmail.com Sat Jun 24 18:51:43 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 24 Jun 2017 18:51:43 -0400 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: <1498339562.2430.3.camel@sipsolutions.net> References: <20170621095155.GE19876@localhost> <1498334875.2430.1.camel@sipsolutions.net> <1498339562.2430.3.camel@sipsolutions.net> Message-ID: On Sat, Jun 24, 2017 at 5:26 PM, Sebastian Berg wrote: > On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote: > > Does this still apply: https://scipy.github.io/old-wiki/pages/License > > _Compatibility.html > > > > Of course, but it talks about putting it into the code base of scipy > not about being able to use the package in any way in a dependency > (i.e. `import package`). > But scipy does bundle a lot of external packages and then the license is relevant. I have no idea if this would apply here, but I find Sylvain's question relevant if closer integration and usage within the scientific python, and maybe Julia, community is desired. LGPL is not as bad as GPL but still adds another hurdle, as far as I know the scipy and related history. Josef > > - Sebastian > > > > Carl > > > > 2017-06-24 22:07 GMT+02:00 Sebastian Berg > >: > > > On Sat, 2017-06-24 at 15:47 -0400, josef.pktd at gmail.com wrote: > > > > > > > > > > > > On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith > > > > wrote: > > > > > On Jun 24, 2017 7:29 AM, "Sylvain Corlay" > > .com > > > > > > wrote: > > > > > > > > > > Also, one quick question: is the LGPL license a deliberate > > > choice > > > > > or is it not important to you? Most projects in the Python > > > > > scientific stack are BSD licensed. So the LGPL choice makes it > > > > > unlikely that a higher-level project adopts it as a dependency. > > > If > > > > > you are the only copyright holder, you would still have the > > > > > possibility to license it under a more permissive license such > > > as > > > > > BSD or MIT... > > > > > > > > > > Why would LGPL be a problem in a dependency? That doesn't stop > > > you > > > > > making your code BSD, and it's less restrictive license-wise > > > than > > > > > depending on MKL or the windows C runtime... > > > > > > > > > > > > > Is scipy still including any LGPL code, I thought not. > > > > There might still be some optional dependencies that not many > > > users > > > > are using by default. ? > > > > Julia packages are mostly MIT, AFAIK. (except for the GPL parts > > > > because of cholmod, which we (?) avoid) > > > > > > > > > > > > > Well, I don't think scipy has many dependencies (but I would not be > > > surprised if those are LGPL). Not a specialist, but as a dependency > > > it > > > should be fine (that is the point of the L in LGPL after all as far > > > as > > > I understand, it is much less viral). > > > If you package it with your own stuff, you have to make sure to > > > point > > > out that parts are LGPL of course (just like there is a reason you > > > get > > > the GPL printed out with some devices) and if you modify it provide > > > these modifications, etc. > > > > > > Of course you cannot include it into the scipy codebase itself, but > > > there is probably no aim of doing so here, so without a specific > > > reason > > > I would think that LGPL is a great license. > > > > > > - Sebastian > > > > > > > > > > Josef > > > > > > > > > -n > > > > > > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at python.org > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michelemartone at users.sourceforge.net Sun Jun 25 06:01:03 2017 From: michelemartone at users.sourceforge.net (Michele Martone) Date: Sun, 25 Jun 2017 12:01:03 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> Message-ID: <20170625100103.GC2167@localhost> On 20170624 at 16:29, Sylvain Corlay wrote: > Hi Michele, Hi Sylvain, > This is really interesting. I am a co-author of the xtensor project and one > thing that could be interesting is to wrap the various sparse matrix data > structures in the form of xtensor expressions. A byproduct of doing so is > that it would simplify creating bindings for multiple scientific computing > languages (Python, Julia, R, and more coming). You can see the blog post > http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- > xtensor.html for reference... This article exemplifies manipulation of numerical arrays. Now I ask you: Given an interactive language $L of the one you cite above, can xtensor provide objects with custom type and operators for manipulation in *that* language, like in e.g. the pyrsb case: a=rsb_matrix((4,4)) print(a+a) # + operator and 'print' interfacing ? > Also, one quick question: is the LGPL license a deliberate choice or is it > not important to you? Most projects in the Python scientific stack are BSD > licensed. So the LGPL choice makes it unlikely that a higher-level project > adopts it as a dependency. If you are the only copyright holder, you would > still have the possibility to license it under a more permissive license > such as BSD or MIT... No important choice. No problems relicensing the PyRSB prototype to BSD or MIT. > Congratulations on the release! Thanks for the interest and welcome constructive feedback :-) > Sylvain > > > > > > On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone sourceforge.net> wrote: > > > Hi. > > > > I'm the author of the high performance multithreaded sparse matrix > > library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ > > > > I'm *not* a user of SciPy/NumPy/Python, but using Cython I have > > written a proof-of-concept interface to librsb, named `PyRSB': > > https://github.com/michelemartone/pyrsb > > > > PyRSB is in a prototypal state; e.g. still lacks good error handling. > > Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. > > Advantages over csr_matrix are in fast multithreaded multiplication > > of huge sparse matrices. > > Intended application area is iterative solution of linear systems; > > particularly fast if with symmetric matrices and many rhs. > > > > With this email I am looking for prospective: > > - users/testers > > - developers (any interest to collaborate/adopt/include the project?) > > > > Looking forward for your feedback, > > Michele > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From charlesr.harris at gmail.com Sun Jun 25 12:32:26 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 25 Jun 2017 10:32:26 -0600 Subject: [Numpy-discussion] Boolean binary '-' operator Message-ID: Hi All, The boolean binary '-' operator was deprecated back in NumPy 1.9 and changed to an error in 1.13. This caused a number of failures in downstream projects. The choices now are to continue the deprecation for another couple of releases, or simply give up on the change. For booleans, `a - b` was implemented as `a xor b`, which leads to the somewhat unexpected identity `a - b == b - a`, but it is a handy operator that allows simplification of some functions, `numpy.diff` among therm. At this point I'm inclined to give up on the deprecation and retain the old behavior. It is a bit impure but perhaps we can consider it a feature rather than a bug. The unary `-` operator for booleans, now an error, was also deprecated in 1.9 and changed to an error in 1.13. There have been no complaints about that (yet), and it seems like a reasonable thing to do, so I am inclined to leave that error in place. What do others think the correct way forward is? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Sun Jun 25 12:45:38 2017 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sun, 25 Jun 2017 09:45:38 -0700 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: Message-ID: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Hi Chuck On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: > The boolean binary '-' operator was deprecated back in NumPy 1.9 and > changed to an error in 1.13. This caused a number of failures in > downstream projects. The choices now are to continue the deprecation > for another couple of releases, or simply give up on the change. For > booleans, `a - b` was implemented as `a xor b`, which leads to the > somewhat unexpected identity `a - b == b - a`, but it is a handy > operator that allows simplification of some functions, `numpy.diff` > among therm. At this point I'm inclined to give up on the deprecation > and retain the old behavior. It is a bit impure but perhaps we can > consider it a feature rather than a bug. What was the original motivation behind the deprecation? `xor` seems like exactly what one would expect when subtracting boolean arrays. But, in principle, I'm not against the deprecation (we've had to fix a few problems that arose in skimage, but nothing big). St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sun Jun 25 12:59:09 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 25 Jun 2017 18:59:09 +0200 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On 25.06.2017 18:45, Stefan van der Walt wrote: > Hi Chuck > > On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: >> The boolean binary '-' operator was deprecated back in NumPy 1.9 and >> changed to an error in 1.13. This caused a number of failures in >> downstream projects. The choices now are to continue the deprecation >> for another couple of releases, or simply give up on the change. For >> booleans, `a - b` was implemented as `a xor b`, which leads to the >> somewhat unexpected identity `a - b == b - a`, but it is a handy >> operator that allows simplification of some functions, `numpy.diff` >> among therm. At this point I'm inclined to give up on the deprecation >> and retain the old behavior. It is a bit impure but perhaps we can >> consider it a feature rather than a bug. > > What was the original motivation behind the deprecation? `xor` seems > like exactly what one would expect when subtracting boolean arrays. > > But, in principle, I'm not against the deprecation (we've had to fix a > few problems that arose in skimage, but nothing big). > > St?fan > > I am against this deprecation for apparently cosmetic reasons. Is there any practical drawback in that it makes subtraction commutative for booleans? numpy should not be imposing change of style when the existing sub par historical style does not cause actual bugs. While I don't like it I can accept a deprecation warning that will never be acted upon. From sylvain.corlay at gmail.com Sun Jun 25 15:24:37 2017 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Sun, 25 Jun 2017 21:24:37 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: <20170625100103.GC2167@localhost> References: <20170621095155.GE19876@localhost> <20170625100103.GC2167@localhost> Message-ID: Hi Michele, Regarding xtensor, the article focused on how we can wrap an existing data structure in an "xtensor expression", i.e. something that you can operate upon with the numpy style API in C++ (see the numpy to xtensor cheat sheet http://xtensor.readthedocs.io/en/latest/numpy.html). Since it is an expression system, you can do "a + b * sin(t)" with numpy-style broadcasting and it returns an un-evaluated expression, which is computed only upon access or assignment to a container. "a", "b", and "c" could all be compound expressions, backed by filesystem operations, be an in-memory container, or a wrapper on your sparse matrices... Now, we are working on automating the "*C++ tensor expression-type -> python wrapper with access and math operators" *generation of code, which would be what you are describing below... On the subject of the LGPL, regarding the distinction with the GPL that were brought up earlier in the thread, it is indeed not as problematic as the GPL, but still less permissive as BSD and MIT. In general, I think that using a BSD or MIT license averts these concerns in the Python ecosystem, and is preferable *if you don't care about copyleft*... Best, Sylvain On Sun, Jun 25, 2017 at 12:01 PM, Michele Martone < michelemartone at users.sourceforge.net> wrote: > On 20170624 at 16:29, Sylvain Corlay wrote: > > Hi Michele, > > Hi Sylvain, > > > This is really interesting. I am a co-author of the xtensor project and > one > > thing that could be interesting is to wrap the various sparse matrix data > > structures in the form of xtensor expressions. A byproduct of doing so is > > that it would simplify creating bindings for multiple scientific > computing > > languages (Python, Julia, R, and more coming). You can see the blog post > > http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- > > xtensor.html for reference... > This article exemplifies manipulation of numerical arrays. > Now I ask you: Given an interactive language $L of the one you cite above, > can xtensor provide objects with custom type and operators for manipulation > in *that* language, like in e.g. the pyrsb case: > a=rsb_matrix((4,4)) > print(a+a) # + operator and 'print' interfacing > ? > > > Also, one quick question: is the LGPL license a deliberate choice or is > it > > not important to you? Most projects in the Python scientific stack are > BSD > > licensed. So the LGPL choice makes it unlikely that a higher-level > project > > adopts it as a dependency. If you are the only copyright holder, you > would > > still have the possibility to license it under a more permissive license > > such as BSD or MIT... > No important choice. > No problems relicensing the PyRSB prototype to BSD or MIT. > > > Congratulations on the release! > Thanks for the interest and welcome constructive feedback :-) > > > Sylvain > > > > > > > > > > > > On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone > sourceforge.net> wrote: > > > > > Hi. > > > > > > I'm the author of the high performance multithreaded sparse matrix > > > library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ > > > > > > I'm *not* a user of SciPy/NumPy/Python, but using Cython I have > > > written a proof-of-concept interface to librsb, named `PyRSB': > > > https://github.com/michelemartone/pyrsb > > > > > > PyRSB is in a prototypal state; e.g. still lacks good error handling. > > > Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. > > > Advantages over csr_matrix are in fast multithreaded multiplication > > > of huge sparse matrices. > > > Intended application area is iterative solution of linear systems; > > > particularly fast if with symmetric matrices and many rhs. > > > > > > With this email I am looking for prospective: > > > - users/testers > > > - developers (any interest to collaborate/adopt/include the project?) > > > > > > Looking forward for your feedback, > > > Michele > > > > > > _______________________________________________ > > > SciPy-Dev mailing list > > > SciPy-Dev at python.org > > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michelemartone at users.sourceforge.net Sun Jun 25 06:01:03 2017 From: michelemartone at users.sourceforge.net (Michele Martone) Date: Sun, 25 Jun 2017 12:01:03 +0200 Subject: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library In-Reply-To: References: <20170621095155.GE19876@localhost> Message-ID: On 20170624 at 16:29, Sylvain Corlay wrote: > Hi Michele, Hi Sylvain, > This is really interesting. I am a co-author of the xtensor project and one > thing that could be interesting is to wrap the various sparse matrix data > structures in the form of xtensor expressions. A byproduct of doing so is > that it would simplify creating bindings for multiple scientific computing > languages (Python, Julia, R, and more coming). You can see the blog post > http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- > xtensor.html for reference... This article exemplifies manipulation of numerical arrays. Now I ask you: Given an interactive language $L of the one you cite above, can xtensor provide objects with custom type and operators for manipulation in *that* language, like in e.g. the pyrsb case: a=rsb_matrix((4,4)) print(a+a) # + operator and 'print' interfacing ? > Also, one quick question: is the LGPL license a deliberate choice or is it > not important to you? Most projects in the Python scientific stack are BSD > licensed. So the LGPL choice makes it unlikely that a higher-level project > adopts it as a dependency. If you are the only copyright holder, you would > still have the possibility to license it under a more permissive license > such as BSD or MIT... No important choice. No problems relicensing the PyRSB prototype to BSD or MIT. > Congratulations on the release! Thanks for the interest and welcome constructive feedback :-) > Sylvain > > > > > > On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone sourceforge.net> wrote: > > > Hi. > > > > I'm the author of the high performance multithreaded sparse matrix > > library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ > > > > I'm *not* a user of SciPy/NumPy/Python, but using Cython I have > > written a proof-of-concept interface to librsb, named `PyRSB': > > https://github.com/michelemartone/pyrsb > > > > PyRSB is in a prototypal state; e.g. still lacks good error handling. > > Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. > > Advantages over csr_matrix are in fast multithreaded multiplication > > of huge sparse matrices. > > Intended application area is iterative solution of linear systems; > > particularly fast if with symmetric matrices and many rhs. > > > > With this email I am looking for prospective: > > - users/testers > > - developers (any interest to collaborate/adopt/include the project?) > > > > Looking forward for your feedback, > > Michele > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: Unsupported File Types Alert.txt Type: application/octet-stream Size: 253 bytes Desc: Unsupported File Types Alert.txt URL: From pierre.debuyl at kuleuven.be Mon Jun 26 06:49:53 2017 From: pierre.debuyl at kuleuven.be (Pierre de Buyl) Date: Mon, 26 Jun 2017 12:49:53 +0200 Subject: [Numpy-discussion] EuroSciPy 2017 call for contributions - extension of deadline Message-ID: <20170626104953.GA16487@pi-x230> (Apologies if you receive multiple copies of this message) 10th European Conference on Python in Science August 28 - September 1, 2017 in Erlangen, Germany The Call for Papers is extended to July 02, 2017 23:00 CEST Description: The EuroSciPy meeting is a cross-disciplinary gathering focused on the use and development of the Python language in scientific research. This event strives to bring together both users and developers of scientific tools, as well as academic research and state of the art industry. Erlangen is one of Germany's major science hubs and located north of Munich (90 minutes by train). The Call for Papers is extended to July 02, 2017 23:00 CEST Regards, The EuroSciPy team https://www.euroscipy.org/2017/ From sebastian at sipsolutions.net Mon Jun 26 18:35:46 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 27 Jun 2017 00:35:46 +0200 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: <1498516546.30558.2.camel@sipsolutions.net> On Sun, 2017-06-25 at 18:59 +0200, Julian Taylor wrote: > On 25.06.2017 18:45, Stefan van der Walt wrote: > > Hi Chuck > > > > On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: > > > The boolean binary '-' operator was deprecated back in NumPy 1.9 > > > and > > > changed to an error in 1.13. This caused a number of failures in > > > downstream projects. The choices now are to continue the > > > deprecation > > > for another couple of releases, or simply give up on the change. > > > For > > > booleans,??`a - b` was implemented as `a xor b`, which leads to > > > the > > > somewhat unexpected identity `a - b == b - a`, but it is a handy > > > operator that allows simplification of some functions, > > > `numpy.diff` > > > among therm. At this point I'm inclined to give up on the > > > deprecation > > > and retain the old behavior. It is a bit impure but perhaps we > > > can > > > consider it a feature rather than a bug. > > > > What was the original motivation behind the deprecation???`xor` > > seems > > like exactly what one would expect when subtracting boolean arrays. > > > > But, in principle, I'm not against the deprecation (we've had to > > fix a > > few problems that arose in skimage, but nothing big). > > > > St?fan > > > > > > I am against this deprecation for apparently cosmetic reasons. > Is there any practical drawback in that it makes subtraction > commutative > for booleans? > > numpy should not be imposing change of style when the existing sub > par > historical style does not cause actual bugs. > > While I don't like it I can accept a deprecation warning that will > never > be acted upon. Dunno, that is also weird, then a UserWarning might even be better ;), but more visible.... For the unary minus, there are good reasons. For subtract, I don't remember really, but I don't think there was any huge argument for it. Probably it was mostly that many feel that: `False - True == -1` as is the case in python while we have: `np.False_ - np.True_ == np.True_`. And going to a deprecation would open up that possibility (though maybe you could go there directly). Not that I am convinced of that option. So, I don't mind much either way, but unless there is a concrete plan with quite a bit of support we should maybe just go with the conservative option. - Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Mon Jun 26 19:25:04 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Jun 2017 16:25:04 -0700 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Sun, Jun 25, 2017 at 9:45 AM, Stefan van der Walt wrote: > Hi Chuck > > On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: > >> The boolean binary '-' operator was deprecated back in NumPy 1.9 and changed >> to an error in 1.13. This caused a number of failures in downstream >> projects. The choices now are to continue the deprecation for another couple >> of releases, or simply give up on the change. For booleans, `a - b` was >> implemented as `a xor b`, which leads to the somewhat unexpected identity `a >> - b == b - a`, but it is a handy operator that allows simplification of some >> functions, `numpy.diff` among therm. At this point I'm inclined to give up >> on the deprecation and retain the old behavior. It is a bit impure but >> perhaps we can consider it a feature rather than a bug. > > > What was the original motivation behind the deprecation? `xor` seems like > exactly what one would expect when subtracting boolean arrays. > > But, in principle, I'm not against the deprecation (we've had to fix a few > problems that arose in skimage, but nothing big). I believe that this happened as part of a review of the whole arithmetic system for np.bool_. Traditionally, we have + is "or", binary - is "xor", and unary - is "not". Here are some identities you might expect, if 'a' and 'b' are np.bool_ objects: a - b = a + (-b) a + b - b = a bool(a + b) = bool(a) + bool(b) bool(a - b) = bool(a) - bool(b) bool(-a) = -bool(a) But in fact none of these identities hold. Furthermore, the np.bool_ arithmetic operations are all confusing synonyms for operations that could be written more clearly using the proper boolean operators |, ^, ~, so they violate TOOWTDI. So I think the general idea was to deprecate all of this nonsense. It looks like what actually happened is that binary - and unary - got deprecated a while back and are now raising errors in 1.13.0, but + did not. This is sort of unfortunate, because binary - is the only one of these that's somewhat defensible (it doesn't match the builtin bool type, but it does at least correspond to subtraction in Z/2, so identities like 'a - (b - b) = a' do hold). I guess my preference would be: 1) deprecate + 2) move binary - back to deprecated-but-not-an-error 3) fix np.diff to use logical_xor when the inputs are boolean, since that seems to be what people expect 4) keep unary - as an error And if we want to be less aggressive, then a reasonable alternative would be: 1) deprecate + 2) un-deprecate binary - 3) keep unary - as an error -n -- Nathaniel J. Smith -- https://vorpus.org From jni.soma at gmail.com Mon Jun 26 20:14:08 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Tue, 27 Jun 2017 10:14:08 +1000 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: OMG deprecating + would be a nightmare. I can?t even begin to count the number of times I?ve used e.g. np.sum(arr == num)? Originally with a dtype cast but generally I?ve removed it because it worked. ? But I just saw the behaviour of `sum` is different from that of adding arrays together (where it indeed means `or`), which I agree is confusing. As long as the sum and mean behaviours are unchanged, I won?t raise too much of a fuss. =P Generally, although one might expect xor, what *I* would expect is for the behaviour to match the Python bool type, which is not the case right now. So my vote would be to modify ***in NumPy 2.0*** the behaviour of + and - to match Python?s built-in bool (ie upcasting to int). And, in general, I?m in favour of something as foundational as NumPy, in version 1.x, to follow semantic versioning and not break APIs until 2.x. Juan. On 27 Jun 2017, 9:25 AM +1000, Nathaniel Smith , wrote: > On Sun, Jun 25, 2017 at 9:45 AM, Stefan van der Walt > wrote: > > Hi Chuck > > > > On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: > > > > > The boolean binary '-' operator was deprecated back in NumPy 1.9 and changed > > > to an error in 1.13. This caused a number of failures in downstream > > > projects. The choices now are to continue the deprecation for another couple > > > of releases, or simply give up on the change. For booleans, `a - b` was > > > implemented as `a xor b`, which leads to the somewhat unexpected identity `a > > > - b == b - a`, but it is a handy operator that allows simplification of some > > > functions, `numpy.diff` among therm. At this point I'm inclined to give up > > > on the deprecation and retain the old behavior. It is a bit impure but > > > perhaps we can consider it a feature rather than a bug. > > > > > > What was the original motivation behind the deprecation? `xor` seems like > > exactly what one would expect when subtracting boolean arrays. > > > > But, in principle, I'm not against the deprecation (we've had to fix a few > > problems that arose in skimage, but nothing big). > > I believe that this happened as part of a review of the whole > arithmetic system for np.bool_. Traditionally, we have + is "or", > binary - is "xor", and unary - is "not". > > Here are some identities you might expect, if 'a' and 'b' are np.bool_ objects: > > a - b = a + (-b) > a + b - b = a > bool(a + b) = bool(a) + bool(b) > bool(a - b) = bool(a) - bool(b) > bool(-a) = -bool(a) > > But in fact none of these identities hold. Furthermore, the np.bool_ > arithmetic operations are all confusing synonyms for operations that > could be written more clearly using the proper boolean operators |, ^, > ~, so they violate TOOWTDI. So I think the general idea was to > deprecate all of this nonsense. > > It looks like what actually happened is that binary - and unary - got > deprecated a while back and are now raising errors in 1.13.0, but + > did not. This is sort of unfortunate, because binary - is the only one > of these that's somewhat defensible (it doesn't match the builtin bool > type, but it does at least correspond to subtraction in Z/2, so > identities like 'a - (b - b) = a' do hold). > > I guess my preference would be: > 1) deprecate + > 2) move binary - back to deprecated-but-not-an-error > 3) fix np.diff to use logical_xor when the inputs are boolean, since > that seems to be what people expect > 4) keep unary - as an error > > And if we want to be less aggressive, then a reasonable alternative would be: > 1) deprecate + > 2) un-deprecate binary - > 3) keep unary - as an error > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 26 21:56:22 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Jun 2017 19:56:22 -0600 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Mon, Jun 26, 2017 at 6:14 PM, Juan Nunez-Iglesias wrote: > OMG deprecating + would be a nightmare. I can?t even begin to count the > number of times I?ve used e.g. np.sum(arr == num)? Originally with a dtype > cast but generally I?ve removed it because it worked. > > ? But I just saw the behaviour of `sum` is different from that of adding > arrays together (where it indeed means `or`), which I agree is confusing. > As long as the sum and mean behaviours are unchanged, I won?t raise too > much of a fuss. =P > > Generally, although one might expect xor, what *I* would expect is for the > behaviour to match the Python bool type, which is not the case right now. > So my vote would be to modify ***in NumPy 2.0*** the behaviour of + and - > to match Python?s built-in bool (ie upcasting to int). > > And, in general, I?m in favour of something as foundational as NumPy, in > version 1.x, to follow semantic versioning and not break APIs until 2.x. > > Juan. > > On 27 Jun 2017, 9:25 AM +1000, Nathaniel Smith , wrote: > > On Sun, Jun 25, 2017 at 9:45 AM, Stefan van der Walt > wrote: > > Hi Chuck > > On Sun, Jun 25, 2017, at 09:32, Charles R Harris wrote: > > The boolean binary '-' operator was deprecated back in NumPy 1.9 and > changed > to an error in 1.13. This caused a number of failures in downstream > projects. The choices now are to continue the deprecation for another > couple > of releases, or simply give up on the change. For booleans, `a - b` was > implemented as `a xor b`, which leads to the somewhat unexpected identity > `a > - b == b - a`, but it is a handy operator that allows simplification of > some > functions, `numpy.diff` among therm. At this point I'm inclined to give up > on the deprecation and retain the old behavior. It is a bit impure but > perhaps we can consider it a feature rather than a bug. > > > > What was the original motivation behind the deprecation? `xor` seems like > exactly what one would expect when subtracting boolean arrays. > > But, in principle, I'm not against the deprecation (we've had to fix a few > problems that arose in skimage, but nothing big). > > > I believe that this happened as part of a review of the whole > arithmetic system for np.bool_. Traditionally, we have + is "or", > binary - is "xor", and unary - is "not". > > Here are some identities you might expect, if 'a' and 'b' are np.bool_ > objects: > > a - b = a + (-b) > a + b - b = a > bool(a + b) = bool(a) + bool(b) > bool(a - b) = bool(a) - bool(b) > bool(-a) = -bool(a) > > But in fact none of these identities hold. Furthermore, the np.bool_ > arithmetic operations are all confusing synonyms for operations that > could be written more clearly using the proper boolean operators |, ^, > ~, so they violate TOOWTDI. So I think the general idea was to > deprecate all of this nonsense. > > It looks like what actually happened is that binary - and unary - got > deprecated a while back and are now raising errors in 1.13.0, but + > did not. This is sort of unfortunate, because binary - is the only one > of these that's somewhat defensible (it doesn't match the builtin bool > type, but it does at least correspond to subtraction in Z/2, so > identities like 'a - (b - b) = a' do hold). > > That's because xor corresponds to addition in Z/2 and every element is its own additive inverse. > I guess my preference would be: > 1) deprecate + > 2) move binary - back to deprecated-but-not-an-error > 3) fix np.diff to use logical_xor when the inputs are boolean, since > that seems to be what people expect > 4) keep unary - as an error > > And if we want to be less aggressive, then a reasonable alternative would > be: > 1) deprecate + > 2) un-deprecate binary - > 3) keep unary - as an error > > Using '+' for 'or' and '*' for 'and' is pretty common and the variation of '+' for 'xor' was common back in the day because 'and' and 'xor' make boolean algebra a ring, which appealed to mathematicians as opposed to everyone else ;) You can see the same progression in measure theory where eventually intersection and xor (symmetric difference) was replaced with union and complement. Using '-' for xor is something I hadn't seen outside of numpy, but I suspect it must be standard somewhere. I would leave '*' and '+' alone, as the breakage and inconvenience from removing them would be significant. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cppljevans at suddenlink.net Tue Jun 27 05:54:18 2017 From: cppljevans at suddenlink.net (Larry Evans) Date: Tue, 27 Jun 2017 04:54:18 -0500 Subject: [Numpy-discussion] why a[0][0].__mul__(a[0][0]) where a is np.array, gives 'missing 1 required positional argument'? Message-ID: The following code: --{--cut here-- #Purpose: # Demonstrate class with __add__ and __mult__ methods # to demonstrate shortestPath calculations. #OriginalSource: # ./shortestPath.py #References: # [1] http://www.marinamele.com/2014/04/modifying-add-method-of-python-class.html # [2] https://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html#c.PyArray_Descr # [3] https://www.abebooks.com/book-search/title/graphs-networks/author/carre-bernard/ # Table 3.1, path algebra P2. # [4] https://docs.python.org/3/library/abc.html ########################################### import numpy as np from abc import ABCMeta, abstractmethod class PathAbstract(metaclass=ABCMeta): @abstractmethod def __repr__(self): pass @abstractmethod def __add__(self, other): pass @abstractmethod def __radd__(self, other): pass @abstractmethod def __mul__(self, other): pass @abstractmethod def __rmul__(self, other): pass class ShortestNull(PathAbstract): '''Null object in ShortestPath algebra[3]''' def __str__(self): return "<0>" def __repr__(self): return self.__str__() def __addreal__(self, other): return other def __add__(self, other): return other def __radd__(self, other): return other def __mulreal__(self, other): return self def __mul__(self, other): return self def __rmul__(self, other): return self class ShortestUnit(PathAbstract): '''Unit object in ShortestPath algebra[3]''' def __str__(self): return "<1>" def __repr__(self): return self.__str__() def __addreal__(self, other): return self def __add__(self, other): return self def __radd__(self, other): return self def __mulreal__(self, other): return other def __mul__(self, other): return other def __rmul__(self, other): return other class ShortestPath(PathAbstract): def __init__(self, distance): self.distance = distance def __str__(self): return "{0:4.1f}".format(self.distance) def __repr__(self): return self.__str__() def __addreal__(self, other): #paths composed of following shortest of self path or other path. min_dist=min(self.distance,other.distance) return ShortestPath(min_dist) def __add__(self, other): #paths composed of following shortest of self path or other path. if(other == 0): n=ShortestNull() return n.__addreal__(self) else: return other.__addreal__(self) def __radd__(self, other): return self.__add__(other) def __mulreal__(self, other): #path composed of self path followed by other path. min_dist=self.distance+other.distance return ShortestPath(min_dist) def __mul__(self, other): return other.__mulreal__(self) def __rmul__(self, other): return other.__mulreal__(self) d0=ShortestNull() d1=ShortestUnit() d2=ShortestPath(3) d3=ShortestPath(6) print(' d0=',d0,' d1=',d1,' d2=',d2,' d3=',d3) print(' type(d0)=',type(d0).__name__) d4=d0+d2+d3 print('d4=',d4) d5=d2*d3 print('d5=',d5) d6=d0*d2 print('d6=',d6) d7=d1*d2 print('d7=',d7) d8=d2*d0 print('d8=',d8) d9=d2*d1 print('d9=',d9) a=np.array([[ShortestNull,ShortestPath(12)],[ShortestPath(12),ShortestNull()]],dtype=object) print('a=',a,sep='\n') print('a[0]=',a[0],sep='\n') print('a[0][0]=',a[0][0],sep='\n') #a00mul=a[0][0]*a[0][0] a00mul=a[0][0].__mul__(a[0][0]) print('a00mul=',a00mul,sep='\n') #at=a.transpose() #print('at=',at,sep='\n') #aat=a.dot(at) print('aat=',aat,sep='\n') #mp=np.matmul(a,at)#this gives error about Object arrays not supported. #print('mp=',mp) --}--cut here-- When run by make, gives this result: --{--cut here-- make -k python3 shortestPathABC.py d0= <0> d1= <1> d2= 3.0 d3= 6.0 type(d0)= ShortestNull d4= 3.0 d5= 9.0 d6= <0> d7= 3.0 d8= <0> d9= 3.0 a= [[ 12.0] [12.0 <0>]] a[0]= [ 12.0] a[0][0]= Traceback (most recent call last): File "shortestPathABC.py", line 123, in a00mul=a[0][0].__mul__(a[0][0]) TypeError: __mul__() missing 1 required positional argument: 'other' Makefile:7: recipe for target 'all' failed make: *** [all] Error 1 --}--cut here-- I don't understand why. Apparently, a[0][0] is not a ShortestNull because otherwise, the .__mul__ would have the positional argument, 'other' equal to a[0][0]. What am I missing? TIA. -regards, Larry From robert.kern at gmail.com Tue Jun 27 13:30:46 2017 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 27 Jun 2017 10:30:46 -0700 Subject: [Numpy-discussion] why a[0][0].__mul__(a[0][0]) where a is np.array, gives 'missing 1 required positional argument'? In-Reply-To: References: Message-ID: On Tue, Jun 27, 2017 at 2:54 AM, Larry Evans wrote: > a= > [[ 12.0] > [12.0 <0>]] > a[0]= > [ 12.0] > a[0][0]= > > Traceback (most recent call last): > File "shortestPathABC.py", line 123, in > a00mul=a[0][0].__mul__(a[0][0]) > TypeError: __mul__() missing 1 required positional argument: 'other' > Makefile:7: recipe for target 'all' failed > make: *** [all] Error 1 > --}--cut here-- > > I don't understand why. Apparently, a[0][0] is not a ShortestNull because otherwise, the .__mul__ would have the positional argument, > 'other' equal to a[0][0]. > > What am I missing? > a=np.array([[ShortestNull,ShortestPath(12)],[ShortestPath( 12),ShortestNull()]],dtype=object) You didn't instantiate ShortestNull but passed the class object instead. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jun 27 13:36:45 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 27 Jun 2017 19:36:45 +0200 Subject: [Numpy-discussion] why a[0][0].__mul__(a[0][0]) where a is np.array, gives 'missing 1 required positional argument'? In-Reply-To: References: Message-ID: <1498585005.2559.3.camel@sipsolutions.net> On Tue, 2017-06-27 at 04:54 -0500, Larry Evans wrote: > The following code: > > --{--cut here-- > #Purpose: > #??Demonstrate class with __add__ and __mult__ methods > #??to demonstrate shortestPath calculations. > #OriginalSource: > #??./shortestPath.py > #References: > #??[1]? > http://www.marinamele.com/2014/04/modifying-add-method-of-python-clas > s.html > #??[2]? > https://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures > .html#c.PyArray_Descr > #??[3]? > https://www.abebooks.com/book-search/title/graphs-networks/author/car > re-bernard/ > #??????Table 3.1, path algebra P2. > #??[4] https://docs.python.org/3/library/abc.html > ########################################### > import numpy as np > from abc import ABCMeta, abstractmethod > class PathAbstract(metaclass=ABCMeta): > ?????@abstractmethod > ?????def __repr__(self): > ?????????pass > ?????@abstractmethod > ?????def __add__(self, other): > ?????????pass > ?????@abstractmethod > ?????def __radd__(self, other): > ?????????pass > ?????@abstractmethod > ?????def __mul__(self, other): > ?????????pass > ?????@abstractmethod > ?????def __rmul__(self, other): > ?????????pass > > class ShortestNull(PathAbstract): > ?????'''Null object in ShortestPath algebra[3]''' > ?????def __str__(self): > ?????????return "<0>" > ?????def __repr__(self): > ?????????return self.__str__() > ?????def __addreal__(self, other): > ?????????return other > ?????def __add__(self, other): > ?????????return other > ?????def __radd__(self, other): > ?????????return other > ?????def __mulreal__(self, other): > ?????????return self > ?????def __mul__(self, other): > ?????????return self > ?????def __rmul__(self, other): > ?????????return self > > class ShortestUnit(PathAbstract): > ?????'''Unit object in ShortestPath algebra[3]''' > ?????def __str__(self): > ?????????return "<1>" > ?????def __repr__(self): > ?????????return self.__str__() > ?????def __addreal__(self, other): > ?????????return self > ?????def __add__(self, other): > ?????????return self > ?????def __radd__(self, other): > ?????????return self > ?????def __mulreal__(self, other): > ?????????return other > ?????def __mul__(self, other): > ?????????return other > ?????def __rmul__(self, other): > ?????????return other > > class ShortestPath(PathAbstract): > ?????def __init__(self, distance): > ?????????self.distance = distance > ?????def __str__(self): > ?????????return "{0:4.1f}".format(self.distance) > ?????def __repr__(self): > ?????????return self.__str__() > ?????def __addreal__(self, other): > ?????????#paths composed of following shortest of self path or other > path. > ?????????min_dist=min(self.distance,other.distance) > ?????????return ShortestPath(min_dist) > ?????def __add__(self, other): > ?????????#paths composed of following shortest of self path or other > path. > ?????????if(other == 0): > ?????????????n=ShortestNull() > ?????????????return n.__addreal__(self) > ?????????else: > ?????????????return other.__addreal__(self) > ?????def __radd__(self, other): > ?????????return self.__add__(other) > ?????def __mulreal__(self, other): > ?????????#path composed of self path followed by other path. > ?????????min_dist=self.distance+other.distance > ?????????return ShortestPath(min_dist) > ?????def __mul__(self, other): > ?????????return other.__mulreal__(self) > ?????def __rmul__(self, other): > ?????????return other.__mulreal__(self) > > > d0=ShortestNull() > d1=ShortestUnit() > d2=ShortestPath(3) > d3=ShortestPath(6) > print(' d0=',d0,' d1=',d1,' d2=',d2,' d3=',d3) > print(' type(d0)=',type(d0).__name__) > d4=d0+d2+d3 > print('d4=',d4) > d5=d2*d3 > print('d5=',d5) > d6=d0*d2 > print('d6=',d6) > d7=d1*d2 > print('d7=',d7) > d8=d2*d0 > print('d8=',d8) > d9=d2*d1 > print('d9=',d9) > a=np.array([[ShortestNull,ShortestPath(12)],[ShortestPath(12),Shortes > tNull()]],dtype=object) > print('a=',a,sep='\n') > print('a[0]=',a[0],sep='\n') > print('a[0][0]=',a[0][0],sep='\n') > #a00mul=a[0][0]*a[0][0] > a00mul=a[0][0].__mul__(a[0][0]) > print('a00mul=',a00mul,sep='\n') > #at=a.transpose() > #print('at=',at,sep='\n') > #aat=a.dot(at) > print('aat=',aat,sep='\n') > #mp=np.matmul(a,at)#this gives error about Object arrays not > supported. > #print('mp=',mp) > --}--cut here-- > > > When run by make, gives this result: > > > --{--cut here-- > make -k > python3 shortestPathABC.py > ? d0= <0>??d1= <1>??d2=??3.0??d3=??6.0 > ? type(d0)= ShortestNull > d4=??3.0 > d5=??9.0 > d6= <0> > d7=??3.0 > d8= <0> > d9=??3.0 > a= > [[ 12.0] > ? [12.0 <0>]] > a[0]= > [ 12.0] > a[0][0]= > > Traceback (most recent call last): > ???File "shortestPathABC.py", line 123, in > ?????a00mul=a[0][0].__mul__(a[0][0]) > TypeError: __mul__() missing 1 required positional argument: 'other' > Makefile:7: recipe for target 'all' failed > make: *** [all] Error 1 > --}--cut here-- > > I don't understand why.??Apparently, a[0][0] is not a ShortestNull? > because otherwise, the .__mul__ would have the positional argument, > 'other' equal to a[0][0]. > I don't think debugging support really suits the list, but.... how about you see why in your result: [[ 12.0] ? [12.0 <0>]] a[0, 0] and a[1, 1] do not show up as the same thing? - Sebastian > What am I missing? > > TIA. > > -regards, > Larry > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Tue Jun 27 17:35:23 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 14:35:23 -0700 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Jun 26, 2017 6:56 PM, "Charles R Harris" wrote: > On 27 Jun 2017, 9:25 AM +1000, Nathaniel Smith , wrote: > I guess my preference would be: > 1) deprecate + > 2) move binary - back to deprecated-but-not-an-error > 3) fix np.diff to use logical_xor when the inputs are boolean, since > that seems to be what people expect > 4) keep unary - as an error > > And if we want to be less aggressive, then a reasonable alternative would > be: > 1) deprecate + > 2) un-deprecate binary - > 3) keep unary - as an error > > Using '+' for 'or' and '*' for 'and' is pretty common and the variation of '+' for 'xor' was common back in the day because 'and' and 'xor' make boolean algebra a ring, which appealed to mathematicians as opposed to everyone else ;) '+' for 'xor' and '*' for 'and' is perfectly natural; that's just + and * in Z/2. It's not only a ring, it's a field! '+' for 'or' is much weirder; why would you use '+' for an operation that's not even invertible? I guess it's a semi-ring. But we have the '|' character right there; there's no expectation that every weird mathematical notation will be matched in numpy... The most notable is that '*' doesn't mean matrix multiplication. You can see the same progression in measure theory where eventually intersection and xor (symmetric difference) was replaced with union and complement. Using '-' for xor is something I hadn't seen outside of numpy, but I suspect it must be standard somewhere. I would leave '*' and '+' alone, as the breakage and inconvenience from removing them would be significant. '*' doesn't bother me, because it really does have only one sensible behavior; even built-in bool() effectively uses 'and' for '*'. But, now I remember... The major issue here is that some people want dot(a, b) on Boolean matrices to use these semantics, right? Because in this particular case it leads to some useful connections to the matrix representation for logical relations [1]. So it's sort of similar to the diff() case. For the basic operation, using '|' or '^' is fine, but there are these derived operations like 'dot' and 'diff' where people have different expectations. I guess Juan's example of 'sum' is relevant here too. It's pretty weird that if 'a' and 'b' are one-dimensional boolean arrays, 'a @ b' and 'sum(a * b)' give totally different results. So that's the fundamental problem: there are a ton of possible conventions that are each appealing in one narrow context, and they all contradict each other, so trying to shove them all into numpy simultaneously is messy. I'm glad we at least seem to have succeeded in getting rid of unary '-', that one was particularly indefensible in the context of everything else :-). For the rest, I'm really not sure whether it's better to deprecate everything and tell people to use specialized tools for specialized purposes (e.g. add a 'logical_dot'), or to special case the high-level operations people want (make 'dot' and 'diff' continue to work, but deprecate + and -), or just leave the whole incoherent mish-mash alone. -n [1] https://en.wikipedia.org/wiki/Logical_matrix -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Tue Jun 27 18:01:09 2017 From: ben.v.root at gmail.com (Benjamin Root) Date: Tue, 27 Jun 2017 18:01:09 -0400 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: Forgive my ignorance, but what is "Z/2"? On Tue, Jun 27, 2017 at 5:35 PM, Nathaniel Smith wrote: > On Jun 26, 2017 6:56 PM, "Charles R Harris" > wrote: > > >> On 27 Jun 2017, 9:25 AM +1000, Nathaniel Smith , wrote: >> > I guess my preference would be: >> 1) deprecate + >> 2) move binary - back to deprecated-but-not-an-error >> 3) fix np.diff to use logical_xor when the inputs are boolean, since >> that seems to be what people expect >> 4) keep unary - as an error >> >> And if we want to be less aggressive, then a reasonable alternative would >> be: >> 1) deprecate + >> 2) un-deprecate binary - >> 3) keep unary - as an error >> >> > Using '+' for 'or' and '*' for 'and' is pretty common and the variation of > '+' for 'xor' was common back in the day because 'and' and 'xor' make > boolean algebra a ring, which appealed to mathematicians as opposed to > everyone else ;) > > > '+' for 'xor' and '*' for 'and' is perfectly natural; that's just + and * > in Z/2. It's not only a ring, it's a field! '+' for 'or' is much weirder; > why would you use '+' for an operation that's not even invertible? I guess > it's a semi-ring. But we have the '|' character right there; there's no > expectation that every weird mathematical notation will be matched in > numpy... The most notable is that '*' doesn't mean matrix multiplication. > > > You can see the same progression in measure theory where eventually > intersection and xor (symmetric difference) was replaced with union and > complement. Using '-' for xor is something I hadn't seen outside of numpy, > but I suspect it must be standard somewhere. I would leave '*' and '+' > alone, as the breakage and inconvenience from removing them would be > significant. > > > '*' doesn't bother me, because it really does have only one sensible > behavior; even built-in bool() effectively uses 'and' for '*'. > > But, now I remember... The major issue here is that some people want > dot(a, b) on Boolean matrices to use these semantics, right? Because in > this particular case it leads to some useful connections to the matrix > representation for logical relations [1]. So it's sort of similar to the > diff() case. For the basic operation, using '|' or '^' is fine, but there > are these derived operations like 'dot' and 'diff' where people have > different expectations. > > I guess Juan's example of 'sum' is relevant here too. It's pretty weird > that if 'a' and 'b' are one-dimensional boolean arrays, 'a @ b' and 'sum(a > * b)' give totally different results. > > So that's the fundamental problem: there are a ton of possible conventions > that are each appealing in one narrow context, and they all contradict each > other, so trying to shove them all into numpy simultaneously is messy. > > I'm glad we at least seem to have succeeded in getting rid of unary '-', > that one was particularly indefensible in the context of everything else > :-). For the rest, I'm really not sure whether it's better to deprecate > everything and tell people to use specialized tools for specialized > purposes (e.g. add a 'logical_dot'), or to special case the high-level > operations people want (make 'dot' and 'diff' continue to work, but > deprecate + and -), or just leave the whole incoherent mish-mash alone. > > -n > > [1] https://en.wikipedia.org/wiki/Logical_matrix > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jun 27 18:09:18 2017 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 27 Jun 2017 15:09:18 -0700 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Tue, Jun 27, 2017 at 3:01 PM, Benjamin Root wrote: > > Forgive my ignorance, but what is "Z/2"? https://groupprops.subwiki.org/wiki/Cyclic_group:Z2 https://en.wikipedia.org/wiki/Cyclic_group -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Jun 27 19:07:20 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 16:07:20 -0700 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Tue, Jun 27, 2017 at 3:09 PM, Robert Kern wrote: > On Tue, Jun 27, 2017 at 3:01 PM, Benjamin Root wrote: >> >> Forgive my ignorance, but what is "Z/2"? > > https://groupprops.subwiki.org/wiki/Cyclic_group:Z2 > https://en.wikipedia.org/wiki/Cyclic_group This might be a slightly better link? https://en.wikipedia.org/wiki/Modular_arithmetic#Integers_modulo_n Anyway, it's a math-nerd way of saying "the integers modulo two", i.e. the numbers 0 and 1 with * as AND and + as XOR. But the nice thing about Z/2 is that if you know some abstract algebra, then one of the most fundamental theorems is that if p is prime then Z/p is a "field", meaning that * and + are particularly well-behaved. And 2 is a prime, so pointing out that the bools with AND and XOR is the same as Z/2 is a way of saying "this way of defining * and + is internally consistent and well-behaved". -n -- Nathaniel J. Smith -- https://vorpus.org From wieser.eric+numpy at gmail.com Tue Jun 27 19:35:13 2017 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Tue, 27 Jun 2017 23:35:13 +0000 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: It seems to me that after a healthy post-deprecation cycle, and if we choose to keep the Z/2 meaning of __sub__, it might be worth reintroducing __neg__ as a no-op? AFAICT, this is consistent with the Z/2 interpretation? Eric On Wed, 28 Jun 2017 at 00:08 Nathaniel Smith wrote: > On Tue, Jun 27, 2017 at 3:09 PM, Robert Kern > wrote: > > On Tue, Jun 27, 2017 at 3:01 PM, Benjamin Root > wrote: > >> > >> Forgive my ignorance, but what is "Z/2"? > > > > https://groupprops.subwiki.org/wiki/Cyclic_group:Z2 > > https://en.wikipedia.org/wiki/Cyclic_group > > This might be a slightly better link? > https://en.wikipedia.org/wiki/Modular_arithmetic#Integers_modulo_n > > Anyway, it's a math-nerd way of saying "the integers modulo two", i.e. > the numbers 0 and 1 with * as AND and + as XOR. But the nice thing > about Z/2 is that if you know some abstract algebra, then one of the > most fundamental theorems is that if p is prime then Z/p is a "field", > meaning that * and + are particularly well-behaved. And 2 is a prime, > so pointing out that the bools with AND and XOR is the same as Z/2 is > a way of saying "this way of defining * and + is internally consistent > and well-behaved". > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Tue Jun 27 19:58:10 2017 From: alan.isaac at gmail.com (Alan Isaac) Date: Tue, 27 Jun 2017 19:58:10 -0400 Subject: [Numpy-discussion] : Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: <4d7842ac-3d21-c408-98a7-7825c10199ab@gmail.com> On 6/27/2017 5:35 PM, Nathaniel Smith wrote: > I remember... The major issue here is that some people want dot(a, b) on Boolean matrices to use these semantics, right? Yes; this has worked in the past, and loss of this functionality is unexpected. That said, I haven't used this outside of a teaching context, so for me it only affects some course notes. I suppose loss of this behavior could be somewhat mitigated if numpy could provide an optimized general inner product (along the line of the Wolfram Language's `Inner`). But note that numpy.linalg.matrix_power is also lost (e.g., the derived graphs that allow an intuitive representation of transitive closure). fwiw, Alan From m.h.vankerkwijk at gmail.com Wed Jun 28 10:48:29 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 28 Jun 2017 10:48:29 -0400 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: My two ?: keep things as they are. There is just two much code that uses the C definition of bools, 0=False, 1=True. Coupled with casting every outcome that is unequal to 0 as True, * as AND, + as OR, and - as XOR makes sense (and -True would indeed be True, but I'm quite happy to have that one removed...). I lost track a little, but isn't this way also consistent with python, the one difference being that numpy does an implicit cast to bool on the result? -- Marten From nathan12343 at gmail.com Wed Jun 28 10:55:29 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 28 Jun 2017 09:55:29 -0500 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: Just as a comment: It would be really nice if NumPy could slow down the pace of deprecations, or at least make the warnings about deprecations more visible. It seems like every release breaks some subset of our test suite (we only had one or two cases of using the binary - operator on boolean arrays so it wasn't a big deal this time). For projects that don't have resources for ongoing maintenance this is a recipe for bitrot... On Wed, Jun 28, 2017 at 9:48 AM, Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > My two ?: keep things as they are. There is just two much code that > uses the C definition of bools, 0=False, 1=True. Coupled with casting > every outcome that is unequal to 0 as True, * as AND, + as OR, and - > as XOR makes sense (and -True would indeed be True, but I'm quite > happy to have that one removed...). > > I lost track a little, but isn't this way also consistent with python, > the one difference being that numpy does an implicit cast to bool on > the result? > > -- Marten > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Jun 28 16:59:51 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 28 Jun 2017 16:59:51 -0400 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: About visibility of deprecations: this is *very* tricky - if we make it more visible, every user is going to see deprecation warnings all the time, about things they can do nothing about, because they occur inside other packages. I think in the end the only choice is to have automated testing that turns warnings into errors, so that these warnings are caught early enough. That said, I've certainly taken note of this as an example of the importance of not changing things just because the current implementation is not quite logical; there should be a real benefit to the change. That said, in fairness, I'd argue at least a few of the deprecation warnings are the result of dealing with bit-rot within numpy! -- Marten From josef.pktd at gmail.com Wed Jun 28 18:06:02 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Jun 2017 18:06:02 -0400 Subject: [Numpy-discussion] Boolean binary '-' operator In-Reply-To: References: <1498409138.2263517.1020694248.793C96C2@webmail.messagingengine.com> Message-ID: On Wed, Jun 28, 2017 at 10:48 AM, Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > My two ?: keep things as they are. There is just two much code that > uses the C definition of bools, 0=False, 1=True. Coupled with casting > every outcome that is unequal to 0 as True, * as AND, + as OR, and - > as XOR makes sense (and -True would indeed be True, but I'm quite > happy to have that one removed...). > I'm also in favor of practicality beats mathematical purity. AFAIK, the hybrid behavior between boolean and the diff/sum/dot behavior works pretty well when working, e.g., with masks, as for example in masked array stats. Josef > > I lost track a little, but isn't this way also consistent with python, > the one difference being that numpy does an implicit cast to bool on > the result? > > -- Marten > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jun 29 14:07:55 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Jun 2017 12:07:55 -0600 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint Message-ID: Hi All, I will be running the NumPy sprint at Scipy 2017 and I'm trying to put together a suitable list of things to sprint on. In my experience, sprinting on NumPy is hard, enhancements generally need lengthy review and even finding and doing simple bug fixes can take time. What I have in mind at this point, apart from what might be a getting started tutorial, could mostly be classified as janitorial work. 1. Triage issues and close those that no longer apply. This is mind numbing work, but it has been almost three years since the last pass. 2. Move the contents of `numpy/doc` into `doc/source` and make them normal *.rst files. 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to regular tests. Might be tricky as it mostly checks print formatting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jun 29 14:09:36 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Jun 2017 12:09:36 -0600 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: References: Message-ID: On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > Hi All, > > I will be running the NumPy sprint at Scipy 2017 and I'm trying to put > together a suitable list of things to sprint on. In my experience, > sprinting on NumPy is hard, enhancements generally need lengthy review and > even finding and doing simple bug fixes can take time. What I have in mind > at this point, apart from what might be a getting started tutorial, could > mostly be classified as janitorial work. > > > 1. Triage issues and close those that no longer apply. This is mind > numbing work, but it has been almost three years since the last pass. > 2. Move the contents of `numpy/doc` into `doc/source` and make them > normal *.rst files. > 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to > regular tests. Might be tricky as it mostly checks print formatting. > > > Oops, sent prematurely by accident. Anyway, I'm looking for suggestions and comments as to things to do that would be useful. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Thu Jun 29 14:15:45 2017 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 29 Jun 2017 11:15:45 -0700 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: References: Message-ID: <1498760145.3918433.1025609912.4766E01C@webmail.messagingengine.com> On Thu, Jun 29, 2017, at 11:09, Charles R Harris wrote: > > On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris > wrote:>> I will be running the NumPy sprint at Scipy 2017 and I'm trying to >> put together a suitable list of things to sprint on. In my >> experience, sprinting on NumPy is hard, enhancements generally >> need lengthy review and even finding and doing simple bug fixes >> can take time. What I have in mind at this point, apart from what >> might be a getting started tutorial, could mostly be classified as >> janitorial work. Here's a random idea: how about building a NumPy gallery? scikit- {image,learn} has it, and while those projects may have more visual datasets, I can imagine something along the lines of Nicolas Rougier's beautiful book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/ St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jun 29 14:45:33 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Jun 2017 12:45:33 -0600 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: <1498760145.3918433.1025609912.4766E01C@webmail.messagingengine.com> References: <1498760145.3918433.1025609912.4766E01C@webmail.messagingengine.com> Message-ID: On Thu, Jun 29, 2017 at 12:15 PM, Stefan van der Walt wrote: > On Thu, Jun 29, 2017, at 11:09, Charles R Harris wrote: > > > On Thu, Jun 29, 2017 at 12:07 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > > I will be running the NumPy sprint at Scipy 2017 and I'm trying to put > together a suitable list of things to sprint on. In my experience, > sprinting on NumPy is hard, enhancements generally need lengthy review and > even finding and doing simple bug fixes can take time. What I have in mind > at this point, apart from what might be a getting started tutorial, could > mostly be classified as janitorial work. > > > Here's a random idea: how about building a NumPy gallery? > scikit-{image,learn} has it, and while those projects may have more visual > datasets, I can imagine something along the lines of Nicolas Rougier's > beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to-numpy/ > So that would be added in the numpy /numpy.org repo? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Jun 29 14:50:45 2017 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 29 Jun 2017 20:50:45 +0200 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: References: <1498760145.3918433.1025609912.4766E01C@webmail.messagingengine.com> Message-ID: <7576d1af-b1f2-341c-de6f-69b664f88b79@iki.fi> Charles R Harris kirjoitti 29.06.2017 klo 20:45: > Here's a random idea: how about building a NumPy gallery? > scikit-{image,learn} has it, and while those projects may have more > visual datasets, I can imagine something along the lines of Nicolas > Rougier's beautiful book: > > http://www.labri.fr/perso/nrougier/from-python-to-numpy/ > > > > So that would be added in the numpy > /numpy.org > repo? Or https://scipy-cookbook.readthedocs.io/ ? (maybe minus bitrot and images added :) From evgeny.burovskiy at gmail.com Thu Jun 29 15:27:54 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 29 Jun 2017 22:27:54 +0300 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: References: Message-ID: > Convert the doctest in `numpy/lib/tests/test_polynomial.py` to regular tests. Might be tricky as it mostly checks print formatting. Port scipy's refguide-check and enhance/fix/improve code examples in docstrings? Also somewhat janitorial though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Thu Jun 29 15:38:26 2017 From: allanhaldane at gmail.com (Allan Haldane) Date: Thu, 29 Jun 2017 15:38:26 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 Message-ID: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Hello all, There are various updates to array printing in preparation for numpy 1.14. See https://github.com/numpy/numpy/pull/9139/ Some are quite likely to break other projects' doc-tests which expect a particular str or repr of arrays, so I'd like to warn the list in case anyone has opinions. The current proposed changes, from most to least painful by my reckoning, are: 1. For float arrays, an extra space previously used for the sign position will now be omitted in many cases. Eg, `repr(arange(4.))` will now return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. 2. The printing of 0d arrays is overhauled. This is a bit finicky to describe, please see the release note in the PR. As an example of the effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like "array('2005-04-04', dtype='datetime64[D]')" instead of "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". 3. User-defined dtypes which did not properly implement their `repr` (and `str`) should do so now. Otherwise it now falls back to `object.__repr__`, which will return something ugly like ``. (Previously you could depend on only implementing the `item` method and the repr of that would be printed. But no longer, because this risks infinite recursions.). 4. Bool arrays of size 1 with a 'True' value will now omit a space, so that `repr(array([True]))` is now 'array([True])' instead of 'array([ True])'. Allan From m.h.vankerkwijk at gmail.com Thu Jun 29 16:41:55 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Thu, 29 Jun 2017 16:41:55 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Message-ID: To add to Allan's message: point (2), the printing of 0-d arrays, is the one that is the most important in the sense that it rectifies a really strange situation, where the printing cannot be logically controlled by the same mechanism that controls >=1-d arrays (see PR). While point 3 can also be considered a bug fix, 1 & 4 are at some level matters of taste; my own reason for supporting their implementation now is that the 0-d arrays already forces me (or, specifically, astropy) to rewrite quite a few doctests, and I'd rather have everything in one go -- in this respect, it is a pity that this is separate from the earlier change in printing for structured arrays (which was also much for the better, but broke a lot of doctests). -- Marten On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane wrote: > Hello all, > > There are various updates to array printing in preparation for numpy > 1.14. See https://github.com/numpy/numpy/pull/9139/ > > Some are quite likely to break other projects' doc-tests which expect a > particular str or repr of arrays, so I'd like to warn the list in case > anyone has opinions. > > The current proposed changes, from most to least painful by my > reckoning, are: > > 1. > For float arrays, an extra space previously used for the sign position > will now be omitted in many cases. Eg, `repr(arange(4.))` will now > return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. > > 2. > The printing of 0d arrays is overhauled. This is a bit finicky to > describe, please see the release note in the PR. As an example of the > effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` > instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like > "array('2005-04-04', dtype='datetime64[D]')" instead of > "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > > 3. > User-defined dtypes which did not properly implement their `repr` (and > `str`) should do so now. Otherwise it now falls back to > `object.__repr__`, which will return something ugly like > ``. (Previously you could depend on > only implementing the `item` method and the repr of that would be > printed. But no longer, because this risks infinite recursions.). > > 4. > Bool arrays of size 1 with a 'True' value will now omit a space, so that > `repr(array([True]))` is now 'array([True])' instead of > 'array([ True])'. > > Allan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From mikhailwas at gmail.com Thu Jun 29 20:16:04 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 30 Jun 2017 02:16:04 +0200 Subject: [Numpy-discussion] Array blitting (pasting one array into another) Message-ID: Hello all I often need to copy one array into another array, given an offset. This is how the "blit" function can be understood, i.e. in every graphical lib there is such a function. The common definition is like: blit ( dest, src, offset ): where dest is destination array, src is source array and offset is coordinates in destination where the src should pe blitted. Main feature of such function is that it never gives an error, so if the source does not fit into the destination array, it is simply trimmed. And respectively if there is no intersection area then nothing happens. Hope this is clear. So to make it work with Numpy arrays one need to calculate the slices before copying the data. I cannot find any Numpy or Python method to help with that so probably it does not exist yet. If so, my proposal is to add a Numpy method which helps with that. Namely the proposal will be to add a method which returns the slices for the intersection areas of two arbitrary arrays, given an offset, so then one can "blit" the array into another with simple assignment =. Here is a Python function I use for 2d arrays now: def interslice ( dest, src, offset ): y,x = offset H,W = dest.shape h,w = src.shape dest_starty = max (y,0) dest_endy = min (y+h,H) dest_startx = max (x,0) dest_endx = min (x+w,W) src_starty = 0 src_endy = h if y<0: src_starty = -y by = y+h - H # Y bleed if by>0: src_endy = h - by src_startx = 0 src_endx = w if x<0: src_startx = -x bx = x+w - W # X bleed if bx>0: src_endx = w - bx dest_sliceY = slice(dest_starty,dest_endy) dest_sliceX = slice(dest_startx,dest_endx) src_sliceY = slice(src_starty, src_endy) src_sliceX = slice(src_startx, src_endx) if dest_endy <= dest_starty: print "No Y intersection !" dest_sliceY = ( slice(0, 0) ) src_sliceY = ( slice(0, 0) ) if dest_endx <= dest_startx: print "No X intersection !" dest_sliceX = ( slice(0, 0) ) src_sliceX = ( slice(0, 0) ) dest_slice = ( dest_sliceY, dest_sliceX ) src_slice = ( src_sliceY, src_sliceX ) return ( dest_slice, src_slice ) ------ I have intentionally made it expanded and without contractions so that it is better understandable. It returns the intersection area of two arrays given an offset. First returned tuple element is the slice for DEST array and the second element is the slice for SRC array. If there is no intersection along one of the axis at all it returns the corresponding slice as (0,0) With this helper function one can blit arrays easily e.g. example code: W = 8; H = 8 DEST = numpy.ones([H,W], dtype = "uint8") w = 4; h = 1 SRC = numpy.zeros([h,w], dtype = "uint8") SRC[:]=8 offset = (0,9) ds, ss = interslice (DEST, SRC, offset ) # blit SRC into DEST DEST[ds] = SRC[ss] So changing the offset one can observe how the SRC array is trimmed if it is crosses the DEST boundaries. I think it is very useful function in general and it has well defined behaviour. It has usage not only for graphics, but actually any data copying-pasting between arrays. So I am looking forward to comments on this proposal. Mikhail From jfoxrabinovitz at gmail.com Thu Jun 29 21:34:17 2017 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Thu, 29 Jun 2017 21:34:17 -0400 Subject: [Numpy-discussion] Array blitting (pasting one array into another) In-Reply-To: References: Message-ID: This is a useful idea certainly. I would recommended extending it to an arbitrary number of axes. You could either raise an error if the ndim of the two arrays are unequal, or allow a broadcast of a lesser ndimmed src array. - Joe On Jun 29, 2017 20:17, "Mikhail V" wrote: > Hello all > > I often need to copy one array into another array, given an offset. > This is how the "blit" function can be understood, i.e. in > every graphical lib there is such a function. > The common definition is like: > blit ( dest, src, offset ): > where dest is destination array, src is source array and offset is > coordinates in destination where the src should pe blitted. > Main feature of such function is that it never gives an error, > so if the source does not fit into the destination array, it is simply > trimmed. > And respectively if there is no intersection area then nothing happens. > > Hope this is clear. > So to make it work with Numpy arrays one need to calculate the > slices before copying the data. > I cannot find any Numpy or Python method to help with that so probably > it does not exist yet. > If so, my proposal is to add a Numpy method which helps with that. > Namely the proposal will be to add a method which returns > the slices for the intersection areas of two arbitrary arrays, given an > offset, > so then one can "blit" the array into another with simple assignment =. > > Here is a Python function I use for 2d arrays now: > > def interslice ( dest, src, offset ): > y,x = offset > H,W = dest.shape > h,w = src.shape > > dest_starty = max (y,0) > dest_endy = min (y+h,H) > dest_startx = max (x,0) > dest_endx = min (x+w,W) > > src_starty = 0 > src_endy = h > if y<0: src_starty = -y > by = y+h - H # Y bleed > if by>0: src_endy = h - by > > src_startx = 0 > src_endx = w > if x<0: src_startx = -x > bx = x+w - W # X bleed > if bx>0: src_endx = w - bx > > dest_sliceY = slice(dest_starty,dest_endy) > dest_sliceX = slice(dest_startx,dest_endx) > src_sliceY = slice(src_starty, src_endy) > src_sliceX = slice(src_startx, src_endx) > if dest_endy <= dest_starty: > print "No Y intersection !" > dest_sliceY = ( slice(0, 0) ) > src_sliceY = ( slice(0, 0) ) > if dest_endx <= dest_startx: > print "No X intersection !" > dest_sliceX = ( slice(0, 0) ) > src_sliceX = ( slice(0, 0) ) > dest_slice = ( dest_sliceY, dest_sliceX ) > src_slice = ( src_sliceY, src_sliceX ) > return ( dest_slice, src_slice ) > > > ------ > > I have intentionally made it expanded and without contractions > so that it is better understandable. > It returns the intersection area of two arrays given an offset. > First returned tuple element is the slice for DEST array and the > second element is the slice for SRC array. > If there is no intersection along one of the axis at all > it returns the corresponding slice as (0,0) > > With this helper function one can blit arrays easily e.g. example code: > > W = 8; H = 8 > DEST = numpy.ones([H,W], dtype = "uint8") > w = 4; h = 1 > SRC = numpy.zeros([h,w], dtype = "uint8") > SRC[:]=8 > offset = (0,9) > ds, ss = interslice (DEST, SRC, offset ) > > # blit SRC into DEST > DEST[ds] = SRC[ss] > > So changing the offset one can observe how the > SRC array is trimmed if it is crosses the DEST boundaries. > I think it is very useful function in general and it has > well defined behaviour. It has usage not only for graphics, > but actually any data copying-pasting between arrays. > > So I am looking forward to comments on this proposal. > > > Mikhail > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jun 30 02:04:27 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 30 Jun 2017 08:04:27 +0200 Subject: [Numpy-discussion] Array blitting (pasting one array into another) In-Reply-To: References: Message-ID: <1498802667.31730.1.camel@sipsolutions.net> On Fri, 2017-06-30 at 02:16 +0200, Mikhail V wrote: > Hello all > > I often need to copy one array into another array, given an offset. > This is how the "blit" function can be understood, i.e. in > every graphical lib there is such a function. > The common definition is like: > blit ( dest, src, offset ): > where dest is destination array, src is source array and offset is > coordinates in destination where the src should pe blitted. > Main feature of such function is that it never gives an error, > so if the source does not fit into the destination array, it is > simply trimmed. > And respectively if there is no intersection area then nothing > happens. > > Hope this is clear. > So to make it work with Numpy arrays one need to calculate the > slices before copying the data. > I cannot find any Numpy or Python method to help with that so > probably > it does not exist yet. > If so, my proposal is to add a Numpy method which helps with that. > Namely the proposal will be to add a method which returns > the slices for the intersection areas of two arbitrary arrays, given > an offset, > so then one can "blit" the array into another with simple assignment > =. > > Here is a Python function I use for 2d arrays now: > > def interslice ( dest, src, offset ): > ????y,x = offset > ????H,W = dest.shape > ????h,w = src.shape > > ????dest_starty = max (y,0) > ????dest_endy = min (y+h,H) > ????dest_startx = max (x,0) > ????dest_endx = min (x+w,W) > > ????src_starty = 0 > ????src_endy = h > ????if y<0:?????src_starty = -y > ????by = y+h - H?????????????# Y bleed > ????if by>0: src_endy = h - by > > ????src_startx = 0 > ????src_endx = w > ????if x<0:??src_startx = -x > ????bx = x+w - W?????????????# X bleed > ????if bx>0:??src_endx = w - bx > > ????dest_sliceY =??slice(dest_starty,dest_endy) > ????dest_sliceX =??slice(dest_startx,dest_endx) > ????src_sliceY = slice(src_starty, src_endy) > ????src_sliceX = slice(src_startx, src_endx) > ????if dest_endy <= dest_starty: > ????????print "No Y intersection !" > ????????dest_sliceY = ( slice(0, 0) ) > ????????src_sliceY = ( slice(0, 0) ) > ????if dest_endx <= dest_startx: > ????????print "No X intersection !" > ????????dest_sliceX = ( slice(0, 0) ) > ????????src_sliceX = ( slice(0, 0) ) > ????dest_slice = ( dest_sliceY, dest_sliceX ) > ????src_slice = ( src_sliceY, src_sliceX ) > ????return ( dest_slice, src_slice ) > > > ------ > > I have intentionally made it expanded and without contractions > so that it is better understandable. > It returns the intersection area of two arrays given an offset. > First returned tuple element is the slice for DEST array and the > second element is the slice for SRC array. > If there is no intersection along one of the axis at all > it returns the corresponding slice as (0,0) > > With this helper function one can blit arrays easily e.g. example > code: > > W = 8; H = 8 > DEST = numpy.ones([H,W], dtype = "uint8") > w = 4; h = 1 > SRC = numpy.zeros([h,w], dtype = "uint8") > SRC[:]=8 > offset = (0,9) > ds, ss = interslice (DEST, SRC, offset ) > > # blit SRC into DEST > DEST[ds] = SRC[ss] > > So changing the offset one can observe how the > SRC array is trimmed if it is crosses the DEST boundaries. > I think it is very useful function in general and it has > well defined behaviour. It has usage not only for graphics, > but actually any data copying-pasting between arrays. > > So I am looking forward to comments on this proposal. > First, the slice object provides some help: ``` In [8]: s = slice(1, 40, 2) In [9]: s.indices(20) # length of dimension Out[9]: (1, 20, 2) # and the 40 becomes 20 ``` Second, there is almost no overhead of creating a view, so just create the views first (it may well be faster). Then use the result to see how large they actually are and index those (a second time) instead of creating new slice objects. - Sebastian > > Mikhail > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From jni.soma at gmail.com Fri Jun 30 03:55:52 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Fri, 30 Jun 2017 17:55:52 +1000 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Message-ID: To reiterate my point on a previous thread, I don't think this should happen until NumPy 2.0. This *will* break a massive number of doctests, and what's worse, it will do so in a way that makes it difficult to support doctesting for both 1.13 and 1.14. I don't see a big enough benefit to these changes to justify breaking everyone's tests before an API-breaking version bump. On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk , wrote: > To add to Allan's message: point (2), the printing of 0-d arrays, is > the one that is the most important in the sense that it rectifies a > really strange situation, where the printing cannot be logically > controlled by the same mechanism that controls >=1-d arrays (see PR). > > While point 3 can also be considered a bug fix, 1 & 4 are at some > level matters of taste; my own reason for supporting their > implementation now is that the 0-d arrays already forces me (or, > specifically, astropy) to rewrite quite a few doctests, and I'd rather > have everything in one go -- in this respect, it is a pity that this > is separate from the earlier change in printing for structured arrays > (which was also much for the better, but broke a lot of doctests). > > -- Marten > > > > On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane wrote: > > Hello all, > > > > There are various updates to array printing in preparation for numpy > > 1.14. See https://github.com/numpy/numpy/pull/9139/ > > > > Some are quite likely to break other projects' doc-tests which expect a > > particular str or repr of arrays, so I'd like to warn the list in case > > anyone has opinions. > > > > The current proposed changes, from most to least painful by my > > reckoning, are: > > > > 1. > > For float arrays, an extra space previously used for the sign position > > will now be omitted in many cases. Eg, `repr(arange(4.))` will now > > return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. > > > > 2. > > The printing of 0d arrays is overhauled. This is a bit finicky to > > describe, please see the release note in the PR. As an example of the > > effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` > > instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like > > "array('2005-04-04', dtype='datetime64[D]')" instead of > > "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > > > > 3. > > User-defined dtypes which did not properly implement their `repr` (and > > `str`) should do so now. Otherwise it now falls back to > > `object.__repr__`, which will return something ugly like > > ``. (Previously you could depend on > > only implementing the `item` method and the repr of that would be > > printed. But no longer, because this risks infinite recursions.). > > > > 4. > > Bool arrays of size 1 with a 'True' value will now omit a space, so that > > `repr(array([True]))` is now 'array([True])' instead of > > 'array([ True])'. > > > > Allan > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jun 30 04:11:40 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 30 Jun 2017 10:11:40 +0200 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Message-ID: <1498810300.31730.4.camel@sipsolutions.net> On Fri, 2017-06-30 at 17:55 +1000, Juan Nunez-Iglesias wrote: > To reiterate my point on a previous thread, I don't think this should > happen until NumPy 2.0. This *will* break a massive number of > doctests, and what's worse, it will do so in a way that makes it > difficult to support doctesting for both 1.13 and 1.14. I don't see a > big enough benefit to these changes to justify breaking everyone's > tests before an API-breaking version bump. > Just so we are on the same page, nobody is planning a NumPy 2.0, so insisting on not changing anything until a possible NumPy 2.0 is almost like saying it should never happen. Of course we could enmass deprecations and at some point do many at once and call it 2.0, but I am not sure that helps anyone, when compared to saying that we do deprecations for 1-2 years at least, and longer if someone complains. The question is, do you really see a big advantage in fixing a gazillion tests at once over doing a small part of the fixes one after another? The "big step" thing did not work too well for Python 3.... - Sebastian > On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk > , wrote: > > To add to Allan's message: point (2), the printing of 0-d arrays, > > is > > the one that is the most important in the sense that it rectifies a > > really strange situation, where the printing cannot be logically > > controlled by the same mechanism that controls >=1-d arrays (see > > PR). > > > > While point 3 can also be considered a bug fix, 1 & 4 are at some > > level matters of taste; my own reason for supporting their > > implementation now is that the 0-d arrays already forces me (or, > > specifically, astropy) to rewrite quite a few doctests, and I'd > > rather > > have everything in one go -- in this respect, it is a pity that > > this > > is separate from the earlier change in printing for structured > > arrays > > (which was also much for the better, but broke a lot of doctests). > > > > -- Marten > > > > > > > > On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane > com> wrote: > > > Hello all, > > > > > > There are various updates to array printing in preparation for > > > numpy > > > 1.14. See https://github.com/numpy/numpy/pull/9139/ > > > > > > Some are quite likely to break other projects' doc-tests which > > > expect a > > > particular str or repr of arrays, so I'd like to warn the list in > > > case > > > anyone has opinions. > > > > > > The current proposed changes, from most to least painful by my > > > reckoning, are: > > > > > > 1. > > > For float arrays, an extra space previously used for the sign > > > position > > > will now be omitted in many cases. Eg, `repr(arange(4.))` will > > > now > > > return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., > > > 3.])'. > > > > > > 2. > > > The printing of 0d arrays is overhauled. This is a bit finicky to > > > describe, please see the release note in the PR. As an example of > > > the > > > effect of this, the `repr(np.array(0.))` now prints as > > > 'array(0.)` > > > instead of 'array(0.0)'. Also the repr of 0d datetime arrays is > > > now like > > > "array('2005-04-04', dtype='datetime64[D]')" instead of > > > "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > > > > > > 3. > > > User-defined dtypes which did not properly implement their `repr` > > > (and > > > `str`) should do so now. Otherwise it now falls back to > > > `object.__repr__`, which will return something ugly like > > > ``. (Previously you could depend > > > on > > > only implementing the `item` method and the repr of that would be > > > printed. But no longer, because this risks infinite recursions.). > > > > > > 4. > > > Bool arrays of size 1 with a 'True' value will now omit a space, > > > so that > > > `repr(array([True]))` is now 'array([True])' instead of > > > 'array([ True])'. > > > > > > Allan > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > ?_______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From cournape at gmail.com Fri Jun 30 08:31:49 2017 From: cournape at gmail.com (David Cournapeau) Date: Fri, 30 Jun 2017 13:31:49 +0100 Subject: [Numpy-discussion] Scipy 2017 NumPy sprint In-Reply-To: References: Message-ID: Will it be possible for people not at SciPy to participate ? The timing looks I could finally spend some time on numpy again this year David On Thu, Jun 29, 2017 at 7:07 PM, Charles R Harris wrote: > Hi All, > > I will be running the NumPy sprint at Scipy 2017 and I'm trying to put > together a suitable list of things to sprint on. In my experience, > sprinting on NumPy is hard, enhancements generally need lengthy review and > even finding and doing simple bug fixes can take time. What I have in mind > at this point, apart from what might be a getting started tutorial, could > mostly be classified as janitorial work. > > > 1. Triage issues and close those that no longer apply. This is mind > numbing work, but it has been almost three years since the last pass. > 2. Move the contents of `numpy/doc` into `doc/source` and make them > normal *.rst files. > 3. Convert the doctest in `numpy/lib/tests/test_polynomial.py` to > regular tests. Might be tricky as it mostly checks print formatting. > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Fri Jun 30 09:17:20 2017 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 30 Jun 2017 15:17:20 +0200 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Message-ID: <20170630131720.GE4115401@phare.normalesup.org> Indeed, for scikit-learn, this would be a major problem. Ga?l On Fri, Jun 30, 2017 at 05:55:52PM +1000, Juan Nunez-Iglesias wrote: > To reiterate my point on a previous thread, I don't think this should happen > until NumPy 2.0. This *will* break a massive number of doctests, and what's > worse, it will do so in a way that makes it difficult to support doctesting for > both 1.13 and 1.14. I don't see a big enough benefit to these changes to > justify breaking everyone's tests before an API-breaking version bump. > On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk , > wrote: > To add to Allan's message: point (2), the printing of 0-d arrays, is > the one that is the most important in the sense that it rectifies a > really strange situation, where the printing cannot be logically > controlled by the same mechanism that controls >=1-d arrays (see PR). > While point 3 can also be considered a bug fix, 1 & 4 are at some > level matters of taste; my own reason for supporting their > implementation now is that the 0-d arrays already forces me (or, > specifically, astropy) to rewrite quite a few doctests, and I'd rather > have everything in one go -- in this respect, it is a pity that this > is separate from the earlier change in printing for structured arrays > (which was also much for the better, but broke a lot of doctests). > -- Marten > On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane > wrote: > Hello all, > There are various updates to array printing in preparation for numpy > 1.14. See https://github.com/numpy/numpy/pull/9139/ > Some are quite likely to break other projects' doc-tests which expect a > particular str or repr of arrays, so I'd like to warn the list in case > anyone has opinions. > The current proposed changes, from most to least painful by my > reckoning, are: > 1. > For float arrays, an extra space previously used for the sign position > will now be omitted in many cases. Eg, `repr(arange(4.))` will now > return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. > 2. > The printing of 0d arrays is overhauled. This is a bit finicky to > describe, please see the release note in the PR. As an example of the > effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` > instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now > like > "array('2005-04-04', dtype='datetime64[D]')" instead of > "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > 3. > User-defined dtypes which did not properly implement their `repr` (and > `str`) should do so now. Otherwise it now falls back to > `object.__repr__`, which will return something ugly like > ``. (Previously you could depend on > only implementing the `item` method and the repr of that would be > printed. But no longer, because this risks infinite recursions.). > 4. > Bool arrays of size 1 with a 'True' value will now omit a space, so > that > `repr(array([True]))` is now 'array([True])' instead of > 'array([ True])'. > Allan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -- Gael Varoquaux Researcher, INRIA Parietal NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From allanhaldane at gmail.com Fri Jun 30 11:11:47 2017 From: allanhaldane at gmail.com (Allan Haldane) Date: Fri, 30 Jun 2017 11:11:47 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> Message-ID: <272426de-0c9c-b1d1-c89b-f365a94b055d@gmail.com> On 06/30/2017 03:55 AM, Juan Nunez-Iglesias wrote: > To reiterate my point on a previous thread, I don't think this should > happen until NumPy 2.0. This *will* break a massive number of doctests, > and what's worse, it will do so in a way that makes it difficult to > support doctesting for both 1.13 and 1.14. I don't see a big enough > benefit to these changes to justify breaking everyone's tests before an > API-breaking version bump. I am still on the fence about exactly how annoying this change would be, and it is is good to hear whether this affects you and how badly. Yes, someone would have to spend an hour removing a hundred spaces in doctests, and the 1.13 to 1.14 period is trickier (but virtualenv helps). But none of your end users are going to have their scripts break, there are no new warnings or exceptions. A followup questions is, to what degree can we compromise? Would it be acceptable to skip the big change #1, but keep the other 3 changes? I expect they affect far fewer doctests. Or, for instance, I could scale back #1 so it only affects size-1 (or perhaps, only size-0) arrays. What amount of change would be OK, and how is changing a small number of doctests different from changing more? Also, let me clarify the motivations for the changes. As Marten noted, change #2 is what motivated all the other changes. Currently 0d arrays print in their own special way which was making it very hard to implement fixes to voidtype str/repr, and the datetime and other 0d reprs are weird. The fix is to make 0d arrays print using the same code-path as higher-d ndarrays, but then we ended up with reprs like "array( 1.)" because of the space for the sign position. So I removed the space from the sign position for all float arrays. But as I noted I probably could remove it for only size-1 or 0d arrays and still fix my problem, even though I think it might be pretty hacky to implement in the numpy code. Allan > > On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk > , wrote: >> To add to Allan's message: point (2), the printing of 0-d arrays, is >> the one that is the most important in the sense that it rectifies a >> really strange situation, where the printing cannot be logically >> controlled by the same mechanism that controls >=1-d arrays (see PR). >> >> While point 3 can also be considered a bug fix, 1 & 4 are at some >> level matters of taste; my own reason for supporting their >> implementation now is that the 0-d arrays already forces me (or, >> specifically, astropy) to rewrite quite a few doctests, and I'd rather >> have everything in one go -- in this respect, it is a pity that this >> is separate from the earlier change in printing for structured arrays >> (which was also much for the better, but broke a lot of doctests). >> >> -- Marten >> >> >> >> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane >> wrote: >>> Hello all, >>> >>> There are various updates to array printing in preparation for numpy >>> 1.14. See https://github.com/numpy/numpy/pull/9139/ >>> >>> Some are quite likely to break other projects' doc-tests which expect a >>> particular str or repr of arrays, so I'd like to warn the list in case >>> anyone has opinions. >>> >>> The current proposed changes, from most to least painful by my >>> reckoning, are: >>> >>> 1. >>> For float arrays, an extra space previously used for the sign position >>> will now be omitted in many cases. Eg, `repr(arange(4.))` will now >>> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. >>> >>> 2. >>> The printing of 0d arrays is overhauled. This is a bit finicky to >>> describe, please see the release note in the PR. As an example of the >>> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` >>> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like >>> "array('2005-04-04', dtype='datetime64[D]')" instead of >>> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". >>> >>> 3. >>> User-defined dtypes which did not properly implement their `repr` (and >>> `str`) should do so now. Otherwise it now falls back to >>> `object.__repr__`, which will return something ugly like >>> ``. (Previously you could depend on >>> only implementing the `item` method and the repr of that would be >>> printed. But no longer, because this risks infinite recursions.). >>> >>> 4. >>> Bool arrays of size 1 with a 'True' value will now omit a space, so that >>> `repr(array([True]))` is now 'array([True])' instead of >>> 'array([ True])'. >>> >>> Allan >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From perimosocordiae at gmail.com Fri Jun 30 15:04:37 2017 From: perimosocordiae at gmail.com (CJ Carey) Date: Fri, 30 Jun 2017 15:04:37 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <272426de-0c9c-b1d1-c89b-f365a94b055d@gmail.com> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <272426de-0c9c-b1d1-c89b-f365a94b055d@gmail.com> Message-ID: Is it feasible/desirable to provide a doctest runner that ignores whitespace? That would allow downstream projects to fix their doctests on 1.14+ with a one-line change, without breaking tests on 1.13. On Fri, Jun 30, 2017 at 11:11 AM, Allan Haldane wrote: > On 06/30/2017 03:55 AM, Juan Nunez-Iglesias wrote: > >> To reiterate my point on a previous thread, I don't think this should >> happen until NumPy 2.0. This *will* break a massive number of doctests, and >> what's worse, it will do so in a way that makes it difficult to support >> doctesting for both 1.13 and 1.14. I don't see a big enough benefit to >> these changes to justify breaking everyone's tests before an API-breaking >> version bump. >> > > I am still on the fence about exactly how annoying this change would be, > and it is is good to hear whether this affects you and how badly. > > Yes, someone would have to spend an hour removing a hundred spaces in > doctests, and the 1.13 to 1.14 period is trickier (but virtualenv helps). > But none of your end users are going to have their scripts break, there are > no new warnings or exceptions. > > A followup questions is, to what degree can we compromise? Would it be > acceptable to skip the big change #1, but keep the other 3 changes? I > expect they affect far fewer doctests. Or, for instance, I could scale back > #1 so it only affects size-1 (or perhaps, only size-0) arrays. What amount > of change would be OK, and how is changing a small number of doctests > different from changing more? > > Also, let me clarify the motivations for the changes. As Marten noted, > change #2 is what motivated all the other changes. Currently 0d arrays > print in their own special way which was making it very hard to implement > fixes to voidtype str/repr, and the datetime and other 0d reprs are weird. > The fix is to make 0d arrays print using the same code-path as higher-d > ndarrays, but then we ended up with reprs like "array( 1.)" because of the > space for the sign position. So I removed the space from the sign position > for all float arrays. But as I noted I probably could remove it for only > size-1 or 0d arrays and still fix my problem, even though I think it might > be pretty hacky to implement in the numpy code. > > Allan > > > > > >> On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk < >> m.h.vankerkwijk at gmail.com>, wrote: >> >>> To add to Allan's message: point (2), the printing of 0-d arrays, is >>> the one that is the most important in the sense that it rectifies a >>> really strange situation, where the printing cannot be logically >>> controlled by the same mechanism that controls >=1-d arrays (see PR). >>> >>> While point 3 can also be considered a bug fix, 1 & 4 are at some >>> level matters of taste; my own reason for supporting their >>> implementation now is that the 0-d arrays already forces me (or, >>> specifically, astropy) to rewrite quite a few doctests, and I'd rather >>> have everything in one go -- in this respect, it is a pity that this >>> is separate from the earlier change in printing for structured arrays >>> (which was also much for the better, but broke a lot of doctests). >>> >>> -- Marten >>> >>> >>> >>> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane >>> wrote: >>> >>>> Hello all, >>>> >>>> There are various updates to array printing in preparation for numpy >>>> 1.14. See https://github.com/numpy/numpy/pull/9139/ >>>> >>>> Some are quite likely to break other projects' doc-tests which expect a >>>> particular str or repr of arrays, so I'd like to warn the list in case >>>> anyone has opinions. >>>> >>>> The current proposed changes, from most to least painful by my >>>> reckoning, are: >>>> >>>> 1. >>>> For float arrays, an extra space previously used for the sign position >>>> will now be omitted in many cases. Eg, `repr(arange(4.))` will now >>>> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. >>>> >>>> 2. >>>> The printing of 0d arrays is overhauled. This is a bit finicky to >>>> describe, please see the release note in the PR. As an example of the >>>> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` >>>> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like >>>> "array('2005-04-04', dtype='datetime64[D]')" instead of >>>> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". >>>> >>>> 3. >>>> User-defined dtypes which did not properly implement their `repr` (and >>>> `str`) should do so now. Otherwise it now falls back to >>>> `object.__repr__`, which will return something ugly like >>>> ``. (Previously you could depend on >>>> only implementing the `item` method and the repr of that would be >>>> printed. But no longer, because this risks infinite recursions.). >>>> >>>> 4. >>>> Bool arrays of size 1 with a 'True' value will now omit a space, so that >>>> `repr(array([True]))` is now 'array([True])' instead of >>>> 'array([ True])'. >>>> >>>> Allan >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Fri Jun 30 15:36:59 2017 From: allanhaldane at gmail.com (Allan Haldane) Date: Fri, 30 Jun 2017 15:36:59 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <272426de-0c9c-b1d1-c89b-f365a94b055d@gmail.com> Message-ID: <1fb812cb-125e-2626-2b68-f5d902df0902@gmail.com> On 06/30/2017 03:04 PM, CJ Carey wrote: > Is it feasible/desirable to provide a doctest runner that ignores > whitespace? That would allow downstream projects to fix their doctests on > 1.14+ with a one-line change, without breaking tests on 1.13. Good idea. I have already implemented this actually, see the updated PR. https://github.com/numpy/numpy/pull/9139/ Whether or not the sign position is padded can now be controlled by setting >>> np.set_printoptions(pad_sign=True) >>> np.set_printoptions(pad_sign=False) When pad_sign is True, it gives the old behavior, except for size-1 arrays where it still omits the sign position. (Maybe I should limit it even more, to 0d arrays?) When pad_sign is False (currently default in the PR), it removes the sign padding everywhere if possible. Allan > On Fri, Jun 30, 2017 at 11:11 AM, Allan Haldane > wrote: > >> On 06/30/2017 03:55 AM, Juan Nunez-Iglesias wrote: >> >>> To reiterate my point on a previous thread, I don't think this should >>> happen until NumPy 2.0. This *will* break a massive number of doctests, and >>> what's worse, it will do so in a way that makes it difficult to support >>> doctesting for both 1.13 and 1.14. I don't see a big enough benefit to >>> these changes to justify breaking everyone's tests before an API-breaking >>> version bump. >>> >> >> I am still on the fence about exactly how annoying this change would be, >> and it is is good to hear whether this affects you and how badly. >> >> Yes, someone would have to spend an hour removing a hundred spaces in >> doctests, and the 1.13 to 1.14 period is trickier (but virtualenv helps). >> But none of your end users are going to have their scripts break, there are >> no new warnings or exceptions. >> >> A followup questions is, to what degree can we compromise? Would it be >> acceptable to skip the big change #1, but keep the other 3 changes? I >> expect they affect far fewer doctests. Or, for instance, I could scale back >> #1 so it only affects size-1 (or perhaps, only size-0) arrays. What amount >> of change would be OK, and how is changing a small number of doctests >> different from changing more? >> >> Also, let me clarify the motivations for the changes. As Marten noted, >> change #2 is what motivated all the other changes. Currently 0d arrays >> print in their own special way which was making it very hard to implement >> fixes to voidtype str/repr, and the datetime and other 0d reprs are weird. >> The fix is to make 0d arrays print using the same code-path as higher-d >> ndarrays, but then we ended up with reprs like "array( 1.)" because of the >> space for the sign position. So I removed the space from the sign position >> for all float arrays. But as I noted I probably could remove it for only >> size-1 or 0d arrays and still fix my problem, even though I think it might >> be pretty hacky to implement in the numpy code. >> >> Allan >> >> >> >> >> >>> On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk < >>> m.h.vankerkwijk at gmail.com>, wrote: >>> >>>> To add to Allan's message: point (2), the printing of 0-d arrays, is >>>> the one that is the most important in the sense that it rectifies a >>>> really strange situation, where the printing cannot be logically >>>> controlled by the same mechanism that controls >=1-d arrays (see PR). >>>> >>>> While point 3 can also be considered a bug fix, 1 & 4 are at some >>>> level matters of taste; my own reason for supporting their >>>> implementation now is that the 0-d arrays already forces me (or, >>>> specifically, astropy) to rewrite quite a few doctests, and I'd rather >>>> have everything in one go -- in this respect, it is a pity that this >>>> is separate from the earlier change in printing for structured arrays >>>> (which was also much for the better, but broke a lot of doctests). >>>> >>>> -- Marten >>>> >>>> >>>> >>>> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane >>>> wrote: >>>> >>>>> Hello all, >>>>> >>>>> There are various updates to array printing in preparation for numpy >>>>> 1.14. See https://github.com/numpy/numpy/pull/9139/ >>>>> >>>>> Some are quite likely to break other projects' doc-tests which expect a >>>>> particular str or repr of arrays, so I'd like to warn the list in case >>>>> anyone has opinions. >>>>> >>>>> The current proposed changes, from most to least painful by my >>>>> reckoning, are: >>>>> >>>>> 1. >>>>> For float arrays, an extra space previously used for the sign position >>>>> will now be omitted in many cases. Eg, `repr(arange(4.))` will now >>>>> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. >>>>> >>>>> 2. >>>>> The printing of 0d arrays is overhauled. This is a bit finicky to >>>>> describe, please see the release note in the PR. As an example of the >>>>> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` >>>>> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now like >>>>> "array('2005-04-04', dtype='datetime64[D]')" instead of >>>>> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". >>>>> >>>>> 3. >>>>> User-defined dtypes which did not properly implement their `repr` (and >>>>> `str`) should do so now. Otherwise it now falls back to >>>>> `object.__repr__`, which will return something ugly like >>>>> ``. (Previously you could depend on >>>>> only implementing the `item` method and the repr of that would be >>>>> printed. But no longer, because this risks infinite recursions.). >>>>> >>>>> 4. >>>>> Bool arrays of size 1 with a 'True' value will now omit a space, so that >>>>> `repr(array([True]))` is now 'array([True])' instead of >>>>> 'array([ True])'. >>>>> >>>>> Allan >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at python.org >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From mikhailwas at gmail.com Fri Jun 30 16:08:54 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 30 Jun 2017 22:08:54 +0200 Subject: [Numpy-discussion] Array blitting (pasting one array into another) In-Reply-To: References: Message-ID: On 30 June 2017 at 03:34, Joseph Fox-Rabinovitz wrote: > This is a useful idea certainly. I would recommended extending it to an > arbitrary number of axes. You could either raise an error if the ndim of the > two arrays are unequal, or allow a broadcast of a lesser ndimmed src array. > Now I am thinking that there is probably an even better, more generalised way to provide this functionality. Say if we had a function "intersect" which would be defined as follows: intersect(A, B, offset) where A, B are vector endpoints, and the offset is the distance between their origins. So to find a needed slice I could simply pass the shapes: intersect (DEST.shape, SRC.shape, offset) Hmm. there is something to think about. Could be a better idea to propose this, since it could be used in many other sitiations, not only for finding slice intersection. Although I'll need some time to think out more examples and use cases. Mikhail > > On Jun 29, 2017 20:17, "Mikhail V" wrote: >> >> Hello all >> >> I often need to copy one array into another array, given an offset. >> This is how the "blit" function can be understood, i.e. in >> every graphical lib there is such a function. >> The common definition is like: >> blit ( dest, src, offset ): >> where dest is destination array, src is source array and offset is >> coordinates in destination where the src should pe blitted. >> Main feature of such function is that it never gives an error, >> so if the source does not fit into the destination array, it is simply >> trimmed. >> And respectively if there is no intersection area then nothing happens. >> >> Hope this is clear. >> So to make it work with Numpy arrays one need to calculate the >> slices before copying the data. >> I cannot find any Numpy or Python method to help with that so probably >> it does not exist yet. >> If so, my proposal is to add a Numpy method which helps with that. >> Namely the proposal will be to add a method which returns >> the slices for the intersection areas of two arbitrary arrays, given an >> offset, >> so then one can "blit" the array into another with simple assignment =. >> >> Here is a Python function I use for 2d arrays now: >> >> def interslice ( dest, src, offset ): >> y,x = offset >> H,W = dest.shape >> h,w = src.shape >> >> dest_starty = max (y,0) >> dest_endy = min (y+h,H) >> dest_startx = max (x,0) >> dest_endx = min (x+w,W) >> >> src_starty = 0 >> src_endy = h >> if y<0: src_starty = -y >> by = y+h - H # Y bleed >> if by>0: src_endy = h - by >> >> src_startx = 0 >> src_endx = w >> if x<0: src_startx = -x >> bx = x+w - W # X bleed >> if bx>0: src_endx = w - bx >> >> dest_sliceY = slice(dest_starty,dest_endy) >> dest_sliceX = slice(dest_startx,dest_endx) >> src_sliceY = slice(src_starty, src_endy) >> src_sliceX = slice(src_startx, src_endx) >> if dest_endy <= dest_starty: >> print "No Y intersection !" >> dest_sliceY = ( slice(0, 0) ) >> src_sliceY = ( slice(0, 0) ) >> if dest_endx <= dest_startx: >> print "No X intersection !" >> dest_sliceX = ( slice(0, 0) ) >> src_sliceX = ( slice(0, 0) ) >> dest_slice = ( dest_sliceY, dest_sliceX ) >> src_slice = ( src_sliceY, src_sliceX ) >> return ( dest_slice, src_slice ) >> >> >> ------ >> >> I have intentionally made it expanded and without contractions >> so that it is better understandable. >> It returns the intersection area of two arrays given an offset. >> First returned tuple element is the slice for DEST array and the >> second element is the slice for SRC array. >> If there is no intersection along one of the axis at all >> it returns the corresponding slice as (0,0) >> >> With this helper function one can blit arrays easily e.g. example code: >> >> W = 8; H = 8 >> DEST = numpy.ones([H,W], dtype = "uint8") >> w = 4; h = 1 >> SRC = numpy.zeros([h,w], dtype = "uint8") >> SRC[:]=8 >> offset = (0,9) >> ds, ss = interslice (DEST, SRC, offset ) >> >> # blit SRC into DEST >> DEST[ds] = SRC[ss] >> >> So changing the offset one can observe how the >> SRC array is trimmed if it is crosses the DEST boundaries. >> I think it is very useful function in general and it has >> well defined behaviour. It has usage not only for graphics, >> but actually any data copying-pasting between arrays. >> >> So I am looking forward to comments on this proposal. >> >> >> Mikhail >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From jfoxrabinovitz at gmail.com Fri Jun 30 16:14:16 2017 From: jfoxrabinovitz at gmail.com (Joseph Fox-Rabinovitz) Date: Fri, 30 Jun 2017 16:14:16 -0400 Subject: [Numpy-discussion] Array blitting (pasting one array into another) In-Reply-To: References: Message-ID: If you are serious about adding this to numpy, an even better option might be to create a pull request with the implementation and solicit comments on that. The problem lends itself to an easy solution in pure Python, so this should not be too hard to do. -Joe On Fri, Jun 30, 2017 at 4:08 PM, Mikhail V wrote: > On 30 June 2017 at 03:34, Joseph Fox-Rabinovitz > wrote: > > This is a useful idea certainly. I would recommended extending it to an > > arbitrary number of axes. You could either raise an error if the ndim of > the > > two arrays are unequal, or allow a broadcast of a lesser ndimmed src > array. > > > > > Now I am thinking that there is probably an even better, more generalised > way to provide this functionality. > Say if we had a function "intersect" which would be defined as follows: > > intersect(A, B, offset) > > where A, B are vector endpoints, and the offset is the distance > between their origins. > So to find a needed slice I could simply pass the shapes: > > intersect (DEST.shape, SRC.shape, offset) > > Hmm. there is something to think about. Could be a > better idea to propose this, since it could be used in many > other sitiations, not only for finding slice intersection. > > Although I'll need some time to think out more examples and use cases. > > Mikhail > > > > > On Jun 29, 2017 20:17, "Mikhail V" wrote: > >> > >> Hello all > >> > >> I often need to copy one array into another array, given an offset. > >> This is how the "blit" function can be understood, i.e. in > >> every graphical lib there is such a function. > >> The common definition is like: > >> blit ( dest, src, offset ): > >> where dest is destination array, src is source array and offset is > >> coordinates in destination where the src should pe blitted. > >> Main feature of such function is that it never gives an error, > >> so if the source does not fit into the destination array, it is simply > >> trimmed. > >> And respectively if there is no intersection area then nothing happens. > >> > >> Hope this is clear. > >> So to make it work with Numpy arrays one need to calculate the > >> slices before copying the data. > >> I cannot find any Numpy or Python method to help with that so probably > >> it does not exist yet. > >> If so, my proposal is to add a Numpy method which helps with that. > >> Namely the proposal will be to add a method which returns > >> the slices for the intersection areas of two arbitrary arrays, given an > >> offset, > >> so then one can "blit" the array into another with simple assignment =. > >> > >> Here is a Python function I use for 2d arrays now: > >> > >> def interslice ( dest, src, offset ): > >> y,x = offset > >> H,W = dest.shape > >> h,w = src.shape > >> > >> dest_starty = max (y,0) > >> dest_endy = min (y+h,H) > >> dest_startx = max (x,0) > >> dest_endx = min (x+w,W) > >> > >> src_starty = 0 > >> src_endy = h > >> if y<0: src_starty = -y > >> by = y+h - H # Y bleed > >> if by>0: src_endy = h - by > >> > >> src_startx = 0 > >> src_endx = w > >> if x<0: src_startx = -x > >> bx = x+w - W # X bleed > >> if bx>0: src_endx = w - bx > >> > >> dest_sliceY = slice(dest_starty,dest_endy) > >> dest_sliceX = slice(dest_startx,dest_endx) > >> src_sliceY = slice(src_starty, src_endy) > >> src_sliceX = slice(src_startx, src_endx) > >> if dest_endy <= dest_starty: > >> print "No Y intersection !" > >> dest_sliceY = ( slice(0, 0) ) > >> src_sliceY = ( slice(0, 0) ) > >> if dest_endx <= dest_startx: > >> print "No X intersection !" > >> dest_sliceX = ( slice(0, 0) ) > >> src_sliceX = ( slice(0, 0) ) > >> dest_slice = ( dest_sliceY, dest_sliceX ) > >> src_slice = ( src_sliceY, src_sliceX ) > >> return ( dest_slice, src_slice ) > >> > >> > >> ------ > >> > >> I have intentionally made it expanded and without contractions > >> so that it is better understandable. > >> It returns the intersection area of two arrays given an offset. > >> First returned tuple element is the slice for DEST array and the > >> second element is the slice for SRC array. > >> If there is no intersection along one of the axis at all > >> it returns the corresponding slice as (0,0) > >> > >> With this helper function one can blit arrays easily e.g. example code: > >> > >> W = 8; H = 8 > >> DEST = numpy.ones([H,W], dtype = "uint8") > >> w = 4; h = 1 > >> SRC = numpy.zeros([h,w], dtype = "uint8") > >> SRC[:]=8 > >> offset = (0,9) > >> ds, ss = interslice (DEST, SRC, offset ) > >> > >> # blit SRC into DEST > >> DEST[ds] = SRC[ss] > >> > >> So changing the offset one can observe how the > >> SRC array is trimmed if it is crosses the DEST boundaries. > >> I think it is very useful function in general and it has > >> well defined behaviour. It has usage not only for graphics, > >> but actually any data copying-pasting between arrays. > >> > >> So I am looking forward to comments on this proposal. > >> > >> > >> Mikhail > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Fri Jun 30 18:30:53 2017 From: allanhaldane at gmail.com (Allan Haldane) Date: Fri, 30 Jun 2017 18:30:53 -0400 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <20170630131720.GE4115401@phare.normalesup.org> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <20170630131720.GE4115401@phare.normalesup.org> Message-ID: <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> On 06/30/2017 09:17 AM, Gael Varoquaux wrote: > Indeed, for scikit-learn, this would be a major problem. > > Ga?l I just ran the scikit-learn tests. With the new behavior (removed whitespace), I do get 70 total failures: $ make test-doc Ran 39 tests in 39.503s FAILED (SKIP=3, failures=19) $ make test Ran 8122 tests in 387.650s FAILED (SKIP=58, failures=51) After setting `np.set_printoptions(pad_sign=True)` (see other email) I get only 1 failure in total, which is due to the presence of a 0d array in gaussian_process.rst. So it looks like the pad_sign option as currently implemented is good enough to avoid almost all doctest errors. Allan > On Fri, Jun 30, 2017 at 05:55:52PM +1000, Juan Nunez-Iglesias wrote: >> To reiterate my point on a previous thread, I don't think this should happen >> until NumPy 2.0. This *will* break a massive number of doctests, and what's >> worse, it will do so in a way that makes it difficult to support doctesting for >> both 1.13 and 1.14. I don't see a big enough benefit to these changes to >> justify breaking everyone's tests before an API-breaking version bump. > >> On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk , >> wrote: > >> To add to Allan's message: point (2), the printing of 0-d arrays, is >> the one that is the most important in the sense that it rectifies a >> really strange situation, where the printing cannot be logically >> controlled by the same mechanism that controls >=1-d arrays (see PR). > >> While point 3 can also be considered a bug fix, 1 & 4 are at some >> level matters of taste; my own reason for supporting their >> implementation now is that the 0-d arrays already forces me (or, >> specifically, astropy) to rewrite quite a few doctests, and I'd rather >> have everything in one go -- in this respect, it is a pity that this >> is separate from the earlier change in printing for structured arrays >> (which was also much for the better, but broke a lot of doctests). > >> -- Marten > > > >> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane >> wrote: > >> Hello all, > >> There are various updates to array printing in preparation for numpy >> 1.14. See https://github.com/numpy/numpy/pull/9139/ > >> Some are quite likely to break other projects' doc-tests which expect a >> particular str or repr of arrays, so I'd like to warn the list in case >> anyone has opinions. > >> The current proposed changes, from most to least painful by my >> reckoning, are: > >> 1. >> For float arrays, an extra space previously used for the sign position >> will now be omitted in many cases. Eg, `repr(arange(4.))` will now >> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. > >> 2. >> The printing of 0d arrays is overhauled. This is a bit finicky to >> describe, please see the release note in the PR. As an example of the >> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` >> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now >> like >> "array('2005-04-04', dtype='datetime64[D]')" instead of >> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > >> 3. >> User-defined dtypes which did not properly implement their `repr` (and >> `str`) should do so now. Otherwise it now falls back to >> `object.__repr__`, which will return something ugly like >> ``. (Previously you could depend on >> only implementing the `item` method and the repr of that would be >> printed. But no longer, because this risks infinite recursions.). > >> 4. >> Bool arrays of size 1 with a 'True' value will now omit a space, so >> that >> `repr(array([True]))` is now 'array([True])' instead of >> 'array([ True])'. > >> Allan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > From ralf.gommers at gmail.com Fri Jun 30 19:05:08 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 1 Jul 2017 11:05:08 +1200 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <272426de-0c9c-b1d1-c89b-f365a94b055d@gmail.com> Message-ID: On Sat, Jul 1, 2017 at 7:04 AM, CJ Carey wrote: > Is it feasible/desirable to provide a doctest runner that ignores > whitespace? > Yes, and yes. Due to doctest being in the stdlib that is going to take forever to have any effect though; a separate our-sane-doctest module would be the way to ship this I think. And not only whitespace, also provide sane floating point comparison behavior (AstroPy has something for that that can be reused: https://github.com/astropy/astropy/issues/6312) as well as things a bit more specific to the needs of scientific Python projects like ignoring the hashes in returned matplotlib objects. > That would allow downstream projects to fix their doctests on 1.14+ with a > one-line change, without breaking tests on 1.13. > It's worth reading https://docs.python.org/2/library/doctest.html#soapbox. At least the first 2 paragraphs; the rest is mainly an illustration of why doctest default behavior is evil ("doctest also makes an excellent tool for regression testing" - eh, no). The only valid reason nowadays to use doctests is to test that doc examples run and are correct. None of {whitespace, blank lines, small floating point differences between platforms/libs, hashes} are valid reasons to get a test failure. At the moment there's no polished alternative to using stdlib doctest, so I'm sympathetic to the argument of "this causes a lot of work". On the other hand, exact repr's are not part of the NumPy (or Python for that matter) backwards compatibility guarantees. So imho we should provide that alternative to doctest, and then no longer worry about these kinds of changes and just make them. Until we have that alternative, I think https://github.com/scipy/scipy/blob/master/tools/refguide_check.py may be useful to other projects - it checks that your examples are not broken, without doing the detailed string comparisons that are so fragile. Ralf > > On Fri, Jun 30, 2017 at 11:11 AM, Allan Haldane > wrote: > >> On 06/30/2017 03:55 AM, Juan Nunez-Iglesias wrote: >> >>> To reiterate my point on a previous thread, I don't think this should >>> happen until NumPy 2.0. This *will* break a massive number of doctests, and >>> what's worse, it will do so in a way that makes it difficult to support >>> doctesting for both 1.13 and 1.14. I don't see a big enough benefit to >>> these changes to justify breaking everyone's tests before an API-breaking >>> version bump. >>> >> >> I am still on the fence about exactly how annoying this change would be, >> and it is is good to hear whether this affects you and how badly. >> >> Yes, someone would have to spend an hour removing a hundred spaces in >> doctests, and the 1.13 to 1.14 period is trickier (but virtualenv helps). >> But none of your end users are going to have their scripts break, there are >> no new warnings or exceptions. >> >> A followup questions is, to what degree can we compromise? Would it be >> acceptable to skip the big change #1, but keep the other 3 changes? I >> expect they affect far fewer doctests. Or, for instance, I could scale back >> #1 so it only affects size-1 (or perhaps, only size-0) arrays. What amount >> of change would be OK, and how is changing a small number of doctests >> different from changing more? >> >> Also, let me clarify the motivations for the changes. As Marten noted, >> change #2 is what motivated all the other changes. Currently 0d arrays >> print in their own special way which was making it very hard to implement >> fixes to voidtype str/repr, and the datetime and other 0d reprs are weird. >> The fix is to make 0d arrays print using the same code-path as higher-d >> ndarrays, but then we ended up with reprs like "array( 1.)" because of the >> space for the sign position. So I removed the space from the sign position >> for all float arrays. But as I noted I probably could remove it for only >> size-1 or 0d arrays and still fix my problem, even though I think it might >> be pretty hacky to implement in the numpy code. >> >> Allan >> >> >> >> >> >>> On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk < >>> m.h.vankerkwijk at gmail.com>, wrote: >>> >>>> To add to Allan's message: point (2), the printing of 0-d arrays, is >>>> the one that is the most important in the sense that it rectifies a >>>> really strange situation, where the printing cannot be logically >>>> controlled by the same mechanism that controls >=1-d arrays (see PR). >>>> >>>> While point 3 can also be considered a bug fix, 1 & 4 are at some >>>> level matters of taste; my own reason for supporting their >>>> implementation now is that the 0-d arrays already forces me (or, >>>> specifically, astropy) to rewrite quite a few doctests, and I'd rather >>>> have everything in one go -- in this respect, it is a pity that this >>>> is separate from the earlier change in printing for structured arrays >>>> (which was also much for the better, but broke a lot of doctests). >>>> >>>> -- Marten >>>> >>>> >>>> >>>> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane >>>> wrote: >>>> >>>>> Hello all, >>>>> >>>>> There are various updates to array printing in preparation for numpy >>>>> 1.14. See https://github.com/numpy/numpy/pull/9139/ >>>>> >>>>> Some are quite likely to break other projects' doc-tests which expect a >>>>> particular str or repr of arrays, so I'd like to warn the list in case >>>>> anyone has opinions. >>>>> >>>>> The current proposed changes, from most to least painful by my >>>>> reckoning, are: >>>>> >>>>> 1. >>>>> For float arrays, an extra space previously used for the sign position >>>>> will now be omitted in many cases. Eg, `repr(arange(4.))` will now >>>>> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. >>>>> >>>>> 2. >>>>> The printing of 0d arrays is overhauled. This is a bit finicky to >>>>> describe, please see the release note in the PR. As an example of the >>>>> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` >>>>> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now >>>>> like >>>>> "array('2005-04-04', dtype='datetime64[D]')" instead of >>>>> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". >>>>> >>>>> 3. >>>>> User-defined dtypes which did not properly implement their `repr` (and >>>>> `str`) should do so now. Otherwise it now falls back to >>>>> `object.__repr__`, which will return something ugly like >>>>> ``. (Previously you could depend on >>>>> only implementing the `item` method and the repr of that would be >>>>> printed. But no longer, because this risks infinite recursions.). >>>>> >>>>> 4. >>>>> Bool arrays of size 1 with a 'True' value will now omit a space, so >>>>> that >>>>> `repr(array([True]))` is now 'array([True])' instead of >>>>> 'array([ True])'. >>>>> >>>>> Allan >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at python.org >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Fri Jun 30 19:47:21 2017 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 1 Jul 2017 01:47:21 +0200 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <20170630131720.GE4115401@phare.normalesup.org> <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> Message-ID: <20170630234721.GK4115401@phare.normalesup.org> One problem is that it becomes hard (impossible?) for downstream packages such as scikit-learn to doctest under multiple versions of the numpy. Past experience has shown that it could be useful. Ga?l On Fri, Jun 30, 2017 at 06:30:53PM -0400, Allan Haldane wrote: > On 06/30/2017 09:17 AM, Gael Varoquaux wrote: > > Indeed, for scikit-learn, this would be a major problem. > > Ga?l > I just ran the scikit-learn tests. > With the new behavior (removed whitespace), I do get 70 total failures: > $ make test-doc > Ran 39 tests in 39.503s > FAILED (SKIP=3, failures=19) > $ make test > Ran 8122 tests in 387.650s > FAILED (SKIP=58, failures=51) > After setting `np.set_printoptions(pad_sign=True)` (see other email) I > get only 1 failure in total, which is due to the presence of a 0d array > in gaussian_process.rst. > So it looks like the pad_sign option as currently implemented is good > enough to avoid almost all doctest errors. > Allan > > On Fri, Jun 30, 2017 at 05:55:52PM +1000, Juan Nunez-Iglesias wrote: > >> To reiterate my point on a previous thread, I don't think this should happen > >> until NumPy 2.0. This *will* break a massive number of doctests, and what's > >> worse, it will do so in a way that makes it difficult to support doctesting for > >> both 1.13 and 1.14. I don't see a big enough benefit to these changes to > >> justify breaking everyone's tests before an API-breaking version bump. > >> On 30 Jun 2017, 6:42 AM +1000, Marten van Kerkwijk , > >> wrote: > >> To add to Allan's message: point (2), the printing of 0-d arrays, is > >> the one that is the most important in the sense that it rectifies a > >> really strange situation, where the printing cannot be logically > >> controlled by the same mechanism that controls >=1-d arrays (see PR). > >> While point 3 can also be considered a bug fix, 1 & 4 are at some > >> level matters of taste; my own reason for supporting their > >> implementation now is that the 0-d arrays already forces me (or, > >> specifically, astropy) to rewrite quite a few doctests, and I'd rather > >> have everything in one go -- in this respect, it is a pity that this > >> is separate from the earlier change in printing for structured arrays > >> (which was also much for the better, but broke a lot of doctests). > >> -- Marten > >> On Thu, Jun 29, 2017 at 3:38 PM, Allan Haldane > >> wrote: > >> Hello all, > >> There are various updates to array printing in preparation for numpy > >> 1.14. See https://github.com/numpy/numpy/pull/9139/ > >> Some are quite likely to break other projects' doc-tests which expect a > >> particular str or repr of arrays, so I'd like to warn the list in case > >> anyone has opinions. > >> The current proposed changes, from most to least painful by my > >> reckoning, are: > >> 1. > >> For float arrays, an extra space previously used for the sign position > >> will now be omitted in many cases. Eg, `repr(arange(4.))` will now > >> return 'array([0., 1., 2., 3.])' instead of 'array([ 0., 1., 2., 3.])'. > >> 2. > >> The printing of 0d arrays is overhauled. This is a bit finicky to > >> describe, please see the release note in the PR. As an example of the > >> effect of this, the `repr(np.array(0.))` now prints as 'array(0.)` > >> instead of 'array(0.0)'. Also the repr of 0d datetime arrays is now > >> like > >> "array('2005-04-04', dtype='datetime64[D]')" instead of > >> "array(datetime.date(2005, 4, 4), dtype='datetime64[D]')". > >> 3. > >> User-defined dtypes which did not properly implement their `repr` (and > >> `str`) should do so now. Otherwise it now falls back to > >> `object.__repr__`, which will return something ugly like > >> ``. (Previously you could depend on > >> only implementing the `item` method and the repr of that would be > >> printed. But no longer, because this risks infinite recursions.). > >> 4. > >> Bool arrays of size 1 with a 'True' value will now omit a space, so > >> that > >> `repr(array([True]))` is now 'array([True])' instead of > >> 'array([ True])'. > >> Allan > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at python.org > >> https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -- Gael Varoquaux Researcher, INRIA Parietal NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux From robert.kern at gmail.com Fri Jun 30 19:55:16 2017 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Jun 2017 16:55:16 -0700 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <20170630234721.GK4115401@phare.normalesup.org> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <20170630131720.GE4115401@phare.normalesup.org> <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> <20170630234721.GK4115401@phare.normalesup.org> Message-ID: On Fri, Jun 30, 2017 at 4:47 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > > One problem is that it becomes hard (impossible?) for downstream packages > such as scikit-learn to doctest under multiple versions of the numpy. > Past experience has shown that it could be useful. It's not that hard: wrap the new `set_printoptions(pad=True)` in a `try:` block to catch the error under old versions. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Fri Jun 30 22:23:05 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Sat, 1 Jul 2017 12:23:05 +1000 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <20170630131720.GE4115401@phare.normalesup.org> <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> <20170630234721.GK4115401@phare.normalesup.org> Message-ID: <0ac942e0-c7e9-41f5-8dc3-3993f9ec34c4@Spark> I agree that shipping a sane/sanitising doctest runner would go 95% of the way to alleviating my concerns. Regarding 2.0, this is the whole point of semantic versioning: downstream packages can pin their dependency as 1.x and know that they - will continue to work with any updates - won?t make their users choose between new NumPy 1.x features and running their software. The Python 3.x transition was a huge fail, but the version numbering was not the problem. I do have sympathy for Ralf?s argument that "exact repr's are not part of the NumPy (or Python for that matter) backwards compatibility guarantees?. But it is such a foundational project in Scientific Python that I think extreme care is warranted, beyond any official guarantees. (Hence this thread, yes. Thank you!) Incidentally, I don?t think "array( 1.)? is such a tragic repr fail. I actually would welcome it because I?ve tried to JSON-serialise these buggers quite a few times because I didn?t realise they were 0d arrays instead of floats. So why exactly is it so bad that there is a space there? Anyway, all this is (mostly) moot if the next NumPy ships with this doctest++ thingy. That would be an enormously valuable contribution to the whole ecosystem. Thanks, Juan. On 1 Jul 2017, 9:56 AM +1000, Robert Kern , wrote: > On Fri, Jun 30, 2017 at 4:47 PM, Gael Varoquaux wrote: > > > > One problem is that it becomes hard (impossible?) for downstream packages > > such as scikit-learn to doctest under multiple versions of the numpy. > > Past experience has shown that it could be useful. > > It's not that hard: wrap the new `set_printoptions(pad=True)` in a `try:` block to catch the error under old versions. > > -- > Robert Kern > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 30 23:34:50 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 30 Jun 2017 20:34:50 -0700 Subject: [Numpy-discussion] proposed changes to array printing in 1.14 In-Reply-To: <0ac942e0-c7e9-41f5-8dc3-3993f9ec34c4@Spark> References: <5bc52c06-fbe7-2c34-8695-6cc40de38610@gmail.com> <20170630131720.GE4115401@phare.normalesup.org> <0c381b38-0182-0305-edbb-3cacfd2dc997@gmail.com> <20170630234721.GK4115401@phare.normalesup.org> <0ac942e0-c7e9-41f5-8dc3-3993f9ec34c4@Spark> Message-ID: On Fri, Jun 30, 2017 at 7:23 PM, Juan Nunez-Iglesias wrote: > I agree that shipping a sane/sanitising doctest runner would go 95% of the > way to alleviating my concerns. > > Regarding 2.0, this is the whole point of semantic versioning: downstream > packages can pin their dependency as 1.x and know that they > - will continue to work with any updates > - won?t make their users choose between new NumPy 1.x features and running > their software. Semantic versioning is somewhere between useless and harmful for non-trivial projects. It's a lovely idea, it would be lovely if it worked, but in practice it either means you make every release a major release, which doesn't help anything, or else you never make a major release until eventually everyone gets so frustrated that they fork the project or do a python 3 style break-everything major release, which is a cure that's worse than the original disease. NumPy's strategy instead is to make small, controlled, rolling breaking changes in 1.x releases. Every release breaks something for someone somewhere, but ideally only after debate and appropriate warning, and hopefully most release don't break things for *you*. Change is going to happen one way or another, and it's easier to manage a small amount of breakage every few releases than to manage a giant chunk all at once. (The latter just seems easier because it's in the future, so your brain is like "eh I'm sure I'll be fine" until you get there and realize how doomed you are.) Plus, the reality is that every numpy release ever made has accidentally broken something for someone somewhere, so instead of lying to ourselves and pretending that we can keep things perfectly backwards compatible at all times, we might as well acknowledge that and try to manage the cost of breakage and make them worthwhile. Heck, even bug fixes are frequently compatibility-breaking changes in reality, and here we are debating whether tweaking whitespace in reprs is a compatibility-breaking change. There's no line of demarcation between breaking changes and non-breaking changes, just shades of grey, and we can do better engineering if our processes acknowledge that. Another critique of semantic versioning: https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e The Google philosophy of "error budgets", which is somewhat analogous to the argument I'm making for a compatibility-breakage budget: https://www.usenix.org/node/189332 https://landing.google.com/sre/book/chapters/service-level-objectives.html#xref_risk-management_global-chubby-planned-outage -n -- Nathaniel J. Smith -- https://vorpus.org