From njs at pobox.com Wed Jul 1 00:15:28 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 30 Jun 2015 21:15:28 -0700 Subject: [Numpy-discussion] annoying Deprecation warnings about non-integers In-Reply-To: References: Message-ID: On Jun 30, 2015 9:01 PM, wrote: > > I'm trying to fix some code in statsmodels that creates Deprecation Warnings from numpy > > Most of it are quite easy to fix, mainly cases where we use floats to avoid integer division > > I have two problems > > first, I get Deprecation warnings in the test run that don't specify where they happen. > I try to find them with file searches, but I don't see a `np.ones` that might cause a problem > (needle in a haystack: Close to 4000 unittests and more than 100,000 lines of numpython) > Also, I'm not sure the warnings are only from statsmodels, they could be in numpy, scipy or pandas, couldn't they? > > > second, what's wrong with non-integers in `np.r_[[np.nan] * head, x, [np.nan] * tail]` (see below) > > I tried to set the warnings filter to `error` but then Python itself errored right away. I'd try using a more targeted error filter. Python trends to spit out a bunch of ImportWarnings on startup. You can make a warnings filter that does "error" but only for DeprecationWarning. Or only for DeprecationWarning matching the regexp "non-integer". -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From olof.backing at combitech.se Wed Jul 1 07:30:12 2015 From: olof.backing at combitech.se (Backing Olof) Date: Wed, 1 Jul 2015 11:30:12 +0000 Subject: [Numpy-discussion] F2PY and multi-dimension arrays - unexpected array size error In-Reply-To: <49F675D2-88C7-46BC-8252-153429468219@combitech.se> References: <49F675D2-88C7-46BC-8252-153429468219@combitech.se> Message-ID: Hello, the project is already successfully completed, but I thought for the completeness (and documentation) give myself an answer on what I did wrong and what the correct solution was. I. I didn?t realise that parameter ordering was crucial. Somehow I just had forgotten this knowledge, but not matter why I did. So after some talk with my colleagues I did finally sort that out. II. You can integrate Python and FORTRAN in two different ways: inline with Cf2py or with signature files. The Numpy and f2py documentation describes the Cf2py solution very well. But my project asked for *no* modification of the FORTRAN source files. So I had to enter the undocumented arena. Lesson learnt: When using the signature file option f2py will effectively re-sort all the parameters in the following order: * integer * real * character * real dimension So the real/correct parameter order will be shown inside if Python when using: >>> print (func.___doc___) If I only had known this in the beginning? ;) Al the best, Olof Backing ? Olof Backing Sr Open Source Specialist Combitech AB Box 1004 ? SE-164 21 KISTA ? Sweden Visiting address Torshamnsgatan 30C Phn +46 8 580 861 95 ? Mobile +46 73 437 61 95 olof.backing at combitech.se ? www.combitech.se > On 03 Feb 2015, at 18:51, Backing Olof wrote: > > Hi > I am helping out with a Python and Fortran project. Let me give you some background: > > * Fortran source: > C Bergstrom FCC > C User subroutine VUMAT > subroutine VUMAT( > C Read only - > * nblock, ndir, nshr, nstatev, nprops, > * stepTime, dt, > * props, > * density, strainInc, > * tempOld, > * stressOld, stateOld, enerInternOld, enerInelasOld, > C Write only - > * stressNew, stateNew, enerInternNew, enerInelasNew ) > C > include 'vaba_param.inc' > C > dimension props(nprops), > 1 density(nblock), strainInc(nblock,ndir+nshr), > 2 tempOld(nblock), > 5 stressOld(nblock,ndir+nshr), stateOld(nblock,nstatev), > 6 enerInternOld(nblock), enerInelasOld(nblock), > 2 stressNew(nblock,ndir+nshr), stateNew(nblock,nstatev), > 3 enerInternNew(nblock), enerInelasNew(nblock) > > * Corresponding .pyf > integer :: nblock > integer :: ndir > integer :: nshr > integer :: nstatev > integer :: nprops > real :: steptime > real :: dt > real dimension(nprops) :: props > real dimension(nblock) :: density > real dimension(nblock,ndir+nshr) :: straininc > real dimension(nblock) :: tempold > real dimension(nblock,ndir+nshr) :: stressold > real dimension(nblock,nstatev) :: stateold > real dimension(nblock) :: enerinternold > real dimension(nblock) :: enerinelasold > real dimension(nblock,ndir+nshr),intent(out) :: stressnew > real dimension(nblock,nstatev),intent(out) :: statenew > real dimension(nblock),intent(out) :: enerinternnew > real dimension(nblock),intent(out) :: enerinelasnew > > * Python source with call of Fortran routine: > nblock = 1 > ndir = 3 > nshr = 3 > nstatev = 3 > nprops = 11 > stepTime = 1 > dt = 1 > props = np.array([10, 0.5, 1e10, 5, 1e12, 3e-6, 8e-6, 27, 2], float) > density = np.array([[7.8e3]], float) > strainInc = np.array([[1,-0.5,-0.5,0,0,0]], float) > tempOld = np.array([[1]], float) > stressOld = np.array([[1,1,1,1,1,1]], float) > stateOld = np.array([[1,1,1]], float) > enerInternOld = np.array([1], float) > enerInelasOld = np.array([1], float) > > stressNew = np.array([[]], float) > stateNew = np.array([[]], float) > enerInternNew = np.array([[]], float) > enerInelasNew = np.array([[]], float) > > stressNew, stateNew, enerInternNew, enerInelasNew = vumat(nblock, ndir, nshr, nstatev, nprops, stepTime, dt, props, density, strainInc, tempOld, stressOld, stateOld, enerInternOld, enerInelasOld) > > When trying to run with Python 2.7 I get: > olof at ubuntu:~$ ./demo.py > unexpected array size: new_size=4, got array with arr_size=1 > Traceback (most recent call last): > File "./demo.py", line 33, in > main() > File "./demo.py", line 30, in main > stressNew, stateNew, enerInternNew, enerInelasNew = vumat(nblock, ndir, nshr, nstatev, nprops, stepTime, dt, props, density, strainInc, tempOld, stressOld, stateOld, enerInternOld, enerInelasOld) > VUMAT_Bergstrom_FCC.error: failed in converting 9th argument `stressold' of VUMAT_Bergstrom_FCC.vumat to C/Fortran array > > Other stuff: > * python 2.7.6 > * numpy/f2py 1.8.2 > * gcc/gfortran 4.8.2 > * ubuntu 14.04 LTS 32-bit > > I have tried to google, read the f2py manual, fortran tutorials etc, but to no avail. I must also admit that my knowledge in python is so-so and fortran even less(!). What is the missing statement/syntax that I can?t get correct? > > Your humble programmer, Olof From josef.pktd at gmail.com Wed Jul 1 10:05:23 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 1 Jul 2015 10:05:23 -0400 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? Message-ID: About the deprecation warning for using another type than integers, in ones, reshape, indexing and so on: Wouldn't it be nicer to accept floats that are equal to an integer? for example >>> 5.0 == 5 True >>> np.ones(10 / 2) array([ 1., 1., 1., 1., 1.]) >>> 10 / 2 == 5 True or the python 2 version >>> np.ones(10. / 2) array([ 1., 1., 1., 1., 1.]) >>> 10. / 2 == 5 True I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional and doesn't raise if the numbers are not close or equal to an integer (which would be a bug) Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Jul 1 10:32:10 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 01 Jul 2015 16:32:10 +0200 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: References: Message-ID: <1435761130.2293.15.camel@sipsolutions.net> On Mi, 2015-07-01 at 10:05 -0400, josef.pktd at gmail.com wrote: > About the deprecation warning for using another type than integers, in > ones, reshape, indexing and so on: > > > Wouldn't it be nicer to accept floats that are equal to an integer? > Hmmm, the biggest point was that the old solution was to basically (besides strings) use `int(...)`, which means it does not raise any errors as you also mention. I am open to think about allowing exact floats for most of this (frankly, not advanced indexing at least for the moment, but we never did there), I think scipy may be doing that for some functions? The disadvantage I see is, that some weirder calculations would possible work most of the times, but not always, what I mean is such a case. A -- possibly silly -- example: In [8]: for i in range(10): ...: print i, i == i * 0.1 * 10 ...: 0 True 1 True 2 True 3 False 4 True 5 True 6 False 7 False 8 True 9 True I am somewhat opposed to rounding a lot (i.e. not noticing if you got 3.3333 somewhere), so not sure if you can define a "tolerance" reasonable here unless it is exact. Though I guess you are right that `//` will also just round silently already. - Sebastian > > for example > > > >>> 5.0 == 5 > True > > > >>> np.ones(10 / 2) > array([ 1., 1., 1., 1., 1.]) > >>> 10 / 2 == 5 > True > > > or the python 2 version > > > >>> np.ones(10. / 2) > array([ 1., 1., 1., 1., 1.]) > >>> 10. / 2 == 5 > True > > > I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional > and doesn't raise if the numbers are not close or equal to an integer > (which would be a bug) > > > > > Josef > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From josef.pktd at gmail.com Wed Jul 1 11:07:43 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 1 Jul 2015 11:07:43 -0400 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: <1435761130.2293.15.camel@sipsolutions.net> References: <1435761130.2293.15.camel@sipsolutions.net> Message-ID: On Wed, Jul 1, 2015 at 10:32 AM, Sebastian Berg wrote: > On Mi, 2015-07-01 at 10:05 -0400, josef.pktd at gmail.com wrote: > > About the deprecation warning for using another type than integers, in > > ones, reshape, indexing and so on: > > > > > > Wouldn't it be nicer to accept floats that are equal to an integer? > > > > Hmmm, the biggest point was that the old solution was to basically > (besides strings) use `int(...)`, which means it does not raise any > errors as you also mention. > I am open to think about allowing exact floats for most of this > (frankly, not advanced indexing at least for the moment, but we never > did there), I think scipy may be doing that for some functions? > > The disadvantage I see is, that some weirder calculations would possible > work most of the times, but not always, what I mean is such a case. > A -- possibly silly -- example: > > In [8]: for i in range(10): > ...: print i, i == i * 0.1 * 10 > ...: > 0 True > 1 True > 2 True > 3 False > 4 True > 5 True > 6 False > 7 False > 8 True > 9 True > > I am somewhat opposed to rounding a lot (i.e. not noticing if you got > 3.3333 somewhere), so not sure if you can define a "tolerance" > reasonable here unless it is exact. Though I guess you are right that > `//` will also just round silently already. > Yes, I thought about this, something like `int_if_close` in analogy to real_if_close would be useful. However, given that we need to decide on a threshold in this case, I thought it's overkill to put that into reshape, ones and indexing and similar. Simpler cases would work number if triangular elements >>> for i in range(10): print(i, i * (i - 1) / 2. == int(i * (i - 1) / 2.)) 0 True 1 True 2 True 3 True 4 True 5 True 6 True 7 True 8 True 9 True also np.ceil and np.trunc return floats, not integers. One disadvantage of raising or warning after the equality check is that developers have a tendency to write "nice" unit tests. Then the casting doesn't break in the unit tests but might raise an exception at some random data. For reference: here are my changes in cleaning up https://github.com/statsmodels/statsmodels/pull/2490/files Josef > > - Sebastian > > > > > for example > > > > > > >>> 5.0 == 5 > > True > > > > > > >>> np.ones(10 / 2) > > array([ 1., 1., 1., 1., 1.]) > > >>> 10 / 2 == 5 > > True > > > > > > or the python 2 version > > > > > > >>> np.ones(10. / 2) > > array([ 1., 1., 1., 1., 1.]) > > >>> 10. / 2 == 5 > > True > > > > > > I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional > > and doesn't raise if the numbers are not close or equal to an integer > > (which would be a bug) > > > > > > > > > > Josef > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Jul 2 08:40:13 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 02 Jul 2015 08:40:13 -0400 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? References: <1435761130.2293.15.camel@sipsolutions.net> Message-ID: josef.pktd at gmail.com wrote: > On Wed, Jul 1, 2015 at 10:32 AM, Sebastian Berg > wrote: > >> On Mi, 2015-07-01 at 10:05 -0400, josef.pktd at gmail.com wrote: >> > About the deprecation warning for using another type than integers, in >> > ones, reshape, indexing and so on: >> > >> > >> > Wouldn't it be nicer to accept floats that are equal to an integer? >> > >> I'd be concerned that checking each index for exactness would be costly. I'm also concerned that using floats for an index is frequently a mistake and that a warning is what I want. From solipsis at pitrou.net Thu Jul 2 09:37:19 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Jul 2015 15:37:19 +0200 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? References: <1435761130.2293.15.camel@sipsolutions.net> Message-ID: <20150702153719.5c14515a@fsol> On Thu, 02 Jul 2015 08:40:13 -0400 Neal Becker wrote: > I'd be concerned that checking each index for exactness would be costly. > I'm also concerned that using floats for an index is frequently a mistake > and that a warning is what I want. Or just follow Python: Python 3.4.3 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:29:08) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> [1][0.0] Traceback (most recent call last): File "", line 1, in TypeError: list indices must be integers, not float I don't think relaxing type checking here makes any good. Regards Antoine. From bachir.aoun at e-aoun.com Thu Jul 2 10:37:24 2015 From: bachir.aoun at e-aoun.com (Bachir Aoun) Date: Thu, 2 Jul 2015 09:37:24 -0500 Subject: [Numpy-discussion] Biased random number generator Message-ID: Dear All, I have recently developed some features that I think can be useful for others. I would like to contribute by providing the code of the following definitions BiasedRandomFloatGenerator BiasedRandomIntegerGenerator please find the help of those two classes here http://bachiraoun.github.io/fullrmc/fullrmc.Core.html#module-fullrmc.Core.Collection Personally, I am using those generators to model molecular structures by reverse engineering experimental data. The generators accumulate experience through the whole modelling process and automatically update the generation scheme (numbers probability) according to some success / failure parameter. If you think this is something that might be interesting and has the potential to being distributed in the coming Numpy versions please let me know. regards -- Bachir AOUN -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Thu Jul 2 13:38:17 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 2 Jul 2015 17:38:17 +0000 (UTC) Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? References: <20150702153719.5c14515a@fsol> Message-ID: <1488095213457551406.797965sturla.molden-gmail.com@news.gmane.org> Antoine Pitrou wrote: > I don't think relaxing type checking here makes any good. I agee. NumPy should do the same as Python in this case. Sturla From njs at pobox.com Thu Jul 2 15:59:05 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 2 Jul 2015 12:59:05 -0700 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: Hi all, Meeting is starting in a few minutes! Hangouts link: https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a Google doc for agenda and notes: https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing Wiki page: https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting -n On Fri, Jun 26, 2015 at 2:32 AM, Nathaniel Smith wrote: > Hi all, > > In a week and a half, this is happening: > > https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting > > It's somewhat short notice (my bad :-/), but I think it would be good > to have a short video meeting sometime this week as a kind of > "pre-meeting" -- to at least briefly go over the main issues we see > facing the project to prime the pump, get a better idea about what we > want to accomplish at the meeting itself, and gather some early > feedback from anyone who won't be able to make it to SciPy (we'll miss > you). > > The obligatory doodle: > http://doodle.com/6b4s6thqt9xt4vnh > > Depending on the interest level, I'm thinking we'll either use Google > Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf > used for the similar SciPy meeting a few months ago; needs a plugin > installed but is available for Windows / OS X / 64-bit Linux / Android > / iOS, or regular telephone, or h323 softphone). > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org -- Nathaniel J. Smith -- http://vorpus.org From honi at brandeis.edu Thu Jul 2 16:01:26 2015 From: honi at brandeis.edu (Honi Sanders) Date: Thu, 2 Jul 2015 16:01:26 -0400 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: I?m interested in listening in just to see what it?s like, but I have to leave after ~15 minutes because I have a meeting at 4:30. Is that too disruptive? > On Jul 2, 2015, at 3:59 PM, Nathaniel Smith wrote: > > Hi all, > > Meeting is starting in a few minutes! > > Hangouts link: > https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a > > Google doc for agenda and notes: > https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing > > Wiki page: > https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting > > -n > > On Fri, Jun 26, 2015 at 2:32 AM, Nathaniel Smith wrote: >> Hi all, >> >> In a week and a half, this is happening: >> >> https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting >> >> It's somewhat short notice (my bad :-/), but I think it would be good >> to have a short video meeting sometime this week as a kind of >> "pre-meeting" -- to at least briefly go over the main issues we see >> facing the project to prime the pump, get a better idea about what we >> want to accomplish at the meeting itself, and gather some early >> feedback from anyone who won't be able to make it to SciPy (we'll miss >> you). >> >> The obligatory doodle: >> http://doodle.com/6b4s6thqt9xt4vnh >> >> Depending on the interest level, I'm thinking we'll either use Google >> Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf >> used for the similar SciPy meeting a few months ago; needs a plugin >> installed but is available for Windows / OS X / 64-bit Linux / Android >> / iOS, or regular telephone, or h323 softphone). >> >> -n >> >> -- >> Nathaniel J. Smith -- http://vorpus.org > > > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at gmail.com Thu Jul 2 16:04:06 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Jul 2015 22:04:06 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: On Thu, Jul 2, 2015 at 10:01 PM, Honi Sanders wrote: > I?m interested in listening in just to see what it?s like, but I have to > leave after ~15 minutes because I have a meeting at 4:30. Is that too > disruptive? > Don't worry about it, just join as long as you can. Ralf > > > On Jul 2, 2015, at 3:59 PM, Nathaniel Smith wrote: > > > > Hi all, > > > > Meeting is starting in a few minutes! > > > > Hangouts link: > > https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a > > > > Google doc for agenda and notes: > > > https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing > > > > Wiki page: > > https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting > > > > -n > > > > On Fri, Jun 26, 2015 at 2:32 AM, Nathaniel Smith wrote: > >> Hi all, > >> > >> In a week and a half, this is happening: > >> > >> https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting > >> > >> It's somewhat short notice (my bad :-/), but I think it would be good > >> to have a short video meeting sometime this week as a kind of > >> "pre-meeting" -- to at least briefly go over the main issues we see > >> facing the project to prime the pump, get a better idea about what we > >> want to accomplish at the meeting itself, and gather some early > >> feedback from anyone who won't be able to make it to SciPy (we'll miss > >> you). > >> > >> The obligatory doodle: > >> http://doodle.com/6b4s6thqt9xt4vnh > >> > >> Depending on the interest level, I'm thinking we'll either use Google > >> Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf > >> used for the similar SciPy meeting a few months ago; needs a plugin > >> installed but is available for Windows / OS X / 64-bit Linux / Android > >> / iOS, or regular telephone, or h323 softphone). > >> > >> -n > >> > >> -- > >> Nathaniel J. Smith -- http://vorpus.org > > > > > > > > -- > > Nathaniel J. Smith -- http://vorpus.org > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Thu Jul 2 16:29:26 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Thu, 2 Jul 2015 22:29:26 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: unfortunately I can't start the hangout. Both Firefox and chrome hangs. Tried it again and again. For this reason a short status: I can now build the mingwpy toolchain as pip installable wheel. %USERPROFILE%\pydistutils.cfg should be configured to use the mingw compiler. With the preliminary mingwpy wheels deployed at binstar: (2.7, 3.3, 3.4 / amd64, win32): pip install -i https://pypi.binstar.org/carlkl/simple mingwpy the toolchain can be installed. Test i.e. it with bottleneck: pip install bottleneck compiles and installs the bottleneck package on the fly. I would like to add some documenation to the package and wait for some feedback from others before deployment on pypi. Supplementary libraries (like OpenBLAS) should be provided as individual packages. I will need some time at the weekend to put all this stuff on github as well as many of my usual manual steps had to be scripted first. There is a lot of my TODO list: - further work numpy, scipy compiled with mingwpy and OpenBLAS - setup documenation and mingwpy.org - deploy on PYPI - win32 bugs in the math code of mingwpw - precision failures with scipy.test - alternative implemention of some transz. numeric functions in minw-w64 Wait for your feedback Cheers Carl 2015-07-02 22:04 GMT+02:00 Ralf Gommers : > > > On Thu, Jul 2, 2015 at 10:01 PM, Honi Sanders wrote: > >> I?m interested in listening in just to see what it?s like, but I have to >> leave after ~15 minutes because I have a meeting at 4:30. Is that too >> disruptive? >> > > Don't worry about it, just join as long as you can. > > Ralf > > > >> >> > On Jul 2, 2015, at 3:59 PM, Nathaniel Smith wrote: >> > >> > Hi all, >> > >> > Meeting is starting in a few minutes! >> > >> > Hangouts link: >> > https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a >> > >> > Google doc for agenda and notes: >> > >> https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing >> > >> > Wiki page: >> > https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting >> > >> > -n >> > >> > On Fri, Jun 26, 2015 at 2:32 AM, Nathaniel Smith wrote: >> >> Hi all, >> >> >> >> In a week and a half, this is happening: >> >> >> >> https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting >> >> >> >> It's somewhat short notice (my bad :-/), but I think it would be good >> >> to have a short video meeting sometime this week as a kind of >> >> "pre-meeting" -- to at least briefly go over the main issues we see >> >> facing the project to prime the pump, get a better idea about what we >> >> want to accomplish at the meeting itself, and gather some early >> >> feedback from anyone who won't be able to make it to SciPy (we'll miss >> >> you). >> >> >> >> The obligatory doodle: >> >> http://doodle.com/6b4s6thqt9xt4vnh >> >> >> >> Depending on the interest level, I'm thinking we'll either use Google >> >> Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf >> >> used for the similar SciPy meeting a few months ago; needs a plugin >> >> installed but is available for Windows / OS X / 64-bit Linux / Android >> >> / iOS, or regular telephone, or h323 softphone). >> >> >> >> -n >> >> >> >> -- >> >> Nathaniel J. Smith -- http://vorpus.org >> > >> > >> > >> > -- >> > Nathaniel J. Smith -- http://vorpus.org >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Jul 2 20:51:01 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 2 Jul 2015 17:51:01 -0700 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: <1435761130.2293.15.camel@sipsolutions.net> References: <1435761130.2293.15.camel@sipsolutions.net> Message-ID: <965438590629112265@unknownmsgid> Sent from my iPhone > > The disadvantage I see is, that some weirder calculations would possible > work most of the times, but not always, > not sure if you can define a "tolerance" > reasonable here unless it is exact. You could use a relative tolerance, but you'd still have to set that. Better to put that decision squarely in the user's hands. > Though I guess you are right that > `//` will also just round silently already. Yes, but if it's in the user's code, it should be obvious -- and then the user can choose to round, or floor, or ceiling.... -CHB > > - Sebastian > >> >> for example >> >> >>>>> 5.0 == 5 >> True >> >> >>>>> np.ones(10 / 2) >> array([ 1., 1., 1., 1., 1.]) >>>>> 10 / 2 == 5 >> True >> >> >> or the python 2 version >> >> >>>>> np.ones(10. / 2) >> array([ 1., 1., 1., 1., 1.]) >>>>> 10. / 2 == 5 >> True >> >> >> I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional >> and doesn't raise if the numbers are not close or equal to an integer >> (which would be a bug) >> >> >> >> >> Josef >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From josef.pktd at gmail.com Thu Jul 2 21:18:57 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Jul 2015 21:18:57 -0400 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: <965438590629112265@unknownmsgid> References: <1435761130.2293.15.camel@sipsolutions.net> <965438590629112265@unknownmsgid> Message-ID: On Thu, Jul 2, 2015 at 8:51 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > Sent from my iPhone > > > > > The disadvantage I see is, that some weirder calculations would possible > > work most of the times, but not always, > > > > not sure if you can define a "tolerance" > > reasonable here unless it is exact. > > You could use a relative tolerance, but you'd still have to set that. > Better to put that decision squarely in the user's hands. > > > Though I guess you are right that > > `//` will also just round silently already. > > Yes, but if it's in the user's code, it should be obvious -- and then > the user can choose to round, or floor, or ceiling.... > round, floor, ceil don't produce integers. I'm writing library code, and I don't have control over what everyone does. round, floor, ceil, and // might hide bugs or user mistakes, if we are supposed to get something that is "like an int" but it's. 42.6 instead. Josef https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Answer_to_the_Ultimate_Question_of_Life.2C_the_Universe.2C_and_Everything_.2842.29 > > -CHB > > > > > - Sebastian > > > >> > >> for example > >> > >> > >>>>> 5.0 == 5 > >> True > >> > >> > >>>>> np.ones(10 / 2) > >> array([ 1., 1., 1., 1., 1.]) > >>>>> 10 / 2 == 5 > >> True > >> > >> > >> or the python 2 version > >> > >> > >>>>> np.ones(10. / 2) > >> array([ 1., 1., 1., 1., 1.]) > >>>>> 10. / 2 == 5 > >> True > >> > >> > >> I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional > >> and doesn't raise if the numbers are not close or equal to an integer > >> (which would be a bug) > >> > >> > >> > >> > >> Josef > >> > >> > >> > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreback at gmail.com Thu Jul 2 21:28:11 2015 From: jeffreback at gmail.com (Jeff Reback) Date: Thu, 2 Jul 2015 21:28:11 -0400 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: References: <1435761130.2293.15.camel@sipsolutions.net> <965438590629112265@unknownmsgid> Message-ID: FYI pandas followed the same pattern to deprecate float indexers (except for indexing in a Float64Index) about a year ago see here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0140-deprecations > On Jul 2, 2015, at 9:18 PM, wrote: > > > >> On Thu, Jul 2, 2015 at 8:51 PM, Chris Barker - NOAA Federal wrote: >> Sent from my iPhone >> >> > >> > The disadvantage I see is, that some weirder calculations would possible >> > work most of the times, but not always, >> >> >> > not sure if you can define a "tolerance" >> > reasonable here unless it is exact. >> >> You could use a relative tolerance, but you'd still have to set that. >> Better to put that decision squarely in the user's hands. >> >> > Though I guess you are right that >> > `//` will also just round silently already. >> >> Yes, but if it's in the user's code, it should be obvious -- and then >> the user can choose to round, or floor, or ceiling.... > > round, floor, ceil don't produce integers. > > I'm writing library code, and I don't have control over what everyone does. > > round, floor, ceil, and // might hide bugs or user mistakes, if we are supposed to get something that is "like an int" but it's. 42.6 instead. > > Josef > https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Answer_to_the_Ultimate_Question_of_Life.2C_the_Universe.2C_and_Everything_.2842.29 > > >> >> -CHB >> >> > >> > - Sebastian >> > >> >> >> >> for example >> >> >> >> >> >>>>> 5.0 == 5 >> >> True >> >> >> >> >> >>>>> np.ones(10 / 2) >> >> array([ 1., 1., 1., 1., 1.]) >> >>>>> 10 / 2 == 5 >> >> True >> >> >> >> >> >> or the python 2 version >> >> >> >> >> >>>>> np.ones(10. / 2) >> >> array([ 1., 1., 1., 1., 1.]) >> >>>>> 10. / 2 == 5 >> >> True >> >> >> >> >> >> I'm using now 10 // 2, or int(10./2 + 1) but this is unconditional >> >> and doesn't raise if the numbers are not close or equal to an integer >> >> (which would be a bug) >> >> >> >> >> >> >> >> >> >> Josef >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbednar at inf.ed.ac.uk Fri Jul 3 16:10:26 2015 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Fri, 3 Jul 2015 21:10:26 +0100 Subject: [Numpy-discussion] ANN: HoloViews 1.3 released Message-ID: <21910.60466.945047.219063@hebb.inf.ed.ac.uk> We are pleased to announce the fourth public release of HoloViews, a Python package for simplifying the exploration of scientific data: http://holoviews.org HoloViews provides composable, sliceable, declarative data structures for building even complex visualizations easily. The goal of HoloViews is to let your data just visualize itself, allowing you to work with large datasets as easily as you work with simple datatypes at the Python prompt. You can obtain the new version using conda or pip: conda install holoviews pip install --upgrade 'holoviews[recommended]' This release includes a substantial number of new features and API improvements, most of which have been suggested by our growing userbase: - Major optimizations throughout, both for working with HoloViews data structures and for visualization. - Improved widget appearance and greatly reduced flickering issues when interactively exploring data in the browser. - Improved handling of unicode and LaTeX text throughout, using Python 3's better unicode support (when available). - New Polygons, ErrorBars, and Spread Element types. - Support for multiple matplotlib backends (vanilla matplotlib, mpld3 and nbagg) with support for other plotting systems (such as Bokeh) in development. Easily switching between backends allows you to take advantage of the unique features of each one, such as good SVG/PDF output, interactive zooming and panning, or 3D viewpoint control. - Streamlined the API based on user feedback; now even more things "just work". This includes new, easy to use constructors for common Element types as well as easy conversion between them. - More customizability of plot and style options, including easier control over font sizes, legend positions, background color, and multiple color bars. Polar projections now supported throughout. - More flexible and customizable Layouts, allowing the user to define blank spaces (using the Empty object) as well as more control over positioning and aspect ratios. - Support for a holoviews.rc file, integration with IPython Notebook interact widgets, improvements to the Pandas interface, easy saving and loading of data via pickling, and much more. And of course we have fixed a number of bugs found by our very dedicated users; please keep filing Github issues if you find any! For the full list of changes, see: https://github.com/ioam/holoviews/releases HoloViews remains freely available under a BSD license, is Python 2 and 3 compatible, and has minimal external dependencies, making it easy to integrate into your workflow. Try out the extensive tutorials at holoviews.org today, and check out our upcoming SciPy and EuroSciPy talks in Austin and Cambridge (or read the paper at http://goo.gl/NH9FTB)! Philipp Rudiger Jean-Luc R. Stevens James A. Bednar The University of Edinburgh School of Informatics -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From njs at pobox.com Fri Jul 3 19:47:32 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 3 Jul 2015 16:47:32 -0700 Subject: [Numpy-discussion] Readings about governance and decision making for F/OSS projects Message-ID: Hi all, As discussed on the call yesterday, here's some links to background reading on F/OSS project governance, to use as background reading for our discussion Tuesday. There's a lot out there, but here are the two I think are the most important/relevant: The classic book on running a F/OSS project is Karl Fogel's "Producing OSS", which is available free online. Chapter 4 is the relevant one for this topic (though the whole book is worth reading): http://producingoss.com/en/producingoss.html#social-infrastructure The IPython governance documents are here: https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance I think it will give us a really good start on Tuesday if everyone is able to find the time to read Chapter 4 + IPEP 29 ahead of time. Something to do on the plane, maybe :-). --- For extra credit, some less crucial links that still might be interesting (and give something of a sense of the range of options): The Apache voting system: http://www.apache.org/foundation/voting.html The GNOME foundation's governance documents: https://www.gnome.org/foundation/governance/ The Debian constitution, esp. section 6 (the technical committee): https://www.debian.org/devel/constitution https://www.debian.org/devel/tech-ctte The GCC steering committee: https://www.gnu.org/software/gcc/steering.html The Node.JS foundation (interesting case -- a foundation in the process of bootstrapping in order to resolve a fork triggered by conflict between outside contributors and the company that started the project): https://nodejs.org/foundation/ Jono Bacon's book "The Art of Community", also available free online: http://artofcommunityonline.org/Art_of_Community_Second_Edition.pdf The governance chapter here is much more oriented towards Ubuntu-sized projects with hundreds-to-thousands of contributors, so not as relevant for us, but it still contains lots of interesting stuff. If you have other links on this topic that you are think are interesting, please add them to the thread! -n -- Nathaniel J. Smith -- http://vorpus.org From chris.barker at noaa.gov Sat Jul 4 01:07:36 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Jul 2015 22:07:36 -0700 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: References: <1435761130.2293.15.camel@sipsolutions.net> <965438590629112265@unknownmsgid> Message-ID: On Thu, Jul 2, 2015 at 6:18 PM, wrote: > round, floor, ceil don't produce integers. > True -- in a dynamic language, they probably should, but that's legacy that won't change. It's annoying, but you do need to do: int(round(want_it_to_be_an_index)) but as they say, explicite is better than implicit. > I'm writing library code, and I don't have control over what everyone does. > I'm confused -- what is the problem here -- if your library code required an integer for an index, then that's what your users need to pass in -- how they get that integer is under their control -- why would you want it otherwise? Or your code does the round|ceil|floor and int conversion -- but hen you know what you're doing. round, floor, ceil, and // might hide bugs or user mistakes, if we are > supposed to get something that is "like an int" but it's. 42.6 instead. > then it will raise an exception -- what's the problem? but what should 42.000000000001 do? IT seems to me, there is no choice but an exception -- or you are really going to hide bugs. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Sat Jul 4 03:26:20 2015 From: sebix at sebix.at (Sebastian) Date: Sat, 04 Jul 2015 09:26:20 +0200 Subject: [Numpy-discussion] floats for indexing, reshape - too strict ? In-Reply-To: <20150702153719.5c14515a@fsol> References: <1435761130.2293.15.camel@sipsolutions.net> <20150702153719.5c14515a@fsol> Message-ID: <55978A9C.2030903@sebix.at> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, On 07/02/2015 03:37 PM, Antoine Pitrou wrote: > > Traceback (most recent call last): > File "", line 1, in > TypeError: list indices must be integers, not float > > > I don't think relaxing type checking here makes any good. Python is also strong-typed which means that types are never converted silently. I think a library should follow the behavior of the language. https://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic%20language%20and%20also%20a%20strongly%20typed%20language Sebastian - -- python programming - mail server - photo - video - https://sebix.at To verify my cryptographic signature or send me encrypted mails, get my key at https://sebix.at/DC9B463B.asc and on public keyservers. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBCAAGBQJVl4qcAAoJEBn0X+vcm0Y7foUP/3CWITL83aoqd+H1JHXLUft4 TbGfW54TkBCLIJt3/DxyNACKpNPBEbanCL/DgW6OHZ2qTfRbYF6wDtPB39a5qK5s v3+z4nUwkTVVUNDNEoyQ3C8VfCaGwNnSGUoBAhoL9Y1njdSqbHJZa+dNNJ3P289D SThXDJ9bmV5AKmpdjPxZYDtcDkmoO+0SiBeCC4OhZY7Jf29VdICzWpnLbeZGSSSi OVleHOxYL1BoA4chKFtjhm6Arkrlp/485erXWtuFTt8V3elruTQRLlbty5wzNVf1 EaTW31nDEJFJxd+9aOsopPgLGlwQZ01LkVA2JSeNV57OisqTIEs5MsmG+vN2X2w1 ++I/IFHQpDHqE6nsKMzFPDzdA3vlEgoYY9J8bzqqzLHFdSgpR+nhUTjJmW6uNuU4 NuqDq7NDh9BlNhRG9ZDD/JktpOjDrbfBhOvx7V+WoAIZQ3b+WHwkNR78+i6KFGdC uxQa8SAkaEpBMPDe55l8zQFuUI2jGsbr1y6LF7EtInjaY562vij0Jte0LzNWEqVP XkRPwHlZ/ZVAVR5IJ083Z+S9uT+BYzoLWOxInxGLes606Mf+xx6LM7f5vE4h0MCJ Fsv5c2vRTbBybIKrAGHo6j+DT0WDt87u/UnSZg7ehpDZslaRThw+zJ+gBs4WqUEI 3m2Hk1hHXnRx5E9KmcNg =wNdm -----END PGP SIGNATURE----- From toddrjen at gmail.com Sat Jul 4 04:22:34 2015 From: toddrjen at gmail.com (Todd) Date: Sat, 4 Jul 2015 10:22:34 +0200 Subject: [Numpy-discussion] Readings about governance and decision making for F/OSS projects In-Reply-To: References: Message-ID: On Jul 4, 2015 1:47 AM, "Nathaniel Smith" wrote: > > > If you have other links on this topic that you are think are > interesting, please add them to the thread! > The KDE e.v. - represents the KDE community in legal and financial matters. https://ev.kde.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Jul 4 14:52:03 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 4 Jul 2015 20:52:03 +0200 Subject: [Numpy-discussion] Readings about governance and decision making for F/OSS projects In-Reply-To: References: Message-ID: On Sat, Jul 4, 2015 at 10:22 AM, Todd wrote: > > On Jul 4, 2015 1:47 AM, "Nathaniel Smith" wrote: > > > > > > If you have other links on this topic that you are think are > > interesting, please add them to the thread! > As promised, here some links about fiscal sponsorship. General info: https://www.councilofnonprofits.org/tools-resources/fiscal-sponsorship-nonprofits NumFOCUS main page on fiscal sponsorship agreements: http://numfocus.org/projects/join-as-project.html See in particular the comprehensive template (the one we need if we want to be a little ambitious regarding obtaining funding for numpy): https://docs.google.com/document/d/11YqMX9UrgfCSgiQEUzmOFyg6Ku-vED6gMxhO6J9lCgg/edit?usp=sharing Ralf The KDE e.v. - represents the KDE community in legal and financial > matters. > https://ev.kde.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Jul 4 17:40:07 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 4 Jul 2015 23:40:07 +0200 Subject: [Numpy-discussion] ANN: Scipy 0.16.0 release candidate 1 Message-ID: Hi, I'm pleased to announce the availability of the first release candidate of Scipy 0.16.0. Please try it out and report any issues on the Github issue tracker or on the scipy-dev mailing list. This first RC is a source-only release. Sources and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.16.0rc1. Note that this is a bit of an experiment - it's the first time we use the Github Releases feature. Feedback welcome. Thanks to everyone who contributed to this release! Ralf ========================== SciPy 0.16.0 Release Notes ========================== .. note:: Scipy 0.16.0 is not released yet! .. contents:: SciPy 0.16.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.15.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Highlights of this release include: - A Cython API for BLAS/LAPACK in `scipy.linalg` - A new benchmark suite. It's now straightforward to add new benchmarks, and they're routinely included with performance enhancement PRs. - Support for the second order sections (SOS) format in `scipy.signal`. New features ============ Benchmark suite --------------- The benchmark suite has switched to using `Airspeed Velocity `__ for benchmarking. You can run the suite locally via ``python runtests.py --bench``. For more details, see ``benchmarks/README.rst``. `scipy.linalg` improvements --------------------------- A full set of Cython wrappers for BLAS and LAPACK has been added in the modules `scipy.linalg.cython_blas` and `scipy.linalg.cython_lapack`. In Cython, these wrappers can now be cimported from their corresponding modules and used without linking directly against BLAS or LAPACK. The functions `scipy.linalg.qr_delete`, `scipy.linalg.qr_insert` and `scipy.linalg.qr_update` for updating QR decompositions were added. The function `scipy.linalg.solve_circulant` solves a linear system with a circulant coefficient matrix. The function `scipy.linalg.invpascal` computes the inverse of a Pascal matrix. The function `scipy.linalg.solve_toeplitz`, a Levinson-Durbin Toeplitz solver, was added. Added wrapper for potentially useful LAPACK function ``*lasd4``. It computes the square root of the i-th updated eigenvalue of a positive symmetric rank-one modification to a positive diagonal matrix. See its LAPACK documentation and unit tests for it to get more info. Added two extra wrappers for LAPACK least-square solvers. Namely, they are ``*gelsd`` and ``*gelsy``. Wrappers for the LAPACK ``*lange`` functions, which calculate various matrix norms, were added. Wrappers for ``*gtsv`` and ``*ptsv``, which solve ``A*X = B`` for tri-diagonal matrix ``A``, were added. `scipy.signal` improvements --------------------------- Support for second order sections (SOS) as a format for IIR filters was added. The new functions are: * `scipy.signal.sosfilt` * `scipy.signal.sosfilt_zi`, * `scipy.signal.sos2tf` * `scipy.signal.sos2zpk` * `scipy.signal.tf2sos` * `scipy.signal.zpk2sos`. Additionally, the filter design functions `iirdesign`, `iirfilter`, `butter`, `cheby1`, `cheby2`, `ellip`, and `bessel` can return the filter in the SOS format. The function `scipy.signal.place_poles`, which provides two methods to place poles for linear systems, was added. The option to use Gustafsson's method for choosing the initial conditions of the forward and backward passes was added to `scipy.signal.filtfilt`. New classes ``TransferFunction``, ``StateSpace`` and ``ZerosPolesGain`` were added. These classes are now returned when instantiating `scipy.signal.lti`. Conversion between those classes can be done explicitly now. An exponential (Poisson) window was added as `scipy.signal.exponential`, and a Tukey window was added as `scipy.signal.tukey`. The function for computing digital filter group delay was added as `scipy.signal.group_delay`. The functionality for spectral analysis and spectral density estimation has been significantly improved: `scipy.signal.welch` became ~8x faster and the functions `scipy.signal.spectrogram`, `scipy.signal.coherence` and `scipy.signal.csd` (cross-spectral density) were added. `scipy.signal.lsim` was rewritten - all known issues are fixed, so this function can now be used instead of ``lsim2``; ``lsim`` is orders of magnitude faster than ``lsim2`` in most cases. `scipy.sparse` improvements --------------------------- The function `scipy.sparse.norm`, which computes sparse matrix norms, was added. The function `scipy.sparse.random`, which allows to draw random variates from an arbitrary distribution, was added. `scipy.spatial` improvements ---------------------------- `scipy.spatial.cKDTree` has seen a major rewrite, which improved the performance of the ``query`` method significantly, added support for parallel queries, pickling, and options that affect the tree layout. See pull request 4374 for more details. The function `scipy.spatial.procrustes` for Procrustes analysis (statistical shape analysis) was added. `scipy.stats` improvements -------------------------- The Wishart distribution and its inverse have been added, as `scipy.stats.wishart` and `scipy.stats.invwishart`. The Exponentially Modified Normal distribution has been added as `scipy.stats.exponnorm`. The Generalized Normal distribution has been added as `scipy.stats.gennorm`. All distributions now contain a ``random_state`` property and allow specifying a specific ``numpy.random.RandomState`` random number generator when generating random variates. Many statistical tests and other `scipy.stats` functions that have multiple return values now return ``namedtuples``. See pull request 4709 for details. `scipy.optimize` improvements ----------------------------- A new derivative-free method DF-SANE has been added to the nonlinear equation system solving function `scipy.optimize.root`. Deprecated features =================== ``scipy.stats.pdf_fromgamma`` is deprecated. This function was undocumented, untested and rarely used. Statsmodels provides equivalent functionality with ``statsmodels.distributions.ExpandedNormal``. ``scipy.stats.fastsort`` is deprecated. This function is unnecessary, ``numpy.argsort`` can be used instead. ``scipy.stats.signaltonoise`` and ``scipy.stats.mstats.signaltonoise`` are deprecated. These functions did not belong in ``scipy.stats`` and are rarely used. See issue #609 for details. ``scipy.stats.histogram2`` is deprecated. This function is unnecessary, ``numpy.histogram2d`` can be used instead. Backwards incompatible changes ============================== The deprecated global optimizer ``scipy.optimize.anneal`` was removed. The following deprecated modules have been removed: ``scipy.lib.blas``, ``scipy.lib.lapack``, ``scipy.linalg.cblas``, ``scipy.linalg.fblas``, ``scipy.linalg.clapack``, ``scipy.linalg.flapack``. They had been deprecated since Scipy 0.12.0, the functionality should be accessed as `scipy.linalg.blas` and `scipy.linalg.lapack`. The deprecated function ``scipy.special.all_mat`` has been removed. The deprecated functions ``fprob``, ``ksprob``, ``zprob``, ``randwcdf`` and ``randwppf`` have been removed from `scipy.stats`. Other changes ============= The version numbering for development builds has been updated to comply with PEP 440. Building with ``python setup.py develop`` is now supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Mon Jul 6 11:18:13 2015 From: faltet at gmail.com (Francesc Alted) Date: Mon, 6 Jul 2015 17:18:13 +0200 Subject: [Numpy-discussion] Question about unaligned access Message-ID: Hi, I have stumbled into this: In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int32)]) In [63]: %timeit sa['f0'].sum() 100 loops, best of 3: 4.52 ms per loop In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int64)]) In [65]: %timeit sa['f0'].sum() 1000 loops, best of 3: 896 ?s per loop The first structured array is made of 12-byte records, while the second is made by 16-byte records, but the latter performs 5x faster. Also, using an structured array that is made of 8-byte records is the fastest (expected): In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', np.int64)]) In [67]: %timeit sa['f0'].sum() 1000 loops, best of 3: 567 ?s per loop Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform quite well on unaligned data: http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ So, if 4 years-old Intel architectures do not have a penalty for unaligned access, why I am seeing that in NumPy? That strikes like a quite strange thing to me. Thanks, Francesc -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Mon Jul 6 11:28:37 2015 From: faltet at gmail.com (Francesc Alted) Date: Mon, 6 Jul 2015 17:28:37 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: Oops, forgot to mention my NumPy version: In [72]: np.__version__ Out[72]: '1.9.2' Francesc 2015-07-06 17:18 GMT+02:00 Francesc Alted : > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', > np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', > np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the second is > made by 16-byte records, but the latter performs 5x faster. Also, using an > structured array that is made of 8-byte records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', > np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform > quite well on unaligned data: > > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for unaligned > access, why I am seeing that in NumPy? That strikes like a quite strange > thing to me. > > Thanks, > Francesc > > -- > Francesc Alted > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Mon Jul 6 12:04:11 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Mon, 6 Jul 2015 11:04:11 -0500 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted wrote: > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', > np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', > np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the second is > made by 16-byte records, but the latter performs 5x faster. Also, using an > structured array that is made of 8-byte records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', > np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform > quite well on unaligned data: > > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for unaligned > access, why I am seeing that in NumPy? That strikes like a quite strange > thing to me. > I believe that the way numpy is setup, it never does unaligned access, regardless of the platform, in case it gets run on one that would go up in flames if you tried to. So my guess would be that you are seeing chunked copies into a buffer, as opposed to bulk copying or no copying at all, and that would explain your timing differences. But Julian or Sebastian can probably give you a more informed answer. Jaime > > Thanks, > Francesc > > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Mon Jul 6 12:21:20 2015 From: faltet at gmail.com (Francesc Alted) Date: Mon, 6 Jul 2015 18:21:20 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o : > On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted wrote: > >> Hi, >> >> I have stumbled into this: >> >> In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), >> dtype=[('f0', np.int64), ('f1', np.int32)]) >> >> In [63]: %timeit sa['f0'].sum() >> 100 loops, best of 3: 4.52 ms per loop >> >> In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), >> dtype=[('f0', np.int64), ('f1', np.int64)]) >> >> In [65]: %timeit sa['f0'].sum() >> 1000 loops, best of 3: 896 ?s per loop >> >> The first structured array is made of 12-byte records, while the second >> is made by 16-byte records, but the latter performs 5x faster. Also, using >> an structured array that is made of 8-byte records is the fastest >> (expected): >> >> In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', >> np.int64)]) >> >> In [67]: %timeit sa['f0'].sum() >> 1000 loops, best of 3: 567 ?s per loop >> >> Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform >> quite well on unaligned data: >> >> >> http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ >> >> So, if 4 years-old Intel architectures do not have a penalty for >> unaligned access, why I am seeing that in NumPy? That strikes like a quite >> strange thing to me. >> > > I believe that the way numpy is setup, it never does unaligned access, > regardless of the platform, in case it gets run on one that would go up in > flames if you tried to. So my guess would be that you are seeing chunked > copies into a buffer, as opposed to bulk copying or no copying at all, and > that would explain your timing differences. But Julian or Sebastian can > probably give you a more informed answer. > Yes, my guess is that you are right. I suppose that it is possible to improve the numpy codebase to accelerate this particular access pattern on Intel platforms, but provided that structured arrays are not that used (pandas is probably leading this use case by far, and as far as I know, they are not using structured arrays internally in DataFrames), then maybe it is not worth to worry about this too much. Thanks anyway, Francesc > > Jaime > > >> >> Thanks, >> Francesc >> >> -- >> Francesc Alted >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Jul 6 14:11:47 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 06 Jul 2015 20:11:47 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: <559AC4E3.7030004@googlemail.com> On 06.07.2015 18:21, Francesc Alted wrote: > 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o >: > > On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted > wrote: > > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the > second is made by 16-byte records, but the latter performs 5x > faster. Also, using an structured array that is made of 8-byte > records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), > dtype=[('f0', np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should > perform quite well on unaligned data: > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for > unaligned access, why I am seeing that in NumPy? That strikes > like a quite strange thing to me. > > > I believe that the way numpy is setup, it never does unaligned > access, regardless of the platform, in case it gets run on one that > would go up in flames if you tried to. So my guess would be that you > are seeing chunked copies into a buffer, as opposed to bulk copying > or no copying at all, and that would explain your timing > differences. But Julian or Sebastian can probably give you a more > informed answer. > > > Yes, my guess is that you are right. I suppose that it is possible to > improve the numpy codebase to accelerate this particular access pattern > on Intel platforms, but provided that structured arrays are not that > used (pandas is probably leading this use case by far, and as far as I > know, they are not using structured arrays internally in DataFrames), > then maybe it is not worth to worry about this too much. > > Thanks anyway, > Francesc > > > > Jaime > > > > Thanks, > Francesc > > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > planes de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Francesc Alted > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From jtaylor.debian at googlemail.com Mon Jul 6 14:11:47 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 06 Jul 2015 20:11:47 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: <559AC4E3.80800@googlemail.com> On 06.07.2015 18:21, Francesc Alted wrote: > 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o >: > > On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted > wrote: > > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the > second is made by 16-byte records, but the latter performs 5x > faster. Also, using an structured array that is made of 8-byte > records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), > dtype=[('f0', np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should > perform quite well on unaligned data: > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for > unaligned access, why I am seeing that in NumPy? That strikes > like a quite strange thing to me. > > > I believe that the way numpy is setup, it never does unaligned > access, regardless of the platform, in case it gets run on one that > would go up in flames if you tried to. So my guess would be that you > are seeing chunked copies into a buffer, as opposed to bulk copying > or no copying at all, and that would explain your timing > differences. But Julian or Sebastian can probably give you a more > informed answer. > > > Yes, my guess is that you are right. I suppose that it is possible to > improve the numpy codebase to accelerate this particular access pattern > on Intel platforms, but provided that structured arrays are not that > used (pandas is probably leading this use case by far, and as far as I > know, they are not using structured arrays internally in DataFrames), > then maybe it is not worth to worry about this too much. > > Thanks anyway, > Francesc > > > > Jaime > > > > Thanks, > Francesc > > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > planes de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Francesc Alted > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From jtaylor.debian at googlemail.com Mon Jul 6 14:11:47 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 06 Jul 2015 20:11:47 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: <559AC4E3.9050108@googlemail.com> On 06.07.2015 18:21, Francesc Alted wrote: > 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o >: > > On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted > wrote: > > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the > second is made by 16-byte records, but the latter performs 5x > faster. Also, using an structured array that is made of 8-byte > records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), > dtype=[('f0', np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should > perform quite well on unaligned data: > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for > unaligned access, why I am seeing that in NumPy? That strikes > like a quite strange thing to me. > > > I believe that the way numpy is setup, it never does unaligned > access, regardless of the platform, in case it gets run on one that > would go up in flames if you tried to. So my guess would be that you > are seeing chunked copies into a buffer, as opposed to bulk copying > or no copying at all, and that would explain your timing > differences. But Julian or Sebastian can probably give you a more > informed answer. > > > Yes, my guess is that you are right. I suppose that it is possible to > improve the numpy codebase to accelerate this particular access pattern > on Intel platforms, but provided that structured arrays are not that > used (pandas is probably leading this use case by far, and as far as I > know, they are not using structured arrays internally in DataFrames), > then maybe it is not worth to worry about this too much. > > Thanks anyway, > Francesc > > > > Jaime > > > > Thanks, > Francesc > > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > planes de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Francesc Alted > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From jtaylor.debian at googlemail.com Mon Jul 6 14:32:34 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 06 Jul 2015 20:32:34 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: <559AC9C2.4000306@googlemail.com> sorry for the 3 empty mails, my client bugged out... as a workaround you can align structured dtypes to avoid this issue: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=np.dtype([('f0', np.int64), ('f1', np.int32)], align=True)) On 06.07.2015 18:21, Francesc Alted wrote: > 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o >: > > On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted > wrote: > > Hi, > > I have stumbled into this: > > In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int32)]) > > In [63]: %timeit sa['f0'].sum() > 100 loops, best of 3: 4.52 ms per loop > > In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), > dtype=[('f0', np.int64), ('f1', np.int64)]) > > In [65]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 896 ?s per loop > > The first structured array is made of 12-byte records, while the > second is made by 16-byte records, but the latter performs 5x > faster. Also, using an structured array that is made of 8-byte > records is the fastest (expected): > > In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), > dtype=[('f0', np.int64)]) > > In [67]: %timeit sa['f0'].sum() > 1000 loops, best of 3: 567 ?s per loop > > Now, my laptop has a Ivy Bridge processor (i5-3380M) that should > perform quite well on unaligned data: > > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ > > So, if 4 years-old Intel architectures do not have a penalty for > unaligned access, why I am seeing that in NumPy? That strikes > like a quite strange thing to me. > > > I believe that the way numpy is setup, it never does unaligned > access, regardless of the platform, in case it gets run on one that > would go up in flames if you tried to. So my guess would be that you > are seeing chunked copies into a buffer, as opposed to bulk copying > or no copying at all, and that would explain your timing > differences. But Julian or Sebastian can probably give you a more > informed answer. > > > Yes, my guess is that you are right. I suppose that it is possible to > improve the numpy codebase to accelerate this particular access pattern > on Intel platforms, but provided that structured arrays are not that > used (pandas is probably leading this use case by far, and as far as I > know, they are not using structured arrays internally in DataFrames), > then maybe it is not worth to worry about this too much. > > Thanks anyway, > Francesc > > From valentin at haenel.co Mon Jul 6 14:35:47 2015 From: valentin at haenel.co (Valentin Haenel) Date: Mon, 6 Jul 2015 20:35:47 +0200 Subject: [Numpy-discussion] webinterface down Message-ID: <20150706183547.GB12743@kudu.in-berlin.de> Hi, the webinterface at: http://mail.scipy.org/mailman/listinfo/numpy-discussion seems down for me. For anyone else too? V- From matthew.brett at gmail.com Mon Jul 6 17:39:01 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 6 Jul 2015 22:39:01 +0100 Subject: [Numpy-discussion] webinterface down In-Reply-To: <20150706183547.GB12743@kudu.in-berlin.de> References: <20150706183547.GB12743@kudu.in-berlin.de> Message-ID: Hi, On Mon, Jul 6, 2015 at 7:35 PM, Valentin Haenel wrote: > Hi, > > the webinterface at: > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > seems down for me. For anyone else too? Works for me now - but we had enough troubles with the mailman interface for nipy-devel at neuroimaging.scipy.org that we just switched over to a new list at neuroimaging at python.org. Cheers, Matthew From valentin at haenel.co Mon Jul 6 17:44:00 2015 From: valentin at haenel.co (Valentin Haenel) Date: Mon, 6 Jul 2015 23:44:00 +0200 Subject: [Numpy-discussion] webinterface down In-Reply-To: References: <20150706183547.GB12743@kudu.in-berlin.de> Message-ID: <20150706214400.GA23889@kudu.in-berlin.de> It's working again, thanks. * Matthew Brett [2015-07-06]: > Hi, > > On Mon, Jul 6, 2015 at 7:35 PM, Valentin Haenel wrote: > > Hi, > > > > the webinterface at: > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > seems down for me. For anyone else too? > > Works for me now - but we had enough troubles with the mailman > interface for nipy-devel at neuroimaging.scipy.org that we just switched > over to a new list at neuroimaging at python.org. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Tue Jul 7 01:58:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 7 Jul 2015 00:58:31 -0500 Subject: [Numpy-discussion] Dev meeting location Message-ID: Since several people have asked :-) The meeting is scheduled for 9:30 tomorrow (or technically today I guess since it's after midnight), in room "PDR 1". The full details on how to find it are on the wiki page! https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting -- Nathaniel J. Smith -- http://vorpus.org From toddrjen at gmail.com Tue Jul 7 03:53:50 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 7 Jul 2015 09:53:50 +0200 Subject: [Numpy-discussion] Question about unaligned access In-Reply-To: References: Message-ID: On Jul 6, 2015 6:21 PM, "Francesc Alted" wrote: > > 2015-07-06 18:04 GMT+02:00 Jaime Fern?ndez del R?o : >> >> On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted wrote: >>> >>> Hi, >>> >>> I have stumbled into this: >>> >>> In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int32)]) >>> >>> In [63]: %timeit sa['f0'].sum() >>> 100 loops, best of 3: 4.52 ms per loop >>> >>> In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int64)]) >>> >>> In [65]: %timeit sa['f0'].sum() >>> 1000 loops, best of 3: 896 ?s per loop >>> >>> The first structured array is made of 12-byte records, while the second is made by 16-byte records, but the latter performs 5x faster. Also, using an structured array that is made of 8-byte records is the fastest (expected): >>> >>> In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', np.int64)]) >>> >>> In [67]: %timeit sa['f0'].sum() >>> 1000 loops, best of 3: 567 ?s per loop >>> >>> Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform quite well on unaligned data: >>> >>> http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ >>> >>> So, if 4 years-old Intel architectures do not have a penalty for unaligned access, why I am seeing that in NumPy? That strikes like a quite strange thing to me. >> >> >> I believe that the way numpy is setup, it never does unaligned access, regardless of the platform, in case it gets run on one that would go up in flames if you tried to. So my guess would be that you are seeing chunked copies into a buffer, as opposed to bulk copying or no copying at all, and that would explain your timing differences. But Julian or Sebastian can probably give you a more informed answer. > > > Yes, my guess is that you are right. I suppose that it is possible to improve the numpy codebase to accelerate this particular access pattern on Intel platforms, but provided that structured arrays are not that used (pandas is probably leading this use case by far, and as far as I know, they are not using structured arrays internally in DataFrames), then maybe it is not worth to worry about this too much. > That may be more of a chicken-and-egg problem. Structured arrays are pretty complicated to set up, which means they don't get used much, which means they don't get much attention, which means they remain complicated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Jul 7 10:47:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 7 Jul 2015 09:47:47 -0500 Subject: [Numpy-discussion] Dev meeting info Message-ID: Hi all, Here's the shared document we'll be using for note-taking etc.: https://docs.google.com/document/d/1IJcYdsHtk8MVAM4AZqFDBSf_nVG-mrB4Tv2bh9u1g4Y/edit?usp=sharing If anyone wants to peek in remotely -- the conferencing equipment in this room doesn't seem to work with hangouts, but this should work: https://ucberkeley.bluejeans.com/3308071044/ (requires a free plugin but does work on Windows / OS X / 64-bit Linux / phone / whatever). I'll also try to keep an eye on https://gitter.im/numpy/numpy -n -- Nathaniel J. Smith -- http://vorpus.org From theresa.x.nguyen at lmco.com Wed Jul 8 11:53:04 2015 From: theresa.x.nguyen at lmco.com (Nguyen, Theresa X) Date: Wed, 8 Jul 2015 15:53:04 +0000 Subject: [Numpy-discussion] Numpy V1.4.1 software license In-Reply-To: <39B1787764A6BE4C899B7592A27AC56D45863C1D@HDXDSP51.us.lmco.com> References: <39B1787764A6BE4C899B7592A27AC56D45862C33@HDXDSP51.us.lmco.com> <39B1787764A6BE4C899B7592A27AC56D45863C1D@HDXDSP51.us.lmco.com> Message-ID: <39B1787764A6BE4C899B7592A27AC56D45863C58@HDXDSP51.us.lmco.com> Dear Sir/Madam, Currently, I have a chance to review the software Numpy V1.4.1, I found some software that they had the Copyright but *no License* show as below: 1. File: ... numpy-1.4.1\numpy\distutils\fcompiler\absoft.py # on windows: f90 -V -c dummy.f # f90: Copyright Absoft Corporation 1994-1998 mV2; Cray Research, Inc. 1994-1996 CF90 (2.x.x.x f36t87) Version 2.3 Wed Apr 19, 2006 13:05:16 # samt5735(8)$ f90 -V -c dummy.f # f90: Copyright Absoft Corporation 1994-2002; Absoft Pro FORTRAN Version 8.0 2. File: ... numpy-1.4.1\numpy\distutils\fcompiler\intel.py #Intel(R) Fortran Itanium(R) Compiler for Itanium(R)-based applications #Version 9.1 Build 20060928 Package ID: l_fc_c_9.1.039 #Copyright (C) 1985-2006 Intel Corporation. All rights reserved. #30 DAY EVALUATION LICENSE Would you please provide a License for these files. Numpy V1.4.1 is released under BSD Style License. I wonder if these software can be released under the same license? Thank you very much for your assistant. Best Regards, Theresa X. Nguyen -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Jul 8 11:57:28 2015 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 8 Jul 2015 11:57:28 -0400 Subject: [Numpy-discussion] Numpy V1.4.1 software license In-Reply-To: <39B1787764A6BE4C899B7592A27AC56D45863C58@HDXDSP51.us.lmco.com> References: <39B1787764A6BE4C899B7592A27AC56D45862C33@HDXDSP51.us.lmco.com> <39B1787764A6BE4C899B7592A27AC56D45863C1D@HDXDSP51.us.lmco.com> <39B1787764A6BE4C899B7592A27AC56D45863C58@HDXDSP51.us.lmco.com> Message-ID: I think there is a misunderstanding. What you are seeing is documentation on how to use f90 for compiling, which then outputs some stuff to the terminal, which is being shown in the documentation. We don't actually include any compilers in numpy. Ben Root On Wed, Jul 8, 2015 at 11:53 AM, Nguyen, Theresa X < theresa.x.nguyen at lmco.com> wrote: > Dear Sir/Madam, > > > > Currently, I have a chance to review the software Numpy > V1.4.1, I found some software that they had the Copyright but *no License* > show as below: > > > > 1. File: ? numpy-1.4.1\numpy\distutils\fcompiler\ > absoft.py > > > > # on windows: f90 -V -c dummy.f > > # f90: Copyright Absoft Corporation 1994-1998 mV2; Cray > Research, Inc. 1994-1996 CF90 (2.x.x.x f36t87) Version 2.3 Wed Apr 19, > 2006 13:05:16 > > > > # samt5735(8)$ f90 -V -c dummy.f > > # f90: Copyright Absoft Corporation 1994-2002; Absoft Pro > FORTRAN Version 8.0 > > > > 2. File: ? numpy-1.4.1\numpy\distutils\fcompiler\intel.py > > > > #Intel(R) Fortran Itanium(R) Compiler for Itanium(R)-based applications > > #Version 9.1 Build 20060928 Package ID: l_fc_c_9.1.039 > > #Copyright (C) 1985-2006 Intel Corporation. All rights reserved. > > #30 DAY EVALUATION LICENSE > > > > > > Would you please provide a License for these files. > > > > Numpy V1.4.1 is released under BSD Style License. I wonder if these > software can be released under the same license? > > > > Thank you very much for your assistant. > > > > > > Best Regards, > *Theresa X. Nguyen* > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Jul 8 11:58:55 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 8 Jul 2015 16:58:55 +0100 Subject: [Numpy-discussion] Numpy V1.4.1 software license In-Reply-To: <39B1787764A6BE4C899B7592A27AC56D45863C58@HDXDSP51.us.lmco.com> References: <39B1787764A6BE4C899B7592A27AC56D45862C33@HDXDSP51.us.lmco.com> <39B1787764A6BE4C899B7592A27AC56D45863C1D@HDXDSP51.us.lmco.com> <39B1787764A6BE4C899B7592A27AC56D45863C58@HDXDSP51.us.lmco.com> Message-ID: Hi, On Wed, Jul 8, 2015 at 4:53 PM, Nguyen, Theresa X wrote: > Dear Sir/Madam, > > > > Currently, I have a chance to review the software Numpy > V1.4.1, I found some software that they had the Copyright but *no License* > show as below: > > > > 1. File: ? numpy-1.4.1\numpy\distutils\fcompiler\absoft.py > > > > # on windows: f90 -V -c dummy.f > > # f90: Copyright Absoft Corporation 1994-1998 mV2; Cray Research, > Inc. 1994-1996 CF90 (2.x.x.x f36t87) Version 2.3 Wed Apr 19, 2006 13:05:16 > > > > # samt5735(8)$ f90 -V -c dummy.f > > # f90: Copyright Absoft Corporation 1994-2002; Absoft Pro FORTRAN > Version 8.0 > > > > 2. File: ? numpy-1.4.1\numpy\distutils\fcompiler\intel.py > > > > #Intel(R) Fortran Itanium(R) Compiler for Itanium(R)-based applications > > #Version 9.1 Build 20060928 Package ID: l_fc_c_9.1.039 > > #Copyright (C) 1985-2006 Intel Corporation. All rights reserved. > > #30 DAY EVALUATION LICENSE > > > > > > Would you please provide a License for these files. > > > > Numpy V1.4.1 is released under BSD Style License. I wonder if these > software can be released under the same license? > > > > Thank you very much for your assistant. Looking at these files, the parts you highlight look like example output from running these programs on the command line, so the copyright messages apply to those binaries, should you have them, not to numpy. Best, Matthew From olivier.grisel at ensta.org Fri Jul 10 06:24:49 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 12:24:49 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: Hi Carl, Sorry for the slow reply. I ran some tests with your binstar packages: I installed numpy, scipy and mingwpy for Python 2.7 32 bit and Python 3.4 64 bit (downloaded from python.org) on a freshly provisionned windows VM on rackspace. I then used the mingwpy C & C++ compilers to build the master branch of scikit-learn which worked without any issue. Then I ran the tests for numpy, scipy and scikit-learn Here are the results: https://gist.github.com/ogrisel/7542a56d5740100e66b9 To summarize: - there are a few numpy failures on both 32 bit and 64 bit (see gist for details) - there are scipy failures on 32 bit and a segfault on 64 bit, I installed gdb to get a not very informative backtrace - scikit-learn tests all pass on 32 bit - scikit-learn tests segfault on 64 bit (see backtrace as well). Let met know how I can help debug those. I installed gdb using msys2 for 64 bit windows but I did not have msys2 in the path when building scikit-learn. I can also grant you rackspace cloud credentials if you want to debug this by yourself. -- Oliver From olivier.grisel at ensta.org Fri Jul 10 07:40:57 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 13:40:57 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: Good news, The segfaults on scikit-lern and scipy test suites are caused by a bug in openblas core type detection: setting the OPENBLAS_CORETYPE environment variable to "Nehalem" can make the test suite complete without any failure for scikit-learn. I will update my gist with the new test results for scipy. -- Olivier From olivier.grisel at ensta.org Fri Jul 10 09:34:25 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 15:34:25 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: I have updated my gist with more test reports when OPENBLAS_CORETYPE="Nehalem" is fixed as an environment variable. Note that on this machine, OpenBLAS detects the "Barcelona" core type. I used the following ctypes based script to introspect the OpenBLAS runtime: https://gist.github.com/ogrisel/ad4e547a32d0eb18b4ff -- Olivier From cmkleffner at gmail.com Fri Jul 10 10:47:10 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 10 Jul 2015 16:47:10 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: Hi Olivier, yes, this is all explained in https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well. This seems to be necessary for CI systems, right? BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for python-2.6, 2.7, 3.3, 3.4 on anaconda.org. I also added scipy-0.16.0rc1 for python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for the future. python-3.5 support is up in the air right now. Cheers Carl 2015-07-10 15:34 GMT+02:00 Olivier Grisel : > I have updated my gist with more test reports when > OPENBLAS_CORETYPE="Nehalem" is fixed as an environment variable. > > Note that on this machine, OpenBLAS detects the "Barcelona" core type. > I used the following ctypes based script to introspect the OpenBLAS > runtime: > > https://gist.github.com/ogrisel/ad4e547a32d0eb18b4ff > > -- > Olivier > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Fri Jul 10 11:48:15 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 17:48:15 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: I narrowed down the segfault from the scipy tests on my machine to: OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))" Barcelona is the architecture detected by OpenBLAS. If I force Nehalem or if I reduce the matrix to shape (128, 128), it no longer segfaults. To debug it further I think I have to write a test program in C that uses the sgesdd function of the libopenblaspy.dll or against the official libopenblas.dll. Setting the dev environment under windows is painful though. I don't know if it's a bug in OpenBLAS' coretype detection or in the SGESDD kernel for the Barcelona architecture. -- Olivier From njs at pobox.com Fri Jul 10 12:31:42 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 10 Jul 2015 11:31:42 -0500 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: On Jul 10, 2015 10:51 AM, "Olivier Grisel" wrote: > > I narrowed down the segfault from the scipy tests on my machine to: > > OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy > as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))" > > Barcelona is the architecture detected by OpenBLAS. If I force Nehalem > or if I reduce the matrix to shape (128, 128), it no longer segfaults. > To debug it further I think I have to write a test program in C that > uses the sgesdd function of the libopenblaspy.dll or against the > official libopenblas.dll. Setting the dev environment under windows is > painful though. I assume you've already checked that this is a Windows specific issue? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Fri Jul 10 14:20:01 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 10 Jul 2015 20:20:01 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: I could provide you with a debug build of libopenblaspy.dll. The segfault - if ithrown from openblas - could be detected with gdb or with the help of backtrace.dll. Carl 2015-07-10 18:31 GMT+02:00 Nathaniel Smith : > On Jul 10, 2015 10:51 AM, "Olivier Grisel" > wrote: > > > > I narrowed down the segfault from the scipy tests on my machine to: > > > > OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy > > as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))" > > > > Barcelona is the architecture detected by OpenBLAS. If I force Nehalem > > or if I reduce the matrix to shape (128, 128), it no longer segfaults. > > To debug it further I think I have to write a test program in C that > > uses the sgesdd function of the libopenblaspy.dll or against the > > official libopenblas.dll. Setting the dev environment under windows is > > painful though. > > I assume you've already checked that this is a Windows specific issue? > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewm at redtetrahedron.org Fri Jul 10 14:30:40 2015 From: ewm at redtetrahedron.org (Eric Moore) Date: Fri, 10 Jul 2015 14:30:40 -0400 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: It looks like all of the numpy failures there are due to a poor implementation of hypot. One solution would be to force the use of the hypot code in npymath for this tool chain. Currently this is done in numpy/core/src/private/npy_config.h for both MSVC and mingw32. -Eric On Fri, Jul 10, 2015 at 7:40 AM, Olivier Grisel wrote: > Good news, > > The segfaults on scikit-lern and scipy test suites are caused by a bug > in openblas core type detection: setting the OPENBLAS_CORETYPE > environment variable to "Nehalem" can make the test suite complete > without any failure for scikit-learn. > > I will update my gist with the new test results for scipy. > > -- > Olivier > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Fri Jul 10 12:42:13 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 18:42:13 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 18:31 GMT+02:00 Nathaniel Smith : > On Jul 10, 2015 10:51 AM, "Olivier Grisel" wrote: >> >> I narrowed down the segfault from the scipy tests on my machine to: >> >> OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy >> as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))" >> >> Barcelona is the architecture detected by OpenBLAS. If I force Nehalem >> or if I reduce the matrix to shape (128, 128), it no longer segfaults. >> To debug it further I think I have to write a test program in C that >> uses the sgesdd function of the libopenblaspy.dll or against the >> official libopenblas.dll. Setting the dev environment under windows is >> painful though. > > I assume you've already checked that this is a Windows specific issue? I am starting a rackspace VM with linux to check. Hopefully it will also be detected as Barcelona by openblas. On my laptop I have a Sandybridge coretype (I just checked). -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From olivier.grisel at ensta.org Fri Jul 10 13:38:51 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 19:38:51 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 18:42 GMT+02:00 Olivier Grisel : > >> I assume you've already checked that this is a Windows specific issue? > > I am starting a rackspace VM with linux to check. Hopefully it will > also be detected as Barcelona by openblas. I just built OpenBLAS 0.2.14 and numpy 1.9.2 under Linux on a rackspace VM (2GB Standard Instance) that gets detected as "Barcelona" by OpenBLAS and I cannot reproduce the issue. So this is either: - windows specific - mingpy specific (as it was used to build numpy and potentially the embedded libopenblaspy.dll as well) - specific to an old version of OpenBLAS and fixed in 0.2.14 if Carl has used an old version Note bigger rackspace VMs get detected as Nehalem. So use 2GB Standard Instance if you want a Barcelona machine. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From olivier.grisel at ensta.org Fri Jul 10 13:06:21 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 10 Jul 2015 19:06:21 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 16:47 GMT+02:00 Carl Kleffner : > Hi Olivier, > > yes, this is all explained in > https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well. > This seems to be necessary for CI systems, right? The auto detection should work. If not it's a bug and we should find a minimalistic reproduction case to report it to the openblas maintainers. Or we could choose to build openblas for an old architecture that should work everywhere like nehalem for instance. > BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for > python-2.6, 2.7, 3.3, 3.4 on anaconda.org. You mean binstar.org right? What did you change in the new numpy-1.9.2? Have you upgraded OpenBLAS? Which version is embedded by the way? > I also added scipy-0.16.0rc1 for > python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for the > future. I don't think Windows users care about python 3.2 support. > python-3.5 support is up in the air right now. What do you mean? -- Olivier From cmkleffner at gmail.com Fri Jul 10 16:13:35 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Fri, 10 Jul 2015 22:13:35 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 19:06 GMT+02:00 Olivier Grisel : > 2015-07-10 16:47 GMT+02:00 Carl Kleffner : > > Hi Olivier, > > > > yes, this is all explained in > > https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as > well. > > This seems to be necessary for CI systems, right? > > The auto detection should work. If not it's a bug and we should find a > minimalistic reproduction case to report it to the openblas > maintainers. > > I have a debug version of libopenblaspy.dll at hand for 32bit architecture, 64bit I have to build. I propose to build small testcases with python, as @wernsaar, the main developer of the newer OpenBLAS kernels is able to handle this. Debug builds of libopenblaspy.dll can be used as drop in replacement and can be used to emit a stack trace as long the segfault is thrown from numpy or OpenBLAS code. DrMingw may be able to catch segfaults from either, I havn't tried this yet. > Or we could choose to build openblas for an old architecture that > should work everywhere like nehalem for instance. > This is a safe solution. Or one could leave out all kernels above sandybridge in a dynamic build as long OpenBLAS has errors with newer kernels. > > > BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for > > python-2.6, 2.7, 3.3, 3.4 on anaconda.org. > > The best technical solution IMHO would be to make an extra mingwpy.openblas python package to load libopenblaspy.dll into the process space, as this package could be upgraded independant from numpy/scipy. On the other side this would mean an additionally package dependancy for numpy at least on the windows platform. > You mean binstar.org right? What did you change in the new > numpy-1.9.2? Have you upgraded OpenBLAS? Which version is embedded by > the way? > anaconda.org is the new replacement for binstar.org as announced in the anaconda ml and should be used right now. Openblas versions contained: 32bit: 3e33afe (june) 64bit: fb02cb0 (april used due to a segfault in the more recent rev) > > I also added scipy-0.16.0rc1 for > > python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for > the > > future. > > I don't think Windows users care about python 3.2 support. > > > python-3.5 support is up in the air right now. > > What do you mean? > > Python-3.5 uses VC 2105 on windows. This compiler uses a complete new variant of CRT libraries. Google for 'universal CRT' to find more information. The mingw-w64 community does not support this right now. I didn't find the time to explore this. Carl -- > Olivier > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Sat Jul 11 12:30:21 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Sat, 11 Jul 2015 18:30:21 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 20:20 GMT+02:00 Carl Kleffner : > I could provide you with a debug build of libopenblaspy.dll. The segfault - > if ithrown from openblas - could be detected with gdb or with the help of > backtrace.dll. That would be great thanks. Also can you give the build options / instructions you used to build openblas with mingwpy? Also which version of openblas did you use? The last stable or some snapshot from the develop branch? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From olivier.grisel at ensta.org Sat Jul 11 12:33:57 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Sat, 11 Jul 2015 18:33:57 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-11 18:30 GMT+02:00 Olivier Grisel : > 2015-07-10 20:20 GMT+02:00 Carl Kleffner : >> I could provide you with a debug build of libopenblaspy.dll. The segfault - >> if ithrown from openblas - could be detected with gdb or with the help of >> backtrace.dll. > > That would be great thanks. Also can you give the build options / > instructions you used to build openblas with mingwpy? Also which > version of openblas did you use? The last stable or some snapshot from > the develop branch? I just saw you replied to the question on the version number in your last email. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From olivier.grisel at ensta.org Sat Jul 11 13:19:11 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Sat, 11 Jul 2015 19:19:11 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-10 22:13 GMT+02:00 Carl Kleffner : > > > 2015-07-10 19:06 GMT+02:00 Olivier Grisel : >> >> 2015-07-10 16:47 GMT+02:00 Carl Kleffner : >> > Hi Olivier, >> > >> > yes, this is all explained in >> > https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as >> > well. >> > This seems to be necessary for CI systems, right? >> >> The auto detection should work. If not it's a bug and we should find a >> minimalistic reproduction case to report it to the openblas >> maintainers. >> > I have a debug version of libopenblaspy.dll at hand for 32bit architecture, > 64bit I have to build. I propose to build small testcases with python, as > @wernsaar, the main developer of the newer OpenBLAS kernels is able to > handle this. Done here, feel free to join the discussion: https://github.com/xianyi/OpenBLAS/issues/603 > Debug builds of libopenblaspy.dll can be used as drop in replacement and can > be used to emit a stack trace as long the segfault is thrown from numpy or > OpenBLAS code. DrMingw may be able to catch segfaults from either, I havn't > tried this yet. > >> >> Or we could choose to build openblas for an old architecture that >> should work everywhere like nehalem for instance. > This is a safe solution. Is Nehalem the most common denominator? > Or one could leave out all kernels above > sandybridge in a dynamic build as long OpenBLAS has errors with newer > kernels. Yes that's an option. How do you do that? I cannot find such a build option in the documentation. > The best technical solution IMHO would be to make an extra mingwpy.openblas > python package to load libopenblaspy.dll into the process space, as this > package could be upgraded independant from numpy/scipy. On the other side > this would mean an additionally package dependancy for numpy at least on the > windows platform. Having a windows only dependency on such a wheel sounds fine with me. However that would mean to ensure that the dll is properly loaded in the windows dll search path prior to importing extensions such as numpy.core._dotblas right? Could this be done via adding something to `numpy.__init__.py` such as: ``` if sys.platform == 'win32': try: # Ensure that libopenblaspy.dll can be found prior to # loading numpy.core._dotblas from mingwpy import openblas except: warnings.warn('Failed to load mingwpy's openblas') # import numpy.core stuff here ``` > anaconda.org is the new replacement for binstar.org as announced in the > anaconda ml and should be used right now. Ok I did not know. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From cmkleffner at gmail.com Sun Jul 12 04:04:33 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sun, 12 Jul 2015 10:04:33 +0200 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: 2015-07-11 19:19 GMT+02:00 Olivier Grisel : > 2015-07-10 22:13 GMT+02:00 Carl Kleffner : > > > > > > 2015-07-10 19:06 GMT+02:00 Olivier Grisel : > >> > >> 2015-07-10 16:47 GMT+02:00 Carl Kleffner : > >> > Hi Olivier, > >> > > >> > yes, this is all explained in > >> > https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as > >> > well. > >> > This seems to be necessary for CI systems, right? > >> > >> The auto detection should work. If not it's a bug and we should find a > >> minimalistic reproduction case to report it to the openblas > >> maintainers. > >> > > I have a debug version of libopenblaspy.dll at hand for 32bit > architecture, > > 64bit I have to build. I propose to build small testcases with python, as > > @wernsaar, the main developer of the newer OpenBLAS kernels is able to > > handle this. > > Done here, feel free to join the discussion: > > https://github.com/xianyi/OpenBLAS/issues/603 > > > Debug builds of libopenblaspy.dll can be used as drop in replacement and > can > > be used to emit a stack trace as long the segfault is thrown from numpy > or > > OpenBLAS code. DrMingw may be able to catch segfaults from either, I > havn't > > tried this yet. > > > >> > >> Or we could choose to build openblas for an old architecture that > >> should work everywhere like nehalem for instance. > > > This is a safe solution. > > Is Nehalem the most common denominator? > > I think it is PRESCOTT. BTW: For now you could use now OPTERON_SSE3 or OPTERON instead of NEHALEM, as these are AMD kernels without the barcelona assembler part. > > Or one could leave out all kernels above > > sandybridge in a dynamic build as long OpenBLAS has errors with newer > > kernels. > > Yes that's an option. How do you do that? I cannot find such a build > option in the documentation. > > Just patches to be include as an option later on. > > The best technical solution IMHO would be to make an extra > mingwpy.openblas > > python package to load libopenblaspy.dll into the process space, as this > > package could be upgraded independant from numpy/scipy. On the other side > > this would mean an additionally package dependancy for numpy at least on > the > > windows platform. > > Having a windows only dependency on such a wheel sounds fine with me. > However that would mean to ensure that the dll is properly loaded in > the windows dll search path prior to importing extensions such as > numpy.core._dotblas right? Could this be done via adding something to > `numpy.__init__.py` such as: > > ``` > if sys.platform == 'win32': > try: > # Ensure that libopenblaspy.dll can be found prior to > # loading numpy.core._dotblas > from mingwpy import openblas > except: > warnings.warn('Failed to load mingwpy's openblas') > > # import numpy.core stuff here > ``` > > Or equip numpy and scipy with the DLL as a fallback, if no 'package' for OpenBLAS is installed. With an additional OpenBLAS package you then can augument the numpy OpenBLAS with a newer one. > > anaconda.org is the new replacement for binstar.org as announced in the > > anaconda ml and should be used right now. > > Ok I did not know. > > -- > Olivier > http://twitter.com/ogrisel - http://github.com/ogrisel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jul 12 09:34:25 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 12 Jul 2015 08:34:25 -0500 Subject: [Numpy-discussion] Video meeting this week In-Reply-To: References: Message-ID: On Jul 11, 2015 12:22 PM, "Olivier Grisel" wrote: > > 2015-07-10 22:13 GMT+02:00 Carl Kleffner : > > > The best technical solution IMHO would be to make an extra mingwpy.openblas > > python package to load libopenblaspy.dll into the process space, as this > > package could be upgraded independant from numpy/scipy. On the other side > > this would mean an additionally package dependancy for numpy at least on the > > windows platform. > > Having a windows only dependency on such a wheel sounds fine with me. > However that would mean to ensure that the dll is properly loaded in > the windows dll search path prior to importing extensions such as > numpy.core._dotblas right? Could this be done via adding something to > `numpy.__init__.py` such as: > > ``` > if sys.platform == 'win32': > try: > # Ensure that libopenblaspy.dll can be found prior to > # loading numpy.core._dotblas > from mingwpy import openblas > except: > warnings.warn('Failed to load mingwpy's openblas') > > # import numpy.core stuff here > ``` We might even just want to add to numpy/__init__.py something like try: import .distributor_hook except ImportError: pass so wheels (and conda builds, etc.) could be patched by adding a single file without modifying any upstream files. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at ericmart.in Mon Jul 13 02:40:47 2015 From: eric at ericmart.in (Eric Martin) Date: Mon, 13 Jul 2015 01:40:47 -0500 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions Message-ID: Hi, I've been playing around recently with linking Numpy to different BLAS implementations, particularly Eigen and ACML 6 (with openCL support!). I've successfully linked Numpy to both of these libraries, but I found the process overly difficult and confusing. I'm interested in either writing a blog post or adding to Numpy docs to make it easier for others, but I'm hoping to clear up some of my own confusions first. I'll start with a rough outline of what I did to link Numpy with Eigen and ACML. *(1)* Modify numpy/core/setup.py. Change - blas_info = get_info('blas_opt', 0) + blas_info = get_info('blas', 0) and changing get_dotblas_sources to def get_dotblas_sources(ext, build_dir): if blas_info return ext.depends[:3] return None (to remove the check for ('NO_ATLAS_INFO', 1)). *(2)* Compile CBLAS with BLLIB in Makefile.in pointing to the shared object for your BLAS. Make a shared object (not a static library) out of CBLAS. This requires adding -fPIC to the CFLAGS and FFLAGS. *Question: Is it a bug that I couldn't get Numpy working with a static CBLAS library and a shared object BLAS?* *(3)* Modify site.cfg at the top level of the Numpy directory with [blas] library_dirs = /path/to/directory/containing/shared_objects include_dirs = /path/to/headers/from/CBLAS blas_libs = cblas, your_blas_lib where there headers from CBLAS are cblas_f77.h and cblas.h. For the blas_libs variable, the library name "foo" loads libfoo.so, so with the above example the libraries should be called libcblas.so and libyour_blas_lib.so and lie in the listed library_dir. Finally, run "python setup.py build" from the root of the Numpy codebase (same directory that site.cfg lives in). *My questions about this:* CBLAS questions: What does CBLAS do, and why/when is it necessary? For both ACML 6 and Eigen, I could not link directly to the library but could with CBLAS. My understanding is that the BLAS interface is a Fortran ABI, and the CBLAS provides a C ABI (cdecl?) to BLAS. Why can't the user link Numpy directly to the Fortran ABI? How are ATLAS and openBLAS handled? My procedure questions: Is the installation procedure I outlined above reasonable, or does it contain steps that could/should be removed? Having to edit Numpy source seems sketchy to me. I largely came up with this procedure by looking up tutorials online and by trial and error. I don't want to write documentation that encourages people to do something in a non-optimal way, so if there is a better way to do this, please let me know. *Some final thoughts:* Although I linked properly to the library, I discovered ACML 6 didn't work at all on my computer (the ACML6 example code didn't even work). This is very disappointed, as openCL support in ACML 6 + integrated GPU on laptop + openCL on Intel integrated GPUs on Linux through beignet seemed like a potentially very promising performance boost for all of us running Numpy on newer laptops. Eigen has excellent performance. On my i5-5200U (Broadwell) CPU, I found Eigen BLAS compiled with AVX and FMA instructions to take 3.93s to multiply 2 4000x4000 double matrices with a single thread, while my install of Numpy from ubuntu took 9s (and used 4 threads on my 2 cores). My Ubuntu numpy appears to built against "libblas", which I think is the reference implementation. Eigen gave 32GFLOPS of 64 bit performance from a single laptop core, I find this quite impressive! Thanks for any feedback and response to the questions! -Eric Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Jul 13 07:34:27 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 13 Jul 2015 07:34:27 -0400 Subject: [Numpy-discussion] dimension independent copy of corner of array Message-ID: I want to copy an array to the corner of a new array. What is a dimension independent way to say: newarr[:i0,:i1,...] = oldarray where (i0,i1...) is oldarray.shape? From robert.kern at gmail.com Mon Jul 13 07:38:37 2015 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 13 Jul 2015 12:38:37 +0100 Subject: [Numpy-discussion] dimension independent copy of corner of array In-Reply-To: References: Message-ID: newarr[tuple(slice(0, i) for i in oldarray.shape)] = oldarray On Mon, Jul 13, 2015 at 12:34 PM, Neal Becker wrote: > I want to copy an array to the corner of a new array. What is a dimension > independent way to say: > > newarr[:i0,:i1,...] = oldarray > > where (i0,i1...) is oldarray.shape? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Jul 13 08:21:34 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 13 Jul 2015 08:21:34 -0400 Subject: [Numpy-discussion] dimension independent copy of corner of array References: Message-ID: Robert Kern wrote: > newarr[tuple(slice(0, i) for i in oldarray.shape)] = oldarray > > On Mon, Jul 13, 2015 at 12:34 PM, Neal Becker wrote: > >> I want to copy an array to the corner of a new array. What is a >> dimension independent way to say: >> >> newarr[:i0,:i1,...] = oldarray >> >> where (i0,i1...) is oldarray.shape? >> Thanks. I'm making my own class that represents a resizeable array that works by preserving the lower corner of the old data, because I believe that ndarray arr.resize does not actually work this way. I'm only interested in resizing to a larger size. Is this correct? From njs at pobox.com Mon Jul 13 09:54:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Jul 2015 08:54:50 -0500 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: On Jul 13, 2015 1:44 AM, "Eric Martin" wrote: > > My procedure questions: > Is the installation procedure I outlined above reasonable, or does it contain steps that could/should be removed? Having to edit Numpy source seems sketchy to me. I largely came up with this procedure by looking up tutorials online and by trial and error. I don't want to write documentation that encourages people to do something in a non-optimal way, so if there is a better way to do this, please let me know. I'll leave others with more knowledge to answer your other questions, but if you're interested in making it easy for others to link numpy against these libraries, I'd suggest modifying the numpy source further, by submitting a patch that teaches numpy.distutils how to detect and link against these libraries automatically :-). There are several libraries we already know how to do this for that you can compare against for reference, and this will also help for other libraries that use blas and numpy.distutils, like scipy. > Eigen has excellent performance. On my i5-5200U (Broadwell) CPU, I found Eigen BLAS compiled with AVX and FMA instructions to take 3.93s to multiply 2 4000x4000 double matrices with a single thread, while my install of Numpy from ubuntu took 9s (and used 4 threads on my 2 cores). My Ubuntu numpy appears to built against "libblas", which I think is the reference implementation. If you're using the numpy packages distributed by Ubuntu, then it should be possible to switch to openblas just by apt installing openblas and then maybe fiddling with update-alternatives. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.villellas at continuum.io Mon Jul 13 11:07:20 2015 From: oscar.villellas at continuum.io (Oscar Villellas) Date: Mon, 13 Jul 2015 17:07:20 +0200 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: On Mon, Jul 13, 2015 at 8:40 AM, Eric Martin wrote: > [...] > > *My questions about this:* > CBLAS questions: > What does CBLAS do, and why/when is it necessary? For both ACML 6 and > Eigen, I could not link directly to the library but could with CBLAS. My > understanding is that the BLAS interface is a Fortran ABI, and the CBLAS > provides a C ABI (cdecl?) to BLAS. > Why can't the user link Numpy directly to the Fortran ABI? How are ATLAS > and openBLAS handled? > > AFAIK, the cblas symbols is an interface adapted to the calling conventions of C. This involves things like integer being passed by value instead of by reference. This means that in order to use the FORTRAN symbol care must be taken to pass the arguments the way a FORTRAN function expects (like I said, it is a matter of just tweaking the arguments in the call and using a FORTRAN symbol). -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Mon Jul 13 11:32:36 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Mon, 13 Jul 2015 15:32:36 +0000 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: On Mon, Jul 13, 2015 at 7:55 AM Nathaniel Smith wrote: > On Jul 13, 2015 1:44 AM, "Eric Martin" wrote: > > > > My procedure questions: > > Is the installation procedure I outlined above reasonable, or does it > contain steps that could/should be removed? Having to edit Numpy source > seems sketchy to me. I largely came up with this procedure by looking up > tutorials online and by trial and error. I don't want to write > documentation that encourages people to do something in a non-optimal way, > so if there is a better way to do this, please let me know. > > I'll leave others with more knowledge to answer your other questions, but > if you're interested in making it easy for others to link numpy against > these libraries, I'd suggest modifying the numpy source further, by > submitting a patch that teaches numpy.distutils how to detect and link > against these libraries automatically :-). There are several libraries we > already know how to do this for that you can compare against for reference, > and this will also help for other libraries that use blas and > numpy.distutils, like scipy. > Supporting Eigen sounds like a great idea. BLIS would be another one worth supporting at some point. As far as implementation goes, it may be helpful to look at https://github.com/numpy/numpy/pull/3642 and https://github.com/numpy/numpy/pull/4191 for the corresponding set of changes for OpenBLAS. That could be a good starting point. Thanks for bringing this up! -Ian Henriksen > > Eigen has excellent performance. On my i5-5200U (Broadwell) CPU, I > found Eigen BLAS compiled with AVX and FMA instructions to take 3.93s to > multiply 2 4000x4000 double matrices with a single thread, while my install > of Numpy from ubuntu took 9s (and used 4 threads on my 2 cores). My Ubuntu > numpy appears to built against "libblas", which I think is the reference > implementation. > > If you're using the numpy packages distributed by Ubuntu, then it should > be possible to switch to openblas just by apt installing openblas and then > maybe fiddling with update-alternatives. > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at ericmart.in Mon Jul 13 12:44:33 2015 From: eric at ericmart.in (Eric Martin) Date: Mon, 13 Jul 2015 11:44:33 -0500 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: Nathaniel and Ian both mention adding code to Numpy to explicitly support Eigen. It seems to me that a potentially better route than "add code to Numpy to support BLAS library" for each library is to make Numpy easy to configure to compile with an arbitrary BLAS library (like what I've been doing). Reasoning behind this: * numpy.distutils.system_info is already 2300 lines of code. This code (and the full process of linking to BLAS) are not very well documented or flexible, as evidenced by the trouble I had linking against custom BLAS and the fact that changes to numpy's source were required. Adding more code to this seems to make things worse rather than better in my opinion. * I think the "support arbitrary BLAS library" option is better plane on the "ease of use" to "robustness" than adding support for particular libraries. * BLAS is an interface. Shouldn't we (as Numpy developers) take full advantage of the interface and the generality it allows? As I mentioned in my first email, the current user experience of linking Numpy to arbitrary BLAS consists of (0) compile/acquire BLAS binary, (1) modify Numpy source, (2) compile CBLAS against your BLAS, and (3) modify site.cfg and build. Step 0 is outside the scope of Numpy, and none of steps 1-3 were very well documented. I think the ideal user experience would consist solely of steps 0 and 3, that is just acquiring a BLAS and then configuring Numpy to build against that BLAS, and then building Numpy. I'm not sure if it's technically possible, but it would be an improvement on this if the user could change the BLAS for a numpy install without rebuilding (ie Numpy optionally loads some site.cfg equivalent each time it opens and dynamically links based off of that, or something clever with symlinks is done). I believe this is technically possible with symlinks (see Ubuntu's update-alternatives program). I realize this would require a large rewrite of numpy.distutils, but this approach seems far simpler. The building code could do something like: does the user specify a BLAS? -> is ATLAS installed? -> is openBLAS installed? -> use some default. The same approach could be applied to LAPACK and FFT libraries (even though FFT might be a bit trickier since the interface is less standard). Thoughts on this general approach? -Eric On Mon, Jul 13, 2015 at 10:32 AM, Ian Henriksen < insertinterestingnamehere at gmail.com> wrote: > On Mon, Jul 13, 2015 at 7:55 AM Nathaniel Smith wrote: > >> On Jul 13, 2015 1:44 AM, "Eric Martin" wrote: >> > >> > My procedure questions: >> > Is the installation procedure I outlined above reasonable, or does it >> contain steps that could/should be removed? Having to edit Numpy source >> seems sketchy to me. I largely came up with this procedure by looking up >> tutorials online and by trial and error. I don't want to write >> documentation that encourages people to do something in a non-optimal way, >> so if there is a better way to do this, please let me know. >> >> I'll leave others with more knowledge to answer your other questions, but >> if you're interested in making it easy for others to link numpy against >> these libraries, I'd suggest modifying the numpy source further, by >> submitting a patch that teaches numpy.distutils how to detect and link >> against these libraries automatically :-). There are several libraries we >> already know how to do this for that you can compare against for reference, >> and this will also help for other libraries that use blas and >> numpy.distutils, like scipy. >> > Supporting Eigen sounds like a great idea. BLIS would be another > one worth supporting at some point. As far as implementation goes, it may > be helpful to look at https://github.com/numpy/numpy/pull/3642 and > https://github.com/numpy/numpy/pull/4191 for the corresponding set of > changes for OpenBLAS. That could be a good starting point. > Thanks for bringing this up! > -Ian Henriksen > >> > Eigen has excellent performance. On my i5-5200U (Broadwell) CPU, I >> found Eigen BLAS compiled with AVX and FMA instructions to take 3.93s to >> multiply 2 4000x4000 double matrices with a single thread, while my install >> of Numpy from ubuntu took 9s (and used 4 threads on my 2 cores). My Ubuntu >> numpy appears to built against "libblas", which I think is the reference >> implementation. >> >> If you're using the numpy packages distributed by Ubuntu, then it should >> be possible to switch to openblas just by apt installing openblas and then >> maybe fiddling with update-alternatives. >> >> -n >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jul 13 13:08:08 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Jul 2015 12:08:08 -0500 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: On Jul 13, 2015 11:48 AM, "Eric Martin" wrote: > > Nathaniel and Ian both mention adding code to Numpy to explicitly support Eigen. > > It seems to me that a potentially better route than "add code to Numpy to support BLAS library" for each library is to make Numpy easy to configure to compile with an arbitrary BLAS library (like what I've been doing). I think that if you can make this work then it would be great (who doesn't like less code that does more?), but that as a practical matter accomplishing this may be difficult. AFAIK there simply is no way to write generic code to probe the system for "any BLAS", except by explicitly having a list of supported BLASes and checking for each in sequence. Given that new BLAS libraries appear at a rate of < 1/year, and that we already support most of the viable ones, the current approach is practically reasonable even if non-ideal in principle. > Reasoning behind this: > * numpy.distutils.system_info is already 2300 lines of code. This code (and the full process of linking to BLAS) are not very well documented or flexible, as evidenced by the trouble I had linking against custom BLAS and the fact that changes to numpy's source were required. Adding more code to this seems to make things worse rather than better in my opinion. > > * I think the "support arbitrary BLAS library" option is better plane on the "ease of use" to "robustness" than adding support for particular libraries. > > * BLAS is an interface. Shouldn't we (as Numpy developers) take full advantage of the interface and the generality it allows? > > As I mentioned in my first email, the current user experience of linking Numpy to arbitrary BLAS consists of (0) compile/acquire BLAS binary, (1) modify Numpy source, (2) compile CBLAS against your BLAS, and (3) modify site.cfg and build. Step 0 is outside the scope of Numpy, and none of steps 1-3 were very well documented. I think the ideal user experience would consist solely of steps 0 and 3, that is just acquiring a BLAS and then configuring Numpy to build against that BLAS, and then building Numpy. I'm not sure if it's technically possible, but it would be an improvement on this if the user could change the BLAS for a numpy install without rebuilding (ie Numpy optionally loads some site.cfg equivalent each time it opens and dynamically links based off of that, or something clever with symlinks is done). I believe this is technically possible with symlinks (see Ubuntu's update-alternatives program). Keep in mind that any solution needs to support weird systems too, including Windows. I'm not sure we can assume that all BLAS libraries are ABI compatible either. Debian/Ubuntu make sure that this is true for the ones they ship, but not all systems are so carefully managed. For LAPACK in particular I'm pretty sure there are multiple ABIs in active use that scipy supports via code in numpy.distutils. (Hopefully someone with more expertise will speak up.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Mon Jul 13 13:52:05 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Mon, 13 Jul 2015 17:52:05 +0000 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: On Mon, Jul 13, 2015 at 11:08 AM Nathaniel Smith wrote: > I think that if you can make this work then it would be great (who doesn't > like less code that does more?), but that as a practical matter > accomplishing this may be difficult. AFAIK there simply is no way to write > generic code to probe the system for "any BLAS", except by explicitly > having a list of supported BLASes and checking for each in sequence. Given > that new BLAS libraries appear at a rate of < 1/year, and that we already > support most of the viable ones, the current approach is practically > reasonable even if non-ideal in principle. > Agreed. From my experience, the existing BLAS and LAPACK implementations are only mostly interchangeable. There may be some small differences between the libraries that will need some special casing. On the other hand, if you can get it to work properly, go for it. Some generalization and simplification is probably possible. > Keep in mind that any solution needs to support weird systems too, > including Windows. I'm not sure we can assume that all BLAS libraries are > ABI compatible either. Debian/Ubuntu make sure that this is true for the > ones they ship, but not all systems are so carefully managed. For LAPACK in > particular I'm pretty sure there are multiple ABIs in active use that scipy > supports via code in numpy.distutils. (Hopefully someone with more > expertise will speak up.) > Yes, there are ABI patches in SciPy that put together a uniform ABI for all currently supported BLAS and LAPACK libraries ( https://github.com/scipy/scipy/tree/master/scipy/_build_utils). With a little luck, some of these other libraries won't need as much special casing, but, I don't know offhand. -Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Jul 13 17:51:57 2015 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Jul 2015 00:51:57 +0300 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: 13.07.2015, 19:44, Eric Martin kirjoitti: > It seems to me that a potentially better route than "add code to Numpy to > support BLAS library" for each library is to make Numpy easy to configure > to compile with an arbitrary BLAS library (like what I've been doing). Does this work: export ATLAS=None export BLAS=/path/to/libblas.a export LAPACK=/path/to/liblapack.a python setup.py build From pav at iki.fi Mon Jul 13 19:28:32 2015 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Jul 2015 02:28:32 +0300 Subject: [Numpy-discussion] Numpy, BLAS, and CBLAS questions In-Reply-To: References: Message-ID: 13.07.2015, 20:08, Nathaniel Smith kirjoitti: [clip] > Keep in mind that any solution needs to support weird systems too, > including Windows. I'm not sure we can assume that all BLAS libraries are > ABI compatible either. Debian/Ubuntu make sure that this is true for the > ones they ship, but not all systems are so carefully managed. For LAPACK in > particular I'm pretty sure there are multiple ABIs in active use that scipy > supports via code in numpy.distutils. (Hopefully someone with more > expertise will speak up.) On the ABI issue: there are two ABI issues to solve: 1. BLAS/LAPACK ABI <--> Fortran compiler ABI 2. Fortran compiler ABI <--> C Issue 1 is simple for users who have configured their compiler environment so that the fortran/BLAS/LAPACK chain is all ABI compatible; ACML and MKL provide variants for different Fortran compilers. However, on OSX the available Fortran compiler usually (gfortran) is not ABI compatible with the system BLAS/LAPACK (Accelerate). Issue 1+2 is something that is solved by the likes of CBLAS/LAPACKE. Assuming an user who has properly configured Fortran/BLAS/LAPACK/CBLAS/LAPACKE environment, the ABI issues are not a problem that we would need to care about. However, especially on OSX, providing a painless out-of-the-box experience is in conflict with this assumption. *** Examples of Fortran ABIs in the different libraries: Eigen seems to assume gfortran ABI, not sure. OSX Accelerate always uses g77 ABI (modulo bugs). OpenBLAS can be either AFAIK depending on its configuration. MKL provides several ABI variants for different compilers, with different library names. ACML seems to do so too. Both BLAS and LAPACK have functions whose proper call convention in the Fortran interface varies in g77 vs. gfortran. The part that doesn't vary is often considered (eg. by f2py and many others) safe to call directly from C modulo name mangling, although nothing strictly speaking guarantees this. Numpy assumes Fortran ABI -> C compatibility is handled by CBLAS. Numpy only uses LAPACK functions with "safe" calling conventions, so LAPACKE is not really required. If CBLAS/LAPACKE are not used, then you have to implement it yourself. (i) One option is to rely on a Fortran compiler, assume it is ABI compatible with the target BLAS/LAPACK (not necessarily true on OSX), and then connect to C either relying on the "safe" calling conventions or Fortran2003 features. (ii) Alternatively, you can detect which BLAS/LAPACK ABI variant is active, and select a suitable C shim. Scipy uses a combination of both approaches, using a shim layer for the "problematic functions" in all code (C and Fortran). The shim is configured based on which BLAS seems to be active at compile time. If none of the recognized special cases (eg. OSX Accelerate) apply, then it is assumed the Fortran ABI provided by the selected LAPACK/BLAS is compatible with the selected Fortran compiler. (Communication to Fortran is done via the "safe" calling convention subset.) From ndbecker2 at gmail.com Tue Jul 14 07:19:21 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Jul 2015 07:19:21 -0400 Subject: [Numpy-discussion] not inheriting from ndarray Message-ID: I wanted the function of an array that accumulates my results, it starts at zero size, and resizes as needed. New results are added using accumulated += new_array A simple implementation of this is here: https://gist.github.com/2ab48e25fd460990d045.git I have 2 questions: 1. Is this a reasonable approach? 2. I overloaded += to do 1) resize 2) call ndarray += operator Is there some way to just overload all arithmetic operators similarly all at the same time? (I don't need them for the current work, I'm just curious). From ndbecker2 at gmail.com Tue Jul 14 08:39:39 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Jul 2015 08:39:39 -0400 Subject: [Numpy-discussion] not inheriting from ndarray References: Message-ID: Neal Becker wrote: > I wanted the function of an array that accumulates my results, it starts > at > zero size, and resizes as needed. New results are added using > > accumulated += new_array > > A simple implementation of this is here: > > https://gist.github.com/2ab48e25fd460990d045.git > > I have 2 questions: > > 1. Is this a reasonable approach? > > 2. I overloaded += to do 1) resize 2) call ndarray += operator > Is there some way to just overload all arithmetic operators similarly all > at > the same time? (I don't need them for the current work, I'm just > curious). Well one thing, due to (what IMO is) a bug in pickle, https://bugs.python.org/issue5370 you need to modify, this seems to work: def __getattr__(self, name): # delegate all operations not specifically overridden to base array if 'arr' in self.__dict__: # this is to allow unpickle avoid infinite recursion return getattr (self.arr, name) raise AttributeError(name) From efiring at hawaii.edu Tue Jul 14 15:55:45 2015 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 14 Jul 2015 09:55:45 -1000 Subject: [Numpy-discussion] future of f2py and Fortran90+ Message-ID: <55A56941.1050209@hawaii.edu> F2py is a great tool, but my impression is that it is being left behind by the evolution of Fortran from F90 onward. This is unfortunate; it would be nice to be able to easily wrap new Fortran libraries. I'm curious: has anyone been looking into what it would take to enable f2py to handle modern Fortran in general? And into prospects for getting such an effort funded? Eric From sturla.molden at gmail.com Tue Jul 14 21:45:40 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 15 Jul 2015 01:45:40 +0000 (UTC) Subject: [Numpy-discussion] future of f2py and Fortran90+ References: <55A56941.1050209@hawaii.edu> Message-ID: <248388803458617304.609842sturla.molden-gmail.com@news.gmane.org> Eric Firing wrote: > I'm curious: has anyone been looking into what it would take to enable > f2py to handle modern Fortran in general? And into prospects for > getting such an effort funded? No need. Use Cython and Fortran 2003 ISO C bindings. That is the only portable way to interop between Fortran and C (including CPython) anyway. From solipsis at pitrou.net Thu Jul 16 12:18:17 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Jul 2015 18:18:17 +0200 Subject: [Numpy-discussion] Rationale for integer promotion rules Message-ID: <20150716181817.55d826fb@fsol> Hi, We were discussion integer promotion rules amongst the Numba team, and we were wondering about the rationale for Numpy's rules. For example, adding int8 and int8 will give int8 as result (with potential magnitude loss), while adding int8 and uint8 will give int16 as result (promoting to the smallest fitting type). Furthermore, adding int64 and uint64 returns float64. Is there a rationale somewhere documenting and explaining this behaviour (sorry if I've missed it in a obvious location)? Is it the produce of organic evolution? Also, it is set to stay like this, or will it evolve in the future? Thank you Antoine. From njs at pobox.com Thu Jul 16 14:14:10 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 16 Jul 2015 11:14:10 -0700 Subject: [Numpy-discussion] Rationale for integer promotion rules In-Reply-To: <20150716181817.55d826fb@fsol> References: <20150716181817.55d826fb@fsol> Message-ID: On Thu, Jul 16, 2015 at 9:18 AM, Antoine Pitrou wrote: > > Hi, > > We were discussion integer promotion rules amongst the Numba team, and > we were wondering about the rationale for Numpy's rules. For example, > adding int8 and int8 will give int8 as result (with potential > magnitude loss), I believe the motivation here is that if the user took the trouble to set up int8 arrays, they probably actually want int8 arrays -- it would be annoying if int8 + int8 -> int16 (doubling memory use), and presumably also imply that int8 + int8 + int8 -> int32, and then what does add.reduce do? Not to mention int64 + int64. My guess is that while the possibility of overflow is something of a problem, the solution to that is to make it easy to catch overflow and warn/error, rather than trying to define the casting rules so that it can't happen. > while adding int8 and uint8 will give int16 as result > (promoting to the smallest fitting type). I understand this to be a consequence of the previous rule (results should match inputs) combined with the need to find a common input type. > Furthermore, adding int64 > and uint64 returns float64. This is a grievous kluge, on the grounds that no-one is really sure *what* to do in this case. > Is there a rationale somewhere documenting and explaining this behaviour > (sorry if I've missed it in a obvious location)? Is it the produce of > organic evolution? > > Also, it is set to stay like this, or will it evolve in the future? I don't know -- if you make a good case for something better then maybe? -n -- Nathaniel J. Smith -- http://vorpus.org From robert.kern at gmail.com Thu Jul 16 14:19:58 2015 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Jul 2015 19:19:58 +0100 Subject: [Numpy-discussion] Rationale for integer promotion rules In-Reply-To: References: <20150716181817.55d826fb@fsol> Message-ID: On Thu, Jul 16, 2015 at 7:14 PM, Nathaniel Smith wrote: > > On Thu, Jul 16, 2015 at 9:18 AM, Antoine Pitrou wrote: > > while adding int8 and uint8 will give int16 as result > > (promoting to the smallest fitting type). > > I understand this to be a consequence of the previous rule (results > should match inputs) combined with the need to find a common input > type. Specifically, when combining signed and unsigned ints, we need to find a signed int type that can simultaneously hold both of the *inputs*. Whether the *output* would fit or overflow is not what the rules are concerned with. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 16 14:24:34 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 16 Jul 2015 19:24:34 +0100 Subject: [Numpy-discussion] Rationale for integer promotion rules In-Reply-To: References: <20150716181817.55d826fb@fsol> Message-ID: On Thu, Jul 16, 2015 at 7:14 PM, Nathaniel Smith wrote: > On Thu, Jul 16, 2015 at 9:18 AM, Antoine Pitrou wrote: >> >> Hi, >> >> We were discussion integer promotion rules amongst the Numba team, and >> we were wondering about the rationale for Numpy's rules. For example, >> adding int8 and int8 will give int8 as result (with potential >> magnitude loss), > > I believe the motivation here is that if the user took the trouble to > set up int8 arrays, they probably actually want int8 arrays -- it > would be annoying if int8 + int8 -> int16 (doubling memory use), and > presumably also imply that int8 + int8 + int8 -> int32, and then what > does add.reduce do? Not to mention int64 + int64. > > My guess is that while the possibility of overflow is something of a > problem, the solution to that is to make it easy to catch overflow and > warn/error, rather than trying to define the casting rules so that it > can't happen. > >> while adding int8 and uint8 will give int16 as result >> (promoting to the smallest fitting type). > > I understand this to be a consequence of the previous rule (results > should match inputs) combined with the need to find a common input > type. > >> Furthermore, adding int64 >> and uint64 returns float64. > > This is a grievous kluge, on the grounds that no-one is really sure > *what* to do in this case. It doesn't seem unreasonable to me : casting int64 to uint64 or uint64 to int64 could lead to disastrous truncation. float64 can exactly represent integers +/- 2**53 and will have some defined relative error above that. Cheers, Matthew From solipsis at pitrou.net Thu Jul 16 15:22:24 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Jul 2015 21:22:24 +0200 Subject: [Numpy-discussion] Rationale for integer promotion rules References: <20150716181817.55d826fb@fsol> Message-ID: <20150716212224.6010c5ac@fsol> On Thu, 16 Jul 2015 19:19:58 +0100 Robert Kern wrote: > On Thu, Jul 16, 2015 at 7:14 PM, Nathaniel Smith wrote: > > > > On Thu, Jul 16, 2015 at 9:18 AM, Antoine Pitrou > wrote: > > > > while adding int8 and uint8 will give int16 as result > > > (promoting to the smallest fitting type). > > > > I understand this to be a consequence of the previous rule (results > > should match inputs) combined with the need to find a common input > > type. > > Specifically, when combining signed and unsigned ints, we need to find a > signed int type that can simultaneously hold both of the *inputs*. I'm not sure that would be necessary for e.g. addition (as long as you run your code on a 2's complement machine). But thanks for the explanation. Regards Antoine. From solipsis at pitrou.net Thu Jul 16 15:24:28 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Jul 2015 21:24:28 +0200 Subject: [Numpy-discussion] Rationale for integer promotion rules References: <20150716181817.55d826fb@fsol> Message-ID: <20150716212428.6f139796@fsol> On Thu, 16 Jul 2015 11:14:10 -0700 Nathaniel Smith wrote: > > > > Also, it is set to stay like this, or will it evolve in the future? > > I don't know -- if you make a good case for something better then maybe? No, I was just wondering if it would be a good use of our time to try to use a scheme that looks remotely like Numpy's :-) Regards Antoine. From sturla.molden at gmail.com Fri Jul 17 14:16:24 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 17 Jul 2015 18:16:24 +0000 (UTC) Subject: [Numpy-discussion] Rationale for integer promotion rules References: <20150716181817.55d826fb@fsol> Message-ID: <381326271458849419.430154sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: >>> Furthermore, adding int64 >>> and uint64 returns float64. >> >> This is a grievous kluge, on the grounds that no-one is really sure >> *what* to do in this case. > > It doesn't seem unreasonable to me : casting int64 to uint64 or uint64 > to int64 could lead to disastrous truncation. float64 can exactly > represent integers +/- 2**53 and will have some defined relative error > above that. We could promote to Python int on Python 3 and long on Python 2. Sturla From charlesr.harris at gmail.com Tue Jul 21 23:38:30 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Jul 2015 21:38:30 -0600 Subject: [Numpy-discussion] 1,10 and 1.11 releases Message-ID: Hi All, I plan on branching 1.10 this next weekend and the `__numpy_ufunc__` work will be pushed off to 1.11. One of the things we discussed at SciPy 2015 was the need for a faster release cycle. If we are going to manage that we need to start planning now by picking a release manager, listing the work we want done, and setting a preliminary release date. For the release manager, Jaime among others expressed an interest and I would be happy with that. Others who might be interested should add their names here. For work to be done, certainly we want to get the `__numpy_ufunc__` functionality settled. Other possibilities might be merging the ufunc module into multiarray or further cleanups of masked array. Longer term we might want to take a look at scalars and dtypes. Please add other suggestions here. I think December might be a good target date for the beta release, although the holidays have a way of messing with everyone's schedule. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Jul 22 14:45:43 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Jul 2015 14:45:43 -0400 Subject: [Numpy-discussion] difference between dtypes Message-ID: Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? >>> np.bool8 >>> np.bool_ >>> bool Are there any rules and recommendations or is it all folks lore? I'm asking because my intuition picked up by osmosis might be off, and I thought https://github.com/scipy/scipy/pull/5076 is weird (i.e. counter intuitive). Deprecation warnings are always a lot of fun, especially if "This log is too long to be displayed. Please reduce the verbosity of your build or download the raw log." Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 22 22:29:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 Jul 2015 19:29:19 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? Message-ID: Hi all, So one of the things exposed in the numpy namespace are objects called np.int np.float np.bool etc. These are commonly used -- in fact, just yesterday on another project I saw a senior person reviewing a pull request instruct a more junior person that they should use np.float instead of float or np.float64. But AFAICT everyone who is actually using them is doing this based on a very easy-to-fall-for misconception, i.e., that these objects have something to do with numpy. In fact they are just aliases for the regular builtin Python types: 'int', 'float', 'bool', etc. NumPy *does have* special numpy-specific types -- but these are called np.int_, np.float_, np.bool_, etc. Apparently they were set up this way back in numpy 0.something, as a backwards compatibility (!) hack: https://github.com/numpy/numpy/pull/6103#issuecomment-123801937 Now, 10+ years later, they continue to confuse people on a regular, ongoing basis, and new users are still being taught misleading "facts" about them. I suggest that we should deprecate them, with no fixed schedule for actually removing them. (I have no idea if/when people will actually stop using them to the point that we can get away with removing them entirely, but in the mean time we should at least be publicizing that any code which is using them is almost certainly based on a misunderstanding.) The technical challenge here is that historically it has simply been impossible to deprecate a global constant like this without using version-specific hacks or accepting unacceptable slowdowns on every attribute access. But, python 3.5 finally adds the necessary machinery to do this in a future-proof way, so now it can be done safely across all versions of Python that we care about, including future unreleased versions: https://github.com/njsmith/metamodule/ Hence: https://github.com/numpy/numpy/pull/6103 Thoughts? -n P.S.: using metamodule.py also gives us the option of making np.testing lazily imported, which last time this came up was benchmarked to improve numpy's import speed by ~35% [1] -- not too bad given that most production code will never touch np.testing. But this is just a teaser postscript; I'm not proposing that we actually do this at this time :-). [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063147.html -- Nathaniel J. Smith -- http://vorpus.org From c99.smruti at gmail.com Thu Jul 23 22:29:44 2015 From: c99.smruti at gmail.com (SMRUTI RANJAN SAHOO) Date: Fri, 24 Jul 2015 07:59:44 +0530 Subject: [Numpy-discussion] 1,10 and 1.11 releases In-Reply-To: References: Message-ID: i am interested but i dont know how start. can you help me about this ??? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nevion at gmail.com Thu Jul 23 22:55:28 2015 From: nevion at gmail.com (Jason Newton) Date: Thu, 23 Jul 2015 19:55:28 -0700 Subject: [Numpy-discussion] constructing record dtypes from the c-api Message-ID: Hi folks, The moderator for the ML approved my subscription so I can now post this back in the numpy list rather than scipy. Apologies for the duplicate/cross posting. I was trying to figure out how to make a dtype for a c-struct on the c-side and storing that in some boost python libraries I'm making. Imagine the following c-struct, greatly simplified of course from the real ones I need to expose: struct datum{ double position[3]; float velocity[3]; int32_t d0; uint64_t time_of_receipt; }; How would you make the dtype/PyArray_Descr for this? I have as a point of reference compound types in HDF for similar struct descriptions (h5py makes these nearly 1:1 and converts back and forth to dtypes and hdf5 types, it uses Cython to accomplish this) - but I don't want to bring in hdf for this task - I'm not sure how well the offsets would go over in that translation to h5py too. Proper/best solutions would make use of offsetof as we insert entries to the dtype/PyArray_Descr. It's fine if you do this in straight C - I just need to figure out how to accomplish this in a practical way. The language I'm working in is C++11. The end goal is probably going to be to create a helper infrastructure to allow this to be made automatically-ish provided implementation of a [static] visitor pattern in C++. The idea is to make numpy compatible c++ POD types rather than use Boost.Python wrapped proxies for every object which will cut down on some complicated and time consuming code (both of computer and developer) when ndarrays are what's called for. Related - would one work with record dtypes passed to C? How would one lookup fields and offsets within a structure? Thanks for any advisement! -Jason From robert.kern at gmail.com Fri Jul 24 03:46:53 2015 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Jul 2015 08:46:53 +0100 Subject: [Numpy-discussion] difference between dtypes In-Reply-To: References: Message-ID: On Wed, Jul 22, 2015 at 7:45 PM, wrote: > > Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? > > >>> np.bool8 > > >>> np.bool_ > > >>> bool > > > > Are there any rules and recommendations or is it all folks lore? This may help a little: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing Basically, we accept the builtin Python type objects as a dtype argument and do something sensible with them. float -> np.float64 because Python floats are C doubles. int -> np.int32 or np.int64 depending on whatever a C long is (i.e. depending on the 64bitness of your CPU and how your OS chooses to deal with that). We encode those precision choices as aliases to the corresponding specific numpy scalar types (underscored as necessary to avoid shadowing builtins of the same name): np.float_ is np.float64, for example. See here for why the aliases to Python builtin types, np.int, np.float, etc. still exist: https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 If you just need to pass a dtype= argument and want the precision that matches the "native" integer and float for your platform, then I prefer to use the Python builtin types instead of the underscored aliases; they just look cleaner. If you need a true numpy scalar type (e.g. to construct a numpy scalar object), of course, you must use one of the numpy scalar types, and the underscored aliases are convenient for that. Never use the aliases to the Python builtin types. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Jul 24 05:05:41 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 24 Jul 2015 05:05:41 -0400 Subject: [Numpy-discussion] difference between dtypes In-Reply-To: References: Message-ID: On Fri, Jul 24, 2015 at 3:46 AM, Robert Kern wrote: > On Wed, Jul 22, 2015 at 7:45 PM, wrote: > > > > Is there an explanation somewhere of what different basic dtypes mean, > across platforms and python versions? > > > > >>> np.bool8 > > > > >>> np.bool_ > > > > >>> bool > > > > > > > > Are there any rules and recommendations or is it all folks lore? > > This may help a little: > > > http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing > > Basically, we accept the builtin Python type objects as a dtype argument > and do something sensible with them. float -> np.float64 because Python > floats are C doubles. int -> np.int32 or np.int64 depending on whatever a C > long is (i.e. depending on the 64bitness of your CPU and how your OS > chooses to deal with that). We encode those precision choices as aliases to > the corresponding specific numpy scalar types (underscored as necessary to > avoid shadowing builtins of the same name): np.float_ is np.float64, for > example. > > See here for why the aliases to Python builtin types, np.int, np.float, > etc. still exist: > > https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 > > If you just need to pass a dtype= argument and want the precision that > matches the "native" integer and float for your platform, then I prefer to > use the Python builtin types instead of the underscored aliases; they just > look cleaner. If you need a true numpy scalar type (e.g. to construct a > numpy scalar object), of course, you must use one of the numpy scalar > types, and the underscored aliases are convenient for that. Never use the > aliases to the Python builtin types. > (I don't have time to follow up on this for at least two weeks) my thinking was that, if there is no actual difference between bool, np.bool and np.bool_, the np.bool could become an alias and a replacement for np.bool_, so we can get rid of a "ugly" trailing underscore. If np.float is always float64 it could be mapped to that directly. As the previous discussion on python int versus numpy int on python 3.x, int is at least confusing. Also I'm thinking that maybe adjusting the code to the (mis)interpretation, instead of adjusting killing np.float completely might be nicer, (but changing np.int would be riskier?) Josef > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jul 24 06:47:05 2015 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Jul 2015 11:47:05 +0100 Subject: [Numpy-discussion] difference between dtypes In-Reply-To: References: Message-ID: On Fri, Jul 24, 2015 at 10:05 AM, wrote: > > On Fri, Jul 24, 2015 at 3:46 AM, Robert Kern wrote: >> >> On Wed, Jul 22, 2015 at 7:45 PM, wrote: >> > >> > Is there an explanation somewhere of what different basic dtypes mean, across platforms and python versions? >> > >> > >>> np.bool8 >> > >> > >>> np.bool_ >> > >> > >>> bool >> > >> > >> > >> > Are there any rules and recommendations or is it all folks lore? >> >> This may help a little: >> >> http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#arrays-dtypes-constructing >> >> Basically, we accept the builtin Python type objects as a dtype argument and do something sensible with them. float -> np.float64 because Python floats are C doubles. int -> np.int32 or np.int64 depending on whatever a C long is (i.e. depending on the 64bitness of your CPU and how your OS chooses to deal with that). We encode those precision choices as aliases to the corresponding specific numpy scalar types (underscored as necessary to avoid shadowing builtins of the same name): np.float_ is np.float64, for example. >> >> See here for why the aliases to Python builtin types, np.int, np.float, etc. still exist: >> >> https://github.com/numpy/numpy/pull/6103#issuecomment-123652497 >> >> If you just need to pass a dtype= argument and want the precision that matches the "native" integer and float for your platform, then I prefer to use the Python builtin types instead of the underscored aliases; they just look cleaner. If you need a true numpy scalar type (e.g. to construct a numpy scalar object), of course, you must use one of the numpy scalar types, and the underscored aliases are convenient for that. Never use the aliases to the Python builtin types. > > (I don't have time to follow up on this for at least two weeks) > > my thinking was that, if there is no actual difference between bool, np.bool and np.bool_, the np.bool could become an alias and a replacement for np.bool_, so we can get rid of a "ugly" trailing underscore. > If np.float is always float64 it could be mapped to that directly. Well, I'll tell you why that's a bad idea in when you get back in two weeks. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Jul 24 11:51:54 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 24 Jul 2015 17:51:54 +0200 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: References: Message-ID: <55B25F1A.70107@googlemail.com> On 07/23/2015 04:29 AM, Nathaniel Smith wrote: > Hi all, > > So one of the things exposed in the numpy namespace are objects called > np.int > np.float > np.bool > etc. > > These are commonly used -- in fact, just yesterday on another project > I saw a senior person reviewing a pull request instruct a more junior > person that they should use np.float instead of float or np.float64. > But AFAICT everyone who is actually using them is doing this based on > a very easy-to-fall-for misconception, i.e., that these objects have > something to do with numpy. I don't see the issue. They are just aliases so how is np.float worse than just float? Too me this does not seem worth the bother of deprecation. An argument could be made for deprecating creating dtypes from python builtin types as they are ambiguous (C float != python float) and platform dependent. E.g. dtype=int is just an endless source of bugs. But this is also so invasive that the deprecation would never be completed and just be a bother to everyone. So -1 from me. > > P.S.: using metamodule.py also gives us the option of making > np.testing lazily imported, which last time this came up was > benchmarked to improve numpy's import speed by ~35% [1] -- not too bad > given that most production code will never touch np.testing. But this > is just a teaser postscript; I'm not proposing that we actually do > this at this time :-). > > [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063147.html > I doubt these numbers from 2012 are still correct. When this was last profiled last year the import there were two main offenders, add_docs and np.polynomial. Both have been fixed in 1.9. I don't recall np.testing even showing up. From sturla.molden at gmail.com Fri Jul 24 13:03:11 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 24 Jul 2015 17:03:11 +0000 (UTC) Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? References: <55B25F1A.70107@googlemail.com> Message-ID: <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> Julian Taylor wrote: > I don't see the issue. They are just aliases so how is np.float worse > than just float? I have burned my fingers on it. Since np.double is a C double I assumed np.float is a C float. It is not. np.int has the same problem by being a C long. Pure evil. Most users of NumPy probably expect the np.foobar dtype to map to the corresponding foobar C type. This is actually inconsistent and plain dangerous. It would be much better if dtype=float meant Python float, dtype=np.float meant C float, dtype=int meant Python int, and dtype=np.int meant C int. Sturla From chris.barker at noaa.gov Fri Jul 24 15:03:53 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 24 Jul 2015 12:03:53 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> References: <55B25F1A.70107@googlemail.com> <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Jul 24, 2015 at 10:03 AM, Sturla Molden wrote: > > I don't see the issue. They are just aliases so how is np.float worse > > than just float? > > I have burned my fingers on it. > I must have too -- but I don't recall, because I am VERY careful about not using np.float, no.int, etc... but I do have to constantly evangelize and correct code others put in my code base. This really is very, very, ugly. we get away with np.float, because every OS/compiler that gets any regular use has np.float == a c double, which is always 64 bit. but, as Sturla points our, no.int being a C long is a disaster! So +inf on deprecating this, though I have no opinion about the mechanism. Ans sadly, it will be a long time before we can actually remove them, so the evangelizing and code reviews will need to co continue for a long time... -Chris > Since np.double is a C double I assumed np.float is a C float. It is not. > > np.int has the same problem by being a C long. Pure evil. Most users of > NumPy probably expect the np.foobar dtype to map to the corresponding > foobar C type. This is actually inconsistent and plain dangerous. > > It would be much better if dtype=float meant Python float, dtype=np.float > meant C float, dtype=int meant Python int, and dtype=np.int meant C int. > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jul 24 15:39:54 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 24 Jul 2015 21:39:54 +0200 Subject: [Numpy-discussion] ANN: Scipy 0.16.0 release Message-ID: Hi all, On behalf of the Scipy development team I'm pleased to announce the availability of Scipy 0.16.0. This release contains some exciting new features (see release notes below) and more than half a years' worth of maintenance work. 93 people contributed to this release. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Sources, binaries and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.16.0 Enjoy, Ralf -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ========================== SciPy 0.16.0 Release Notes ========================== SciPy 0.16.0 is the culmination of 7 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Highlights of this release include: - - A Cython API for BLAS/LAPACK in `scipy.linalg` - - A new benchmark suite. It's now straightforward to add new benchmarks, and they're routinely included with performance enhancement PRs. - - Support for the second order sections (SOS) format in `scipy.signal`. New features ============ Benchmark suite - --------------- The benchmark suite has switched to using `Airspeed Velocity `__ for benchmarking. You can run the suite locally via ``python runtests.py --bench``. For more details, see ``benchmarks/README.rst``. `scipy.linalg` improvements - --------------------------- A full set of Cython wrappers for BLAS and LAPACK has been added in the modules `scipy.linalg.cython_blas` and `scipy.linalg.cython_lapack`. In Cython, these wrappers can now be cimported from their corresponding modules and used without linking directly against BLAS or LAPACK. The functions `scipy.linalg.qr_delete`, `scipy.linalg.qr_insert` and `scipy.linalg.qr_update` for updating QR decompositions were added. The function `scipy.linalg.solve_circulant` solves a linear system with a circulant coefficient matrix. The function `scipy.linalg.invpascal` computes the inverse of a Pascal matrix. The function `scipy.linalg.solve_toeplitz`, a Levinson-Durbin Toeplitz solver, was added. Added wrapper for potentially useful LAPACK function ``*lasd4``. It computes the square root of the i-th updated eigenvalue of a positive symmetric rank-one modification to a positive diagonal matrix. See its LAPACK documentation and unit tests for it to get more info. Added two extra wrappers for LAPACK least-square solvers. Namely, they are ``*gelsd`` and ``*gelsy``. Wrappers for the LAPACK ``*lange`` functions, which calculate various matrix norms, were added. Wrappers for ``*gtsv`` and ``*ptsv``, which solve ``A*X = B`` for tri-diagonal matrix ``A``, were added. `scipy.signal` improvements - --------------------------- Support for second order sections (SOS) as a format for IIR filters was added. The new functions are: * `scipy.signal.sosfilt` * `scipy.signal.sosfilt_zi`, * `scipy.signal.sos2tf` * `scipy.signal.sos2zpk` * `scipy.signal.tf2sos` * `scipy.signal.zpk2sos`. Additionally, the filter design functions `iirdesign`, `iirfilter`, `butter`, `cheby1`, `cheby2`, `ellip`, and `bessel` can return the filter in the SOS format. The function `scipy.signal.place_poles`, which provides two methods to place poles for linear systems, was added. The option to use Gustafsson's method for choosing the initial conditions of the forward and backward passes was added to `scipy.signal.filtfilt`. New classes ``TransferFunction``, ``StateSpace`` and ``ZerosPolesGain`` were added. These classes are now returned when instantiating `scipy.signal.lti`. Conversion between those classes can be done explicitly now. An exponential (Poisson) window was added as `scipy.signal.exponential`, and a Tukey window was added as `scipy.signal.tukey`. The function for computing digital filter group delay was added as `scipy.signal.group_delay`. The functionality for spectral analysis and spectral density estimation has been significantly improved: `scipy.signal.welch` became ~8x faster and the functions `scipy.signal.spectrogram`, `scipy.signal.coherence` and `scipy.signal.csd` (cross-spectral density) were added. `scipy.signal.lsim` was rewritten - all known issues are fixed, so this function can now be used instead of ``lsim2``; ``lsim`` is orders of magnitude faster than ``lsim2`` in most cases. `scipy.sparse` improvements - --------------------------- The function `scipy.sparse.norm`, which computes sparse matrix norms, was added. The function `scipy.sparse.random`, which allows to draw random variates from an arbitrary distribution, was added. `scipy.spatial` improvements - ---------------------------- `scipy.spatial.cKDTree` has seen a major rewrite, which improved the performance of the ``query`` method significantly, added support for parallel queries, pickling, and options that affect the tree layout. See pull request 4374 for more details. The function `scipy.spatial.procrustes` for Procrustes analysis (statistical shape analysis) was added. `scipy.stats` improvements - -------------------------- The Wishart distribution and its inverse have been added, as `scipy.stats.wishart` and `scipy.stats.invwishart`. The Exponentially Modified Normal distribution has been added as `scipy.stats.exponnorm`. The Generalized Normal distribution has been added as `scipy.stats.gennorm`. All distributions now contain a ``random_state`` property and allow specifying a specific ``numpy.random.RandomState`` random number generator when generating random variates. Many statistical tests and other `scipy.stats` functions that have multiple return values now return ``namedtuples``. See pull request 4709 for details. `scipy.optimize` improvements - ----------------------------- A new derivative-free method DF-SANE has been added to the nonlinear equation system solving function `scipy.optimize.root`. Deprecated features =================== ``scipy.stats.pdf_fromgamma`` is deprecated. This function was undocumented, untested and rarely used. Statsmodels provides equivalent functionality with ``statsmodels.distributions.ExpandedNormal``. ``scipy.stats.fastsort`` is deprecated. This function is unnecessary, ``numpy.argsort`` can be used instead. ``scipy.stats.signaltonoise`` and ``scipy.stats.mstats.signaltonoise`` are deprecated. These functions did not belong in ``scipy.stats`` and are rarely used. See issue #609 for details. ``scipy.stats.histogram2`` is deprecated. This function is unnecessary, ``numpy.histogram2d`` can be used instead. Backwards incompatible changes ============================== The deprecated global optimizer ``scipy.optimize.anneal`` was removed. The following deprecated modules have been removed: ``scipy.lib.blas``, ``scipy.lib.lapack``, ``scipy.linalg.cblas``, ``scipy.linalg.fblas``, ``scipy.linalg.clapack``, ``scipy.linalg.flapack``. They had been deprecated since Scipy 0.12.0, the functionality should be accessed as `scipy.linalg.blas` and `scipy.linalg.lapack`. The deprecated function ``scipy.special.all_mat`` has been removed. The deprecated functions ``fprob``, ``ksprob``, ``zprob``, ``randwcdf`` and ``randwppf`` have been removed from `scipy.stats`. Other changes ============= The version numbering for development builds has been updated to comply with PEP 440. Building with ``python setup.py develop`` is now supported. Authors ======= * @axiru + * @endolith * Elliott Sales de Andrade + * Anne Archibald * Yoshiki V?zquez Baeza + * Sylvain Bellemare * Felix Berkenkamp + * Raoul Bourquin + * Matthew Brett * Per Brodtkorb * Christian Brueffer * Lars Buitinck * Evgeni Burovski * Steven Byrnes * CJ Carey * George Castillo + * Alex Conley + * Liam Damewood + * Rupak Das + * Abraham Escalante + * Matthias Feurer + * Eric Firing + * Clark Fitzgerald * Chad Fulton * Andr? Gaul * Andreea Georgescu + * Christoph Gohlke * Andrey Golovizin + * Ralf Gommers * J.J. Green + * Alex Griffing * Alexander Grigorievskiy + * Hans Moritz Gunther + * Jonas Hahnfeld + * Charles Harris * Ian Henriksen * Andreas Hilboll * ?smund Hjulstad + * Jan Schl?ter + * Janko Slavi? + * Daniel Jensen + * Johannes Ball? + * Terry Jones + * Amato Kasahara + * Eric Larson * Denis Laxalde * Antony Lee * Gregory R. Lee * Perry Lee + * Lo?c Est?ve * Martin Manns + * Eric Martin + * Mat?j Koci?n + * Andreas Mayer + * Nikolay Mayorov + * Robert McGibbon + * Sturla Molden * Nicola Montecchio + * Eric Moore * Jamie Morton + * Nikolas Moya + * Maniteja Nandana + * Andrew Nelson * Joel Nothman * Aldrian Obaja * Regina Ongowarsito + * Paul Ortyl + * Pedro L?pez-Adeva Fern?ndez-Layos + * Stefan Peterson + * Irvin Probst + * Eric Quintero + * John David Reaver + * Juha Remes + * Thomas Robitaille * Clancy Rowley + * Tobias Schmidt + * Skipper Seabold * Aman Singh + * Eric Soroos * Valentine Svensson + * Julian Taylor * Aman Thakral + * Helmut Toplitzer + * Fukumu Tsutsumi + * Anastasiia Tsyplia + * Jacob Vanderplas * Pauli Virtanen * Matteo Visconti + * Warren Weckesser * Florian Wilhelm + * Nathan Woods * Haochen Wu + * Daan Wynen + A total of 93 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.16.0 - ------------------------ - - `#1063 `__: Implement a whishart distribution (Trac #536) - - `#1885 `__: Rbf: floating point warnings - possible bug (Trac #1360) - - `#2020 `__: Rbf default epsilon too large (Trac #1495) - - `#2325 `__: extending distributions, hypergeom, to degenerate cases (Trac... - - `#3502 `__: [ENH] linalg.hessenberg should use ORGHR for calc_q=True - - `#3603 `__: Passing array as window into signal.resample() fails - - `#3675 `__: Intermittent failures for signal.slepian on Windows - - `#3742 `__: Pchipinterpolator inconvenient as ppoly - - `#3786 `__: add procrustes? - - `#3798 `__: scipy.io.savemat fails for empty dicts - - `#3975 `__: Use RandomState in scipy.stats - - `#4022 `__: savemat incorrectly saves logical arrays - - `#4028 `__: scipy.stats.geom.logpmf(1,1) returns nan. The correct value is... - - `#4030 `__: simplify scipy.stats.betaprime.cdf - - `#4031 `__: improve accuracy of scipy.stats.gompertz distribution for small... - - `#4033 `__: improve accuracy of scipy.stats.lomax distribution for small... - - `#4034 `__: improve accuracy of scipy.stats.rayleigh distribution for large... - - `#4035 `__: improve accuracy of scipy.stats.truncexpon distribution for small... - - `#4081 `__: Error when reading matlab file: buffer is too small for requested... - - `#4100 `__: Why does qr(a, lwork=0) not fail? - - `#4134 `__: scipy.stats: rv_frozen has no expect() method - - `#4204 `__: Please add docstring to scipy.optimize.RootResults - - `#4206 `__: Wrap LAPACK tridiagonal solve routine `gtsv` - - `#4208 `__: Empty sparse matrices written to MAT file cannot be read by MATLAB - - `#4217 `__: use a TravisCI configuration with numpy built with NPY_RELAXED_STRIDES_CHECKING=1 - - `#4282 `__: integrate.odeint raises an exception when full_output=1 and the... - - `#4301 `__: scipy and numpy version names do not follow pep 440 - - `#4355 `__: PPoly.antiderivative() produces incorrect output - - `#4391 `__: spsolve becomes extremely slow with large b matrix - - `#4393 `__: Documentation glitsch in sparse.linalg.spilu - - `#4408 `__: Vector-valued constraints in minimize() et al - - `#4412 `__: Documentation of scipy.signal.cwt error - - `#4428 `__: dok.__setitem__ problem with negative indices - - `#4434 `__: Incomplete documentation for sparse.linalg.spsolve - - `#4438 `__: linprog() documentation example wrong - - `#4445 `__: Typo in scipy.special.expit doc - - `#4467 `__: Documentation Error in scipy.optimize options for TNC - - `#4492 `__: solve_toeplitz benchmark is bitrotting already - - `#4506 `__: lobpcg/sparse performance regression Jun 2014? - - `#4520 `__: g77_abi_wrappers needed on Linux for MKL as well - - `#4521 `__: Broken check in uses_mkl for newer versions of the library - - `#4523 `__: rbf with gaussian kernel seems to produce more noise than original... - - `#4526 `__: error in site documentation for poisson.pmf() method - - `#4527 `__: KDTree example doesn't work in Python 3 - - `#4550 `__: `scipy.stats.mode` - UnboundLocalError on empty sequence - - `#4554 `__: filter out convergence warnings in optimization tests - - `#4565 `__: odeint messages - - `#4569 `__: remez: "ValueError: Failure to converge after 25 iterations.... - - `#4582 `__: DOC: optimize: _minimize_scalar_brent does not have a disp option - - `#4585 `__: DOC: Erroneous latex-related characters in tutorial. - - `#4590 `__: sparse.linalg.svds should throw an exception if which not in... - - `#4594 `__: scipy.optimize.linprog IndexError when a callback is providen - - `#4596 `__: scipy.linalg.block_diag misbehavior with empty array inputs (v0.13.3) - - `#4599 `__: scipy.integrate.nquad should call _OptFunc when called with only... - - `#4612 `__: Crash in signal.lfilter on nd input with wrong shaped zi - - `#4613 `__: scipy.io.readsav error on reading sav file - - `#4673 `__: scipy.interpolate.RectBivariateSpline construction locks PyQt... - - `#4681 `__: Broadcasting in signal.lfilter still not quite right. - - `#4705 `__: kmeans k_or_guess parameter error if guess is not square array - - `#4719 `__: Build failure on 14.04.2 - - `#4724 `__: GenGamma _munp function fails due to overflow - - `#4726 `__: FAIL: test_cobyla.test_vector_constraints - - `#4734 `__: Failing tests in stats with numpy master. - - `#4736 `__: qr_update bug or incompatibility with numpy 1.10? - - `#4746 `__: linprog returns solution violating equality constraint - - `#4757 `__: optimize.leastsq docstring mismatch - - `#4774 `__: Update contributor list for v0.16 - - `#4779 `__: circmean and others do not appear in the documentation - - `#4788 `__: problems with scipy sparse linalg isolve iterative.py when complex - - `#4791 `__: BUG: scipy.spatial: incremental Voronoi doesn't increase size... Pull requests for 0.16.0 - ------------------------ - - `#3116 `__: sparse: enhancements for DIA format - - `#3157 `__: ENH: linalg: add the function 'solve_circulant' for solving a... - - `#3442 `__: ENH: signal: Add Gustafsson's method as an option for the filtfilt... - - `#3679 `__: WIP: fix sporadic slepian failures - - `#3680 `__: Some cleanups in stats - - `#3717 `__: ENH: Add second-order sections filtering - - `#3741 `__: Dltisys changes - - `#3956 `__: add note to scipy.signal.resample about prime sample numbers - - `#3980 `__: Add check_finite flag to UnivariateSpline - - `#3996 `__: MAINT: stricter linalg argument checking - - `#4001 `__: BUG: numerical precision in dirichlet - - `#4012 `__: ENH: linalg: Add a function to compute the inverse of a Pascal... - - `#4021 `__: ENH: Cython api for lapack and blas - - `#4089 `__: Fixes for various PEP8 issues. - - `#4116 `__: MAINT: fitpack: trim down compiler warnings (unused labels, variables) - - `#4129 `__: ENH: stats: add a random_state property to distributions - - `#4135 `__: ENH: Add Wishart and inverse Wishart distributions - - `#4195 `__: improve the interpolate docs - - `#4200 `__: ENH: Add t-test from descriptive stats function. - - `#4202 `__: Dendrogram threshold color - - `#4205 `__: BLD: fix a number of Bento build warnings. - - `#4211 `__: add an ufunc for the inverse Box-Cox transfrom - - `#4212 `__: MRG:fix for gh-4208 - - `#4213 `__: ENH: specific warning if matlab file is empty - - `#4215 `__: Issue #4209: splprep documentation updated to reflect dimensional... - - `#4219 `__: DOC: silence several Sphinx warnings when building the docs - - `#4223 `__: MAINT: remove two redundant lines of code - - `#4226 `__: try forcing the numpy rebuild with relaxed strides - - `#4228 `__: BLD: some updates to Bento config files and docs. Closes gh-3978. - - `#4232 `__: wrong references in the docs - - `#4242 `__: DOC: change example sample spacing - - `#4245 `__: Arff fixes - - `#4246 `__: MAINT: C fixes - - `#4247 `__: MAINT: remove some unused code - - `#4249 `__: Add routines for updating QR decompositions - - `#4250 `__: MAINT: Some pyflakes-driven cleanup in linalg and sparse - - `#4252 `__: MAINT trim away >10 kLOC of generated C code - - `#4253 `__: TST: stop shadowing ellip* tests vs boost data - - `#4254 `__: MAINT: special: use NPY_PI, not M_PI - - `#4255 `__: DOC: INSTALL: use Py3-compatible print syntax, and don't mention... - - `#4256 `__: ENH: spatial: reimplement cdist_cosine using np.dot - - `#4258 `__: BUG: io.arff #4429 #2088 - - `#4261 `__: MAINT: signal: PEP8 and related style clean up. - - `#4262 `__: BUG: newton_krylov() was ignoring norm_tol argument, closes #4259 - - `#4263 `__: MAINT: clean up test noise and optimize tests for docstrings... - - `#4266 `__: MAINT: io: Give an informative error when attempting to read... - - `#4268 `__: MAINT: fftpack benchmark integer division vs true division - - `#4269 `__: MAINT: avoid shadowing the eigvals function - - `#4272 `__: BUG: sparse: Fix bench_sparse.py - - `#4276 `__: DOC: remove confusing parts of the documentation related to writing... - - `#4281 `__: Sparse matrix multiplication: only convert array if needed (with... - - `#4284 `__: BUG: integrate: odeint crashed when the integration time was... - - `#4286 `__: MRG: fix matlab output type of logical array - - `#4287 `__: DEP: deprecate stats.pdf_fromgamma. Closes gh-699. - - `#4291 `__: DOC: linalg: fix layout in cholesky_banded docstring - - `#4292 `__: BUG: allow empty dict as proxy for empty struct - - `#4293 `__: MAINT: != -> not_equal in hamming distance implementation - - `#4295 `__: Pole placement - - `#4296 `__: MAINT: some cleanups in tests of several modules - - `#4302 `__: ENH: Solve toeplitz linear systems - - `#4306 `__: Add benchmark for conjugate gradient solver. - - `#4307 `__: BLD: PEP 440 - - `#4310 `__: BUG: make stats.geom.logpmf(1,1) return 0.0 instead of nan - - `#4311 `__: TST: restore a test that uses slogdet now that we have dropped... - - `#4313 `__: Some minor fixes for stats.wishart addition. - - `#4315 `__: MAINT: drop numpy 1.5 compatibility code in sparse matrix tests - - `#4318 `__: ENH: Add random_state to multivariate distributions - - `#4319 `__: MAINT: fix hamming distance regression for exotic arrays, with... - - `#4320 `__: TST: a few changes like self.assertTrue(x == y, message) -> assert_equal(x,... - - `#4321 `__: TST: more changes like self.assertTrue(x == y, message) -> assert_equal(x,... - - `#4322 `__: TST: in test_signaltools, changes like self.assertTrue(x == y,... - - `#4323 `__: MAINT: clean up benchmarks so they can all be run as single files. - - `#4324 `__: Add more detailed committer guidelines, update MAINTAINERS.txt - - `#4326 `__: TST: use numpy.testing in test_hierarchy.py - - `#4329 `__: MAINT: stats: rename check_random_state test function - - `#4330 `__: Update distance tests - - `#4333 `__: MAINT: import comb, factorial from scipy.special, not scipy.misc - - `#4338 `__: TST: more conversions from nose to numpy.testing - - `#4339 `__: MAINT: remove the deprecated all_mat function from special_matrices.py - - `#4340 `__: add several features to frozen distributions - - `#4344 `__: BUG: Fix/test invalid lwork param in qr - - `#4345 `__: Fix test noise visible with Python 3.x - - `#4347 `__: Remove deprecated blas/lapack imports, rename lib to _lib - - `#4349 `__: DOC: add a nontrivial example to stats.binned_statistic. - - `#4350 `__: MAINT: remove optimize.anneal for 0.16.0 (was deprecated in 0.14.0). - - `#4351 `__: MAINT: fix usage of deprecated Numpy C API in optimize... - - `#4352 `__: MAINT: fix a number of special test failures - - `#4353 `__: implement cdf for betaprime distribution - - `#4357 `__: BUG: piecewise polynomial antiderivative - - `#4358 `__: BUG: integrate: fix handling of banded Jacobians in odeint, plus... - - `#4359 `__: MAINT: remove a code path taken for Python version < 2.5 - - `#4360 `__: MAINT: stats.mstats: Remove some unused variables (thanks, pyflakes). - - `#4362 `__: Removed erroneous reference to smoothing parameter #4072 - - `#4363 `__: MAINT: interpolate: clean up in fitpack.py - - `#4364 `__: MAINT: lib: don't export "partial" from decorator - - `#4365 `__: svdvals now returns a length-0 sequence of singular values given... - - `#4367 `__: DOC: slightly improve TeX rendering of wishart/invwishart docstring - - `#4373 `__: ENH: wrap gtsv and ptsv for solve_banded and solveh_banded. - - `#4374 `__: ENH: Enhancements to spatial.cKDTree - - `#4376 `__: BF: fix reading off-spec matlab logical sparse - - `#4377 `__: MAINT: integrate: Clean up some Fortran test code. - - `#4378 `__: MAINT: fix usage of deprecated Numpy C API in signal - - `#4380 `__: MAINT: scipy.optimize, removing further anneal references - - `#4381 `__: ENH: Make DCT and DST accept int and complex types like fft - - `#4392 `__: ENH: optimize: add DF-SANE nonlinear derivative-free solver - - `#4394 `__: Make reordering algorithms 64-bit clean - - `#4396 `__: BUG: bundle cblas.h in Accelerate ABI wrappers to enable compilation... - - `#4398 `__: FIX pdist bug where wminkowski's w.dtype != double - - `#4402 `__: BUG: fix stat.hypergeom argcheck - - `#4404 `__: MAINT: Fill in the full symmetric squareform in the C loop - - `#4405 `__: BUG: avoid X += X.T (refs #4401) - - `#4407 `__: improved accuracy of gompertz distribution for small x - - `#4414 `__: DOC:fix error in scipy.signal.cwt documentation. - - `#4415 `__: ENH: Improve accuracy of lomax for small x. - - `#4416 `__: DOC: correct a parameter name in docstring of SuperLU.solve.... - - `#4419 `__: Restore scipy.linalg.calc_lwork also in master - - `#4420 `__: fix a performance issue with a sparse solver - - `#4423 `__: ENH: improve rayleigh accuracy for large x. - - `#4424 `__: BUG: optimize.minimize: fix overflow issue with integer x0 input. - - `#4425 `__: ENH: Improve accuracy of truncexpon for small x - - `#4426 `__: ENH: improve rayleigh accuracy for large x. - - `#4427 `__: MAINT: optimize: cleanup of TNC code - - `#4429 `__: BLD: fix build failure with numpy 1.7.x and 1.8.x. - - `#4430 `__: BUG: fix a sparse.dok_matrix set/get copy-paste bug - - `#4433 `__: Update _minimize.py - - `#4435 `__: ENH: release GIL around batch distance computations - - `#4436 `__: Fixed incomplete documentation for spsolve - - `#4439 `__: MAINT: integrate: Some clean up in the tests. - - `#4440 `__: Fast permutation t-test - - `#4442 `__: DOC: optimize: fix wrong result in docstring - - `#4447 `__: DOC: signal: Some additional documentation to go along with the... - - `#4448 `__: DOC: tweak the docstring of lapack.linalg module - - `#4449 `__: fix a typo in the expit docstring - - `#4451 `__: ENH: vectorize distance loops with gcc - - `#4456 `__: MAINT: don't fail large data tests on MemoryError - - `#4461 `__: CI: use travis_retry to deal with network timeouts - - `#4462 `__: DOC: rationalize minimize() et al. documentation - - `#4470 `__: MAINT: sparse: inherit dok_matrix.toarray from spmatrix - - `#4473 `__: BUG: signal: Fix validation of the zi shape in sosfilt. - - `#4475 `__: BLD: setup.py: update min numpy version and support "setup.py... - - `#4481 `__: ENH: add a new linalg special matrix: the Helmert matrix - - `#4485 `__: MRG: some changes to allow reading bad mat files - - `#4490 `__: [ENH] linalg.hessenberg: use orghr - rebase - - `#4491 `__: ENH: linalg: Adding wrapper for potentially useful LAPACK function... - - `#4493 `__: BENCH: the solve_toeplitz benchmark used outdated syntax and... - - `#4494 `__: MAINT: stats: remove duplicated code - - `#4496 `__: References added for watershed_ift algorithm - - `#4499 `__: DOC: reshuffle stats distributions documentation - - `#4501 `__: Replace benchmark suite with airspeed velocity - - `#4502 `__: SLSQP should strictly satisfy bound constraints - - `#4503 `__: DOC: forward port 0.15.x release notes and update author name... - - `#4504 `__: ENH: option to avoid computing possibly unused svd matrix - - `#4505 `__: Rebase of PR 3303 (sparse matrix norms) - - `#4507 `__: MAINT: fix lobpcg performance regression - - `#4509 `__: DOC: sparse: replace dead link - - `#4511 `__: Fixed differential evolution bug - - `#4512 `__: Change to fully PEP440 compliant dev version numbers (always... - - `#4525 `__: made tiny style corrections (pep8) - - `#4533 `__: Add exponentially modified gaussian distribution (scipy.stats.expongauss) - - `#4534 `__: MAINT: benchmarks: make benchmark suite importable on all scipy... - - `#4535 `__: BUG: Changed zip() to list(zip()) so that it could work in Python... - - `#4536 `__: Follow up to pr 4348 (exponential window) - - `#4540 `__: ENH: spatial: Add procrustes analysis - - `#4541 `__: Bench fixes - - `#4542 `__: TST: NumpyVersion dev -> dev0 - - `#4543 `__: BUG: Overflow in savgol_coeffs - - `#4544 `__: pep8 fixes for stats - - `#4546 `__: MAINT: use reduction axis arguments in one-norm estimation - - `#4549 `__: ENH : Added group_delay to scipy.signal - - `#4553 `__: ENH: Significantly faster moment function - - `#4556 `__: DOC: document the changes of the sparse.linalg.svds (optional... - - `#4559 `__: DOC: stats: describe loc and scale parameters in the docstring... - - `#4563 `__: ENH: rewrite of stats.ppcc_plot - - `#4564 `__: Be more (or less) forgiving when user passes +-inf instead of... - - `#4566 `__: DEP: remove a bunch of deprecated function from scipy.stats,... - - `#4570 `__: MNT: Suppress LineSearchWarning's in scipy.optimize tests - - `#4572 `__: ENH: Extract inverse hessian information from L-BFGS-B - - `#4576 `__: ENH: Split signal.lti into subclasses, part of #2912 - - `#4578 `__: MNT: Reconcile docstrings and function signatures - - `#4581 `__: Fix build with Intel MKL on Linux - - `#4583 `__: DOC: optimize: remove references to unused disp kwarg - - `#4584 `__: ENH: scipy.signal - Tukey window - - `#4587 `__: Hermite asymptotic - - `#4593 `__: DOC - add example to RegularGridInterpolator - - `#4595 `__: DOC: Fix erroneous latex characters in tutorial/optimize. - - `#4600 `__: Add return codes to optimize.tnc docs - - `#4603 `__: ENH: Wrap LAPACK ``*lange`` functions for matrix norms - - `#4604 `__: scipy.stats: generalized normal distribution - - `#4609 `__: MAINT: interpolate: fix a few inconsistencies between docstrings... - - `#4610 `__: MAINT: make runtest.py --bench-compare use asv continuous and... - - `#4611 `__: DOC: stats: explain rice scaling; add a note to the tutorial... - - `#4614 `__: BUG: lfilter, the size of zi was not checked correctly for nd... - - `#4617 `__: MAINT: integrate: Clean the C code behind odeint. - - `#4618 `__: FIX: Raise error when window length != data length - - `#4619 `__: Issue #4550: `scipy.stats.mode` - UnboundLocalError on empty... - - `#4620 `__: Fixed a problem (#4590) with svds accepting wrong eigenvalue... - - `#4621 `__: Speed up special.ai_zeros/bi_zeros by 10x - - `#4623 `__: MAINT: some tweaks to spatial.procrustes (private file, html... - - `#4628 `__: Speed up signal.lfilter and add a convolution path for FIR filters - - `#4629 `__: Bug: integrate.nquad; resolve issue #4599 - - `#4631 `__: MAINT: integrate: Remove unused variables in a Fortran test function. - - `#4633 `__: MAINT: Fix convergence message for remez - - `#4635 `__: PEP8: indentation (so that pep8 bot does not complain) - - `#4637 `__: MAINT: generalize a sign function to do the right thing for complex... - - `#4639 `__: Amended typo in apple_sgemv_fix.c - - `#4642 `__: MAINT: use lapack for scipy.linalg.norm - - `#4643 `__: RBF default epsilon too large 2020 - - `#4646 `__: Added atleast_1d around poly in invres and invresz - - `#4647 `__: fix doc pdf build - - `#4648 `__: BUG: Fixes #4408: Vector-valued constraints in minimize() et... - - `#4649 `__: Vonmisesfix - - `#4650 `__: Signal example clean up in Tukey and place_poles - - `#4652 `__: DOC: Fix the error in convolve for same mode - - `#4653 `__: improve erf performance - - `#4655 `__: DEP: deprecate scipy.stats.histogram2 in favour of np.histogram2d - - `#4656 `__: DEP: deprecate scipy.stats.signaltonoise - - `#4660 `__: Avoid extra copy for sparse compressed [:, seq] and [seq, :]... - - `#4661 `__: Clean, rebase of #4478, adding ?gelsy and ?gelsd wrappers - - `#4662 `__: MAINT: Correct odeint messages - - `#4664 `__: Update _monotone.py - - `#4672 `__: fix behavior of scipy.linalg.block_diag for empty input - - `#4675 `__: Fix lsim - - `#4676 `__: Added missing colon to :math: directive in docstring. - - `#4679 `__: ENH: sparse randn - - `#4682 `__: ENH: scipy.signal - Addition of CSD, coherence; Enhancement of... - - `#4684 `__: BUG: various errors in weight calculations in orthogonal.py - - `#4685 `__: BUG: Fixes #4594: optimize.linprog IndexError when a callback... - - `#4686 `__: MAINT: cluster: Clean up duplicated exception raising code. - - `#4688 `__: Improve is_distance_dm exception message - - `#4692 `__: MAINT: stats: Simplify the calculation in tukeylambda._ppf - - `#4693 `__: ENH: added functionality to handle scalars in `stats._chk_asarray` - - `#4694 `__: Vectorization of Anderson-Darling computations. - - `#4696 `__: Fix singleton expansion in lfilter. - - `#4698 `__: MAINT: quiet warnings from cephes. - - `#4701 `__: add Bpoly.antiderivatives / integrals - - `#4703 `__: Add citation of published paper - - `#4706 `__: MAINT: special: avoid out-of-bounds access in specfun - - `#4707 `__: MAINT: fix issues with np.matrix as input to functions related... - - `#4709 `__: ENH: `scipy.stats` now returns namedtuples. - - `#4710 `__: scipy.io.idl: make reader more robust to missing variables in... - - `#4711 `__: Fix crash for unknown chunks at the end of file - - `#4712 `__: Reduce onenormest memory usage - - `#4713 `__: MAINT: interpolate: no need to pass dtype around if it can be... - - `#4714 `__: BENCH: Add benchmarks for stats module - - `#4715 `__: MAINT: polish signal.place_poles and signal/test_ltisys.py - - `#4716 `__: DEP: deprecate mstats.signaltonoise ... - - `#4717 `__: MAINT: basinhopping: fix error in tests, silence /0 warning,... - - `#4718 `__: ENH: stats: can specify f-shapes to fix in fitting by name - - `#4721 `__: Document that imresize converts the input to a PIL image - - `#4722 `__: MAINT: PyArray_BASE is not an lvalue unless the deprecated API... - - `#4725 `__: Fix gengamma _nump failure - - `#4728 `__: DOC: add poch to the list of scipy special function descriptions - - `#4735 `__: MAINT: stats: avoid (a spurious) division-by-zero in skew - - `#4738 `__: TST: silence runtime warnings for some corner cases in `stats`... - - `#4739 `__: BLD: try to build numpy instead of using the one on TravisCI - - `#4740 `__: DOC: Update some docstrings with 'versionadded'. - - `#4742 `__: BLD: make sure that relaxed strides checking is in effect on... - - `#4750 `__: DOC: special: TeX typesetting of rel_entr, kl_div and pseudo_huber - - `#4751 `__: BENCH: add sparse null slice benchmark - - `#4753 `__: BUG: Fixed compilation with recent Cython versions. - - `#4756 `__: BUG: Fixes #4733: optimize.brute finish option is not compatible... - - `#4758 `__: DOC: optimize.leastsq default maxfev clarification - - `#4759 `__: improved stats mle fit - - `#4760 `__: MAINT: count bfgs updates more carefully - - `#4762 `__: BUGS: Fixes #4746 and #4594: linprog returns solution violating... - - `#4763 `__: fix small linprog bugs - - `#4766 `__: BENCH: add signal.lsim benchmark - - `#4768 `__: fix python syntax errors in docstring examples - - `#4769 `__: Fixes #4726: test_cobyla.test_vector_constraints - - `#4770 `__: Mark FITPACK functions as thread safe. - - `#4771 `__: edited scipy/stats/stats.py to fix doctest for fisher_exact - - `#4773 `__: DOC: update 0.16.0 release notes. - - `#4775 `__: DOC: linalg: add funm_psd as a docstring example - - `#4778 `__: Use a dictionary for function name synonyms - - `#4780 `__: Include apparently-forgotten functions in docs - - `#4783 `__: Added many missing special functions to docs - - `#4784 `__: add an axis attribute to PPoly and friends - - `#4785 `__: Brief note about origin of Lena image - - `#4786 `__: DOC: reformat the Methods section of the KDE docstring - - `#4787 `__: Add rice cdf and ppf. - - `#4792 `__: CI: add a kludge for detecting test failures which try to disguise... - - `#4795 `__: Make refguide_check smarter about false positives - - `#4797 `__: BUG/TST: numpoints not updated for incremental Voronoi - - `#4799 `__: BUG: spatial: Fix a couple edge cases for the Mahalanobis metric... - - `#4801 `__: BUG: Fix TypeError in scipy.optimize._trust-region.py when disp=True. - - `#4803 `__: Issues with relaxed strides in QR updating routines - - `#4806 `__: MAINT: use an informed initial guess for cauchy fit - - `#4810 `__: PEP8ify codata.py - - `#4812 `__: BUG: Relaxed strides cleanup in decomp_update.pyx.in - - `#4820 `__: BLD: update Bento build for sgemv fix and install cython blas/lapack... - - `#4823 `__: ENH: scipy.signal - Addition of spectrogram function - - `#4827 `__: DOC: add csd and coherence to __init__.py - - `#4833 `__: BLD: fix issue in linalg ``*lange`` wrappers for g77 builds. - - `#4841 `__: TST: fix test failures in scipy.special with mingw32 due to test... - - `#4842 `__: DOC: update site.cfg.example. Mostly taken over from Numpy - - `#4845 `__: BUG: signal: Make spectrogram's return values order match the... - - `#4849 `__: DOC:Fix error in ode docstring example - - `#4856 `__: BUG: fix typo causing memleak Checksums ========= MD5 ~~~ 1c6faa58d12c7b642e64a44b57c311c3 scipy-0.16.0-win32-superpack-python2.7.exe 18b9b53af7216ab897495cc77068285b scipy-0.16.0-win32-superpack-python3.3.exe 9cef8bc882e21854791ec80140258bc9 scipy-0.16.0-win32-superpack-python3.4.exe eb95dda0f36cc3096673993a350cde77 scipy-0.16.0.tar.gz fe1425745ab68d9d4ccaf59e854f1c28 scipy-0.16.0.tar.xz 1764bd452a72698b968ad13e51e28053 scipy-0.16.0.zip SHA256 ~~~~~~ b752366fafa3fddb96352d7b3259f7b3e58fae9f45d5a98eb91bd80005d75dfc scipy-0.16.0-win32-superpack-python2.7.exe 369777d27da760d498a9312696235ab9e90359d9f4e02347669cbf56a42312a8 scipy-0.16.0-win32-superpack-python3.3.exe bcd480ce8e8289942e57e7868d07e9a35982bc30a79150006ad085ce4c06803e scipy-0.16.0-win32-superpack-python3.4.exe 92592f40097098f3fdbe7f5855d535b29bb16719c2bb59c728bce5e7a28790e0 scipy-0.16.0.tar.gz e27f5cfa985cb1253e15aaeddc3dcd512d6853b05e84454f7f43b53b35514071 scipy-0.16.0.tar.xz c9758971df994d238a4d0ff1d47ba5b02f1cb402d6e1925c921a452bc430a3d5 scipy-0.16.0.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVspIaAAoJEO2+o3i/Gl69n3cIAJdoEGIq/8MTsywYL6k5zsqA aBK1Q9aB4qJcCLwM6ULKErxhY9lROzgljSvl22dCaD7YYYgD4Q03+BaXjIrHenbc +sX5CzBPoz+BFjh7tTnfU5a6pVhqjQbW17A0TF0j6jah29pFnM2Xdf3zgHc+3f/B U6JC698wDKROGlvKqWcwKcs2+EPBuu92gNa/rRCmMdnt9dIqVM8+otRNMgPVCZ+R SgfneSGjZ4vXuBK3zWgcP0+r8Ek0DkUuFhEAK3W8NhEFCqd1kHkdvN+RIl6pGfHZ OAHbzds6+VHgvQ3a4g2efJY3CD0LvtOgeS3R3NdmT3gCxkJtZpHAsczFhwKIWHM= =QZFz -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From nevion at gmail.com Sat Jul 25 00:26:01 2015 From: nevion at gmail.com (Jason Newton) Date: Fri, 24 Jul 2015 21:26:01 -0700 Subject: [Numpy-discussion] constructing record dtypes from the c-api In-Reply-To: References: Message-ID: After drilling through the sources a second time, I found it was numpy/core/src/multiarray/descriptor.c was the file to consult with the primary routine being PyArray_DescrConverter and _convert_from_* functions being the most interesting to read and glean the capabilities of this with. So in particular it looks like the _convert_from_dict based path is the way to go to allow custom field offsets to safely and transparently map C POD structs with the offset information generated at compile time to hopefully keep dtype's in perfect sync with C sources vs declaring on the python source side. I plan on building a helper class to generate the dictionaries for this subroutine since something akin to the list dtype specification is more user-friendly (even towards me). -Jason On Thu, Jul 23, 2015 at 7:55 PM, Jason Newton wrote: > Hi folks, > > The moderator for the ML approved my subscription so I can now post > this back in the numpy list rather than scipy. Apologies for the > duplicate/cross posting. > > > I was trying to figure out how to make a dtype for a c-struct on the > c-side and storing that in some boost python libraries I'm making. > > Imagine the following c-struct, greatly simplified of course from the > real ones I need to expose: > > struct datum{ > double position[3]; > float velocity[3]; > int32_t d0; > uint64_t time_of_receipt; > }; > > > How would you make the dtype/PyArray_Descr for this? > > I have as a point of reference compound types in HDF for similar > struct descriptions (h5py makes these nearly 1:1 and converts back and > forth to dtypes and hdf5 types, it uses Cython to accomplish this) - > but I don't want to bring in hdf for this task - I'm not sure how well > the offsets would go over in that translation to h5py too. > > Proper/best solutions would make use of offsetof as we insert entries > to the dtype/PyArray_Descr. It's fine if you do this in straight C - > I just need to figure out how to accomplish this in a practical way. > > The language I'm working in is C++11. The end goal is probably going > to be to create a helper infrastructure to allow this to be made > automatically-ish provided implementation of a [static] visitor > pattern in C++. The idea is to make numpy compatible c++ POD types > rather than use Boost.Python wrapped proxies for every object which > will cut down on some complicated and time consuming code (both of > computer and developer) when ndarrays are what's called for. > > Related - would one work with record dtypes passed to C? How would > one lookup fields and offsets within a structure? > > Thanks for any advisement! > > -Jason From matthew.brett at gmail.com Mon Jul 27 08:05:01 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 27 Jul 2015 13:05:01 +0100 Subject: [Numpy-discussion] np.nonzero - order guarantees? Message-ID: Hi, `np.nonzero` for a 2D array `A` returns: row_inds, col_inds = np.nonzero(A) I notice that `row_inds` appears to be sorted by value, and `col_inds` appears to be sorted by value, within each row. Is this a guarantee of the `np.nonzero` function? If not, does this function guarantee any property of the returned indices, other than the correspondence of the row, column entries? Cheers, Matthew From allanhaldane at gmail.com Sat Jul 25 13:31:19 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Sat, 25 Jul 2015 13:31:19 -0400 Subject: [Numpy-discussion] constructing record dtypes from the c-api In-Reply-To: References: Message-ID: <55B3C7E7.1070402@gmail.com> Hi Jason, As I understand numpy has been set up to mirror C-structs as long as you use the 'align' flag. For example, your struct can be represented as >>> np.dtype('f8,f4,i4,u8', align=True) (assuming 32 bit floats). The offsets of the fields should be exactly the offsets of the elements of the struct. In C you can create the dtype using PyArray_DescrAlignConverter. If you want to get the field offsets given a numpy dtype in C, you need to iterate through the 'names' and 'fields' attributes of the dtype, eg PyObject *key, *tup, *offobj; n = PyTuple_Size(descr->names); for (i = 0; i < n; i++) { key = PyTuple_GetItem(descr->names, i); tup = PyDict_GetItem(descr->fields, key); offobj = PyTuple_GetItem(tup, 1); offset = PyInt_AsSsize_t(offobj); // do something with offset } (error checks & DECREFS might be needed) Allan On 07/25/2015 12:26 AM, Jason Newton wrote: > After drilling through the sources a second time, I found it was > numpy/core/src/multiarray/descriptor.c was the file to consult with > the primary routine being PyArray_DescrConverter and _convert_from_* > functions being the most interesting to read and glean the > capabilities of this with. > > So in particular it looks like the _convert_from_dict based path is > the way to go to allow custom field offsets to safely and > transparently map C POD structs with the offset information generated > at compile time to hopefully keep dtype's in perfect sync with C > sources vs declaring on the python source side. I plan on building a > helper class to generate the dictionaries for this subroutine since > something akin to the list dtype specification is more user-friendly > (even towards me). > > -Jason > > On Thu, Jul 23, 2015 at 7:55 PM, Jason Newton wrote: >> Hi folks, >> >> The moderator for the ML approved my subscription so I can now post >> this back in the numpy list rather than scipy. Apologies for the >> duplicate/cross posting. >> >> >> I was trying to figure out how to make a dtype for a c-struct on the >> c-side and storing that in some boost python libraries I'm making. >> >> Imagine the following c-struct, greatly simplified of course from the >> real ones I need to expose: >> >> struct datum{ >> double position[3]; >> float velocity[3]; >> int32_t d0; >> uint64_t time_of_receipt; >> }; >> >> >> How would you make the dtype/PyArray_Descr for this? >> >> I have as a point of reference compound types in HDF for similar >> struct descriptions (h5py makes these nearly 1:1 and converts back and >> forth to dtypes and hdf5 types, it uses Cython to accomplish this) - >> but I don't want to bring in hdf for this task - I'm not sure how well >> the offsets would go over in that translation to h5py too. >> >> Proper/best solutions would make use of offsetof as we insert entries >> to the dtype/PyArray_Descr. It's fine if you do this in straight C - >> I just need to figure out how to accomplish this in a practical way. >> >> The language I'm working in is C++11. The end goal is probably going >> to be to create a helper infrastructure to allow this to be made >> automatically-ish provided implementation of a [static] visitor >> pattern in C++. The idea is to make numpy compatible c++ POD types >> rather than use Boost.Python wrapped proxies for every object which >> will cut down on some complicated and time consuming code (both of >> computer and developer) when ndarrays are what's called for. >> >> Related - would one work with record dtypes passed to C? How would >> one lookup fields and offsets within a structure? >> >> Thanks for any advisement! >> >> -Jason > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sturla.molden at gmail.com Sun Jul 26 14:19:47 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 26 Jul 2015 18:19:47 +0000 (UTC) Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? References: <55B25F1A.70107@googlemail.com> <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> Message-ID: <963006654459627038.928066sturla.molden-gmail.com@news.gmane.org> Chris Barker wrote: > we get away with np.float, because every OS/compiler that gets any regular > use has np.float == a c double, which is always 64 bit. Not if we are passing an array of np.float to a ac routine that expects float*, e.g. in OpenGL, BLAS or LAPACK. That will for sure give crazy results, just hang, or segfault. I got away with pisting a PR with a "bugfix" which supposedly should fix a case of precision loss in a SciPy routine, because I thought np.float was np.float32 and not np.float64 (which it is). But it did make me feel rather stupid. Sturla From chris.barker at noaa.gov Mon Jul 27 15:26:15 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 27 Jul 2015 12:26:15 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: <963006654459627038.928066sturla.molden-gmail.com@news.gmane.org> References: <55B25F1A.70107@googlemail.com> <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> <963006654459627038.928066sturla.molden-gmail.com@news.gmane.org> Message-ID: On Sun, Jul 26, 2015 at 11:19 AM, Sturla Molden wrote: > > we get away with np.float, because every OS/compiler that gets any > regular > > use has np.float == a c double, which is always 64 bit. > > Not if we are passing an array of np.float to a ac routine that expects > float*, e.g. in OpenGL, BLAS or LAPACK. That will for sure give crazy > results, just hang, or segfault. > well, yes, it is confusing, but at least consistent. So if you use it once correctly in your Python-C transition code, it should work the same way everywhere. As opposed to a np.int which is a python int, which is (if I have this right): 32 bits on all (most) 32 bit platforms 64 bits on 64 bit Linux and OS-X 32 bits on 64 bit Windows (also if compiled by cygwin??) And who knows on a Cray or ARM, or??? Ouch!!! Anyway -- we agree on this -- having the python types in the numpy namespace is confusing and dangerous -- even if it will take forever to deprecate them! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From anton.akhmerov at gmail.com Mon Jul 27 16:10:39 2015 From: anton.akhmerov at gmail.com (Anton Akhmerov) Date: Mon, 27 Jul 2015 22:10:39 +0200 Subject: [Numpy-discussion] Shared memory check on in-place modification. Message-ID: Hi everyone, I have encountered an initially rather confusing problem in a piece of code that attempted to symmetrize a matrix: `h += h.T` The problem of course appears due to `h.T` being a view of `h`, and some elements being overwritten during the __iadd__ call. What made it nasty is that it appeared in code that was already covered by tests, but due to the way __iadd__ works internally, the result only became wrong for matrix sizes > 90. Even worse, it was actually `h += h.T.conj()`, but h was real so conj() was also returning a view. I imagine this was already discussed before (also Charles Harris in https://github.com/numpy/numpy/issues/6119 indicated the issue is known), but I would like to still reopen the question since I was unable to find previous iterations of this discussion. First of all does anyone know of a simple way to test for this issue? Specifically I'm concerned about all in-place array modifications that rely on the data being modified. I considered monkey-patching just to figure out where the problem appears, but numpy.ndarray isn't written in Python so I'm not sure it can be monkey-patched. Secondly, is there no way to automatically check for this behavior in numpy itself? I'm not sure how reliable it is, but would adding np.may_share_memory be a reasonably reliable and fast check? This behavior very much c-like, and in my case it was only discovered by accident in production code that's been out for a couple of years. Best, Anton From sturla.molden at gmail.com Mon Jul 27 16:11:52 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 27 Jul 2015 20:11:52 +0000 (UTC) Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? References: <55B25F1A.70107@googlemail.com> <2053988702459449658.301963sturla.molden-gmail.com@news.gmane.org> <963006654459627038.928066sturla.molden-gmail.com@news.gmane.org> Message-ID: <1260168100459720181.466026sturla.molden-gmail.com@news.gmane.org> Chris Barker wrote: > 32 bits on all (most) 32 bit platforms > 64 bits on 64 bit Linux and OS-X > 32 bits on 64 bit Windows (also if compiled by cygwin??) sizeof(long) is 8 on 64-bit Cygwin. This is to make sure it is inconsistent with MSVC and MinGW-w64, and make sure there will always be ABI mismatches unless the headerfiles are modified accordingly. OTOH, it is one only sane 64-bit compiler on Windows. You can actually take code written for 64 bit Linux or OSX and expect that it will work correctly. Sturla From sturla.molden at gmail.com Mon Jul 27 16:51:52 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 27 Jul 2015 22:51:52 +0200 Subject: [Numpy-discussion] Shared memory check on in-place modification. In-Reply-To: References: Message-ID: On 27/07/15 22:10, Anton Akhmerov wrote: > Hi everyone, > > I have encountered an initially rather confusing problem in a piece of > code that attempted to symmetrize a matrix: `h += h.T` > The problem of course appears due to `h.T` being a view of `h`, and > some elements being overwritten during the __iadd__ call. Here is another example >>> a = np.ones(10) >>> a[1:] += a[:-1] >>> a array([ 1., 2., 3., 2., 3., 2., 3., 2., 3., 2.]) I am not sure I totally dislike this behavior. If it could be made constent it could be used to vectorize recursive algorithms. In the case above I would prefer the output to be: array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.]) It does not happen because we do not enforce that the result of one operation is stored before the next two operands are read. The only way to speed up recursive equations today is to use compiled code. Sturla From sebastian at sipsolutions.net Tue Jul 28 11:45:07 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 28 Jul 2015 15:45:07 +0000 Subject: [Numpy-discussion] np.nonzero - order guarantees? In-Reply-To: References: Message-ID: <43dhsc.ns7fr8.1hge160-qmf@sipsolutions.net> Yes, I think it is guaranteed C order in the results. On Mon Jul 27 14:05:01 2015 GMT+0200, Matthew Brett wrote: > Hi, > > `np.nonzero` for a 2D array `A` returns: > > row_inds, col_inds = np.nonzero(A) > > I notice that `row_inds` appears to be sorted by value, and `col_inds` > appears to be sorted by value, within each row. > > Is this a guarantee of the `np.nonzero` function? If not, does this > function guarantee any property of the returned indices, other than > the correspondence of the row, column entries? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From sebastian at sipsolutions.net Tue Jul 28 11:59:16 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 28 Jul 2015 15:59:16 +0000 Subject: [Numpy-discussion] Shared memory check on in-place modification. In-Reply-To: References: Message-ID: On Mon Jul 27 22:51:52 2015 GMT+0200, Sturla Molden wrote: > On 27/07/15 22:10, Anton Akhmerov wrote: > > Hi everyone, > > > > I have encountered an initially rather confusing problem in a piece of > > code that attempted to symmetrize a matrix: `h += h.T` > > The problem of course appears due to `h.T` being a view of `h`, and > > some elements being overwritten during the __iadd__ call. > I think the typical proposal is to raise a warning. Note there is np.may_share_memoty. But the logic to give the warning is possibly not quite easy, since this is ok to use sometimes. If someone figures it out (mostly) I would be very happy zo see such warnings. > Here is another example > > >>> a = np.ones(10) > >>> a[1:] += a[:-1] > >>> a > array([ 1., 2., 3., 2., 3., 2., 3., 2., 3., 2.]) > > I am not sure I totally dislike this behavior. If it could be made > constent it could be used to vectorize recursive algorithms. In the case > above I would prefer the output to be: > > array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.]) > > It does not happen because we do not enforce that the result of one > operation is stored before the next two operands are read. The only way > to speed up recursive equations today is to use compiled code. > > > Sturla > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthew.brett at gmail.com Tue Jul 28 12:18:01 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 28 Jul 2015 17:18:01 +0100 Subject: [Numpy-discussion] np.nonzero - order guarantees? In-Reply-To: <43dhsc.ns7fr8.1hge160-qmf@sipsolutions.net> References: <43dhsc.ns7fr8.1hge160-qmf@sipsolutions.net> Message-ID: On Tue, Jul 28, 2015 at 4:45 PM, Sebastian Berg wrote: > Yes, I think it is guaranteed C order in the results. > > > On Mon Jul 27 14:05:01 2015 GMT+0200, Matthew Brett wrote: >> Hi, >> >> `np.nonzero` for a 2D array `A` returns: >> >> row_inds, col_inds = np.nonzero(A) >> >> I notice that `row_inds` appears to be sorted by value, and `col_inds` >> appears to be sorted by value, within each row. >> >> Is this a guarantee of the `np.nonzero` function? If not, does this >> function guarantee any property of the returned indices, other than >> the correspondence of the row, column entries? Joscha Reimer just pointed out that this is not guaranteed for scipy sparse arrays: https://github.com/scipy/scipy/pull/4875#discussion_r35528827 >>> A = scipy.sparse.coo_matrix(([2, 1], ([1, 0], [0, 1]))) >>> A.todense() matrix([[0, 1], [2, 0]]) >>> A.nonzero() (array([1, 0], dtype=int32), array([0, 1], dtype=int32)) This seems rather dangerous - I mean a convention that is nearly always observed. I guess at very least we should say what the guarantee is (or isn't) in the nonzero docstring? Cheers, Matthew From sebastian at sipsolutions.net Tue Jul 28 13:52:20 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 28 Jul 2015 17:52:20 +0000 Subject: [Numpy-discussion] np.nonzero - order guarantees? In-Reply-To: References: <43dhsc.ns7fr8.1hge160-qmf@sipsolutions.net> Message-ID: On Tue Jul 28 18:18:01 2015 GMT+0200, Matthew Brett wrote: > On Tue, Jul 28, 2015 at 4:45 PM, Sebastian Berg > wrote: > > Yes, I think it is guaranteed C order in the results. > > > > > > On Mon Jul 27 14:05:01 2015 GMT+0200, Matthew Brett wrote: > >> Hi, > >> > >> `np.nonzero` for a 2D array `A` returns: > >> > >> row_inds, col_inds = np.nonzero(A) > >> > >> I notice that `row_inds` appears to be sorted by value, and `col_inds` > >> appears to be sorted by value, within each row. > >> > >> Is this a guarantee of the `np.nonzero` function? If not, does this > >> function guarantee any property of the returned indices, other than > >> the correspondence of the row, column entries? > > Joscha Reimer just pointed out that this is not guaranteed for scipy > sparse arrays: > > https://github.com/scipy/scipy/pull/4875#discussion_r35528827 > > >>> A = scipy.sparse.coo_matrix(([2, 1], ([1, 0], [0, 1]))) > >>> A.todense() > matrix([[0, 1], > [2, 0]]) > >>> A.nonzero() > (array([1, 0], dtype=int32), array([0, 1], dtype=int32)) > > This seems rather dangerous - I mean a convention that is nearly > always observed. I guess at very least we should say what the > guarantee is (or isn't) in the nonzero docstring? > Yes. We should double check and document it. But if it were to change, I would go for an order keyword anyway. -Sebastian > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ndbecker2 at gmail.com Wed Jul 29 08:07:08 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 29 Jul 2015 08:07:08 -0400 Subject: [Numpy-discussion] strange casting rules Message-ID: np.uint64(-1)+0 Out[36]: 1.8446744073709552e+19 I often work on signal processing requiring bit-exact integral arithmetic. Promoting to float is not helpful - I don't understand the logic of the above example. From robert.kern at gmail.com Wed Jul 29 08:31:02 2015 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Jul 2015 13:31:02 +0100 Subject: [Numpy-discussion] strange casting rules In-Reply-To: References: Message-ID: On Wed, Jul 29, 2015 at 1:07 PM, Neal Becker wrote: > > np.uint64(-1)+0 > Out[36]: 1.8446744073709552e+19 > > I often work on signal processing requiring bit-exact integral arithmetic. > Promoting to float is not helpful - I don't understand the logic of the > above example. See this thread: http://mail.scipy.org/pipermail/numpy-discussion/2015-July/073196.html Cast your 0 to a uint64 or other unsigned int type to avoid this. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Jul 29 11:08:15 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 29 Jul 2015 11:08:15 -0400 Subject: [Numpy-discussion] stats functions with weight support Message-ID: The histogram function supports a weights option, but most others (e.g., percentile) do not. For my problem, I have a trace of the amounts of time (floating point) that my machine under test is in each of N states. I'd like to produce histograms, kde, maybe nice pics with seaborn. I can use weights option to histogram, but cannot do any fancier ops, because they don't accept a weights= input. For that matter, it would even perhaps be nicer if they accepted a histogram input directly. I looked at seaborn distplot, but it doesn't accept weights and doesn't accept a histogram. I thought maybe I could modify it, but it uses functions like percentile, which also doesn't accept either. From ndjensen at gmail.com Wed Jul 29 17:16:04 2015 From: ndjensen at gmail.com (Nathan Jensen) Date: Wed, 29 Jul 2015 16:16:04 -0500 Subject: [Numpy-discussion] numpy where and dtype in 1.9 Message-ID: Hi, The numpy.where() function was rewritten in numpy 1.9 to speed it up. I traced it to this changeset. https://github.com/numpy/numpy/commit/593e3c30c24f0c61a271dc883c614724d7a57e1e The weird thing is the 1.9 behavior changed the resulting dtype in some situations when using scalar values as the second or third argument. To try and illustrate, I wrote a simple test script and ran it against both numpy 1.7 and 1.9. Here are the results: 2.7.9 (default, Jul 25 2015, 03:06:43) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] ***** numpy version 1.7.2 ***** === testing numpy.where with NaNs === numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype float64 numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype float32 numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype float64 numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype float64 === testing numpy.where with integers === numpy.where([True], [numpy.float32(1.0)], 65535).dtype float32 numpy.where([True], [numpy.float32(1.0)], 65536).dtype float32 numpy.where([True], [numpy.float32(1.0)], -32768).dtype float32 numpy.where([True], [numpy.float32(1.0)], -32769).dtype float32 2.7.9 (default, Mar 10 2015, 09:26:44) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] ***** numpy version 1.9.2 ***** === testing numpy.where with NaNs === numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype float64 numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype float32 numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype float64 numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype float64 === testing numpy.where with integers === numpy.where([True], [numpy.float32(1.0)], 65535).dtype float32 numpy.where([True], [numpy.float32(1.0)], 65536).dtype float64 numpy.where([True], [numpy.float32(1.0)], -32768).dtype float32 numpy.where([True], [numpy.float32(1.0)], -32769).dtype float64 Regarding the NaNs with where, the behavior does not differ between 1.7 and 1.9. But it's a little odd that the one scenario returns a dtype of float32 where the other three scenarios return dtype of float64. I'm not sure if that was intentional or a bug? Regarding using ints with where, in 1.7 the resulting dtype is consistent but then in 1.9 the resulting dtype is influenced by the value of the int. It appears it is somehow related to whether the value falls within the range of a short. I'm not sure if this was a side effect of the performance improvement or was intentional? At the very least I think this change in where() should probably be noted in the release notes for 1.9. Our project saw an increase in memory usage with 1.9 due to where(cond, array, scalar) returning arrays of dtype float64 when using scalars not within that limited range. I've attached my simple script if you're interested in running it. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testNumpyWhere.py Type: text/x-python Size: 1247 bytes Desc: not available URL: From ben.root at ou.edu Wed Jul 29 17:33:08 2015 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 29 Jul 2015 17:33:08 -0400 Subject: [Numpy-discussion] numpy where and dtype in 1.9 In-Reply-To: References: Message-ID: What a coincidence! A very related bug just got re-opened today at my behest: https://github.com/numpy/numpy/issues/5095 Not the same, but I wouldn't be surprised if it stems from the same sources. The short of it... np.where(x, 0, x) where x is a masked array, will return a masked array in 1.8.2 and earlier, but will return a regular numpy array in 1.9 and above (drops the mask). That bug took a long time for me to track down! Ben Root On Wed, Jul 29, 2015 at 5:16 PM, Nathan Jensen wrote: > Hi, > > The numpy.where() function was rewritten in numpy 1.9 to speed it up. I > traced it to this changeset. > https://github.com/numpy/numpy/commit/593e3c30c24f0c61a271dc883c614724d7a57e1e > > The weird thing is the 1.9 behavior changed the resulting dtype in some > situations when using scalar values as the second or third argument. To > try and illustrate, I wrote a simple test script and ran it against both > numpy 1.7 and 1.9. Here are the results: > > 2.7.9 (default, Jul 25 2015, 03:06:43) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] > ***** numpy version 1.7.2 ***** > > === testing numpy.where with NaNs === > numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype > float64 > numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype > float32 > numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype > float64 > numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype > float64 > > > === testing numpy.where with integers === > numpy.where([True], [numpy.float32(1.0)], 65535).dtype > float32 > numpy.where([True], [numpy.float32(1.0)], 65536).dtype > float32 > numpy.where([True], [numpy.float32(1.0)], -32768).dtype > float32 > numpy.where([True], [numpy.float32(1.0)], -32769).dtype > float32 > > > > 2.7.9 (default, Mar 10 2015, 09:26:44) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] > ***** numpy version 1.9.2 ***** > > === testing numpy.where with NaNs === > numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype > float64 > numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype > float32 > numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype > float64 > numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype > float64 > > > === testing numpy.where with integers === > numpy.where([True], [numpy.float32(1.0)], 65535).dtype > float32 > numpy.where([True], [numpy.float32(1.0)], 65536).dtype > float64 > numpy.where([True], [numpy.float32(1.0)], -32768).dtype > float32 > numpy.where([True], [numpy.float32(1.0)], -32769).dtype > float64 > > > > Regarding the NaNs with where, the behavior does not differ between 1.7 > and 1.9. But it's a little odd that the one scenario returns a dtype of > float32 where the other three scenarios return dtype of float64. I'm not > sure if that was intentional or a bug? > > Regarding using ints with where, in 1.7 the resulting dtype is consistent > but then in 1.9 the resulting dtype is influenced by the value of the int. > It appears it is somehow related to whether the value falls within the > range of a short. I'm not sure if this was a side effect of the > performance improvement or was intentional? > > At the very least I think this change in where() should probably be noted > in the release notes for 1.9. Our project saw an increase in memory usage > with 1.9 due to where(cond, array, scalar) returning arrays of dtype > float64 when using scalars not within that limited range. > > I've attached my simple script if you're interested in running it. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndjensen at gmail.com Thu Jul 30 13:22:21 2015 From: ndjensen at gmail.com (Nathan Jensen) Date: Thu, 30 Jul 2015 12:22:21 -0500 Subject: [Numpy-discussion] numpy where and dtype in 1.9 In-Reply-To: References: Message-ID: Thanks for the link. I'm glad I'm not the only one tripping over the where() changes. Should I open a new ticket for what I've encountered, or just add a comment to 5095 that the behavior of the output's dtype is also different? It doesn't sound like it's going to be fixed in 1.9, so I'm not sure what path forward my software team will take. We'll either have to move to 1.8 or try to explicitly cast it to float32 throughout the software. Given the size of the ndarrays the software works with, we really need those extra 32 bits. :-) On Wed, Jul 29, 2015 at 4:33 PM, Benjamin Root wrote: > What a coincidence! A very related bug just got re-opened today at my > behest: https://github.com/numpy/numpy/issues/5095 > > Not the same, but I wouldn't be surprised if it stems from the same > sources. The short of it... > > np.where(x, 0, x) > > where x is a masked array, will return a masked array in 1.8.2 and > earlier, but will return a regular numpy array in 1.9 and above (drops the > mask). > That bug took a long time for me to track down! > > Ben Root > > On Wed, Jul 29, 2015 at 5:16 PM, Nathan Jensen wrote: > >> Hi, >> >> The numpy.where() function was rewritten in numpy 1.9 to speed it up. I >> traced it to this changeset. >> https://github.com/numpy/numpy/commit/593e3c30c24f0c61a271dc883c614724d7a57e1e >> >> The weird thing is the 1.9 behavior changed the resulting dtype in some >> situations when using scalar values as the second or third argument. To >> try and illustrate, I wrote a simple test script and ran it against both >> numpy 1.7 and 1.9. Here are the results: >> >> 2.7.9 (default, Jul 25 2015, 03:06:43) >> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] >> ***** numpy version 1.7.2 ***** >> >> === testing numpy.where with NaNs === >> numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype >> float64 >> numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype >> float32 >> numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype >> float64 >> numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype >> float64 >> >> >> === testing numpy.where with integers === >> numpy.where([True], [numpy.float32(1.0)], 65535).dtype >> float32 >> numpy.where([True], [numpy.float32(1.0)], 65536).dtype >> float32 >> numpy.where([True], [numpy.float32(1.0)], -32768).dtype >> float32 >> numpy.where([True], [numpy.float32(1.0)], -32769).dtype >> float32 >> >> >> >> 2.7.9 (default, Mar 10 2015, 09:26:44) >> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] >> ***** numpy version 1.9.2 ***** >> >> === testing numpy.where with NaNs === >> numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype >> float64 >> numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype >> float32 >> numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype >> float64 >> numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype >> float64 >> >> >> === testing numpy.where with integers === >> numpy.where([True], [numpy.float32(1.0)], 65535).dtype >> float32 >> numpy.where([True], [numpy.float32(1.0)], 65536).dtype >> float64 >> numpy.where([True], [numpy.float32(1.0)], -32768).dtype >> float32 >> numpy.where([True], [numpy.float32(1.0)], -32769).dtype >> float64 >> >> >> >> Regarding the NaNs with where, the behavior does not differ between 1.7 >> and 1.9. But it's a little odd that the one scenario returns a dtype of >> float32 where the other three scenarios return dtype of float64. I'm not >> sure if that was intentional or a bug? >> >> Regarding using ints with where, in 1.7 the resulting dtype is consistent >> but then in 1.9 the resulting dtype is influenced by the value of the int. >> It appears it is somehow related to whether the value falls within the >> range of a short. I'm not sure if this was a side effect of the >> performance improvement or was intentional? >> >> At the very least I think this change in where() should probably be noted >> in the release notes for 1.9. Our project saw an increase in memory usage >> with 1.9 due to where(cond, array, scalar) returning arrays of dtype >> float64 when using scalars not within that limited range. >> >> I've attached my simple script if you're interested in running it. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jul 31 01:55:51 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 31 Jul 2015 07:55:51 +0200 Subject: [Numpy-discussion] ANN: PyWavelets 0.3.0 release Message-ID: Dear all, On behalf of the PyWavelets development team I'm excited to announce the availability of PyWavelets 0.3.0. This is the first release of the package in 3 years. It is the result of a significant effort of a growing development team to modernize the package, to provide Python 3.x support and to make a start with providing new features as well as improved performance. A 0.4.0 release will follow shortly, and will contain more significant new features as well as changes/deprecations to streamline the API. This release requires Python 2.6, 2.7 or 3.3-3.5 and Numpy 1.6.2 or greater. Sources and release notes can be found on https://pypi.python.org/pypi/PyWavelets and https://github.com/PyWavelets/pywt/releases. Activity on the project is picking up quickly. If you're interested in wavelets in Python, you are welcome and invited to join us at https://github.com/PyWavelets/pywt Enjoy, Ralf ============================== PyWavelets 0.3.0 Release Notes ============================== PyWavelets 0.3.0 is the first release of the package in 3 years. It is the result of a significant effort of a growing development team to modernize the package, to provide Python 3.x support and to make a start with providing new features as well as improved performance. A 0.4.0 release will follow shortly, and will contain more significant new features as well as changes/deprecations to streamline the API. This release requires Python 2.6, 2.7 or 3.3-3.5 and NumPy 1.6.2 or greater. Highlights of this release include: - Support for Python 3.x (>=3.3) - Added a test suite (based on nose, coverage up to 61% so far) - Maintenance work: C style complying to the Numpy style guide, improved templating system, more complete docstrings, pep8/pyflakes compliance, and more. New features ============ Test suite ---------- The test suite can be run with ``nosetests pywt`` or with:: >>> import pywt >>> pywt.test() n-D Inverse Discrete Wavelet Transform -------------------------------------- The function ``pywt.idwtn``, which provides n-dimensional inverse DWT, has been added. It complements ``idwt``, ``idwt2`` and ``dwtn``. Thresholding ------------ The function `pywt.threshold` has been added. It unifies the four thresholding functions that are still provided in the ``pywt.thresholding`` namespace. Backwards incompatible changes ============================== None in this release. Other changes ============= Development has moved to `a new repo `_. Everyone with an interest in wavelets is welcome to contribute! Building wheels, building with ``python setup.py develop`` and many other standard ways to build and install PyWavelets are supported now. Authors ======= * Ankit Agrawal + * Fran?ois Boulogne + * Ralf Gommers + * David Men?ndez Hurtado + * Gregory R. Lee + * David McInnis + * Helder Oliveira + * Filip Wasilewski * Kai Wohlfahrt + A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nevion at gmail.com Fri Jul 31 02:24:31 2015 From: nevion at gmail.com (Jason Newton) Date: Thu, 30 Jul 2015 23:24:31 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: <55B25F1A.70107@googlemail.com> References: <55B25F1A.70107@googlemail.com> Message-ID: Been using numpy in it's various forms since like 2005. burned on int, int_ just today with boost.python / ndarray conversions and a number of times before that. intc being C's int!? Didn't even know it existed till today. This isn't the first time, esp with float. Bool is actually expected for me and I'd prefer it stay 1 byte for storage efficiency - I'll use a long if I want it machine word wide. This really needs changing though. scientific researchers don't catch this subtlety and expect it to be just like the c and matlab types they know a little about. I can't even keep it straight in all circumstances, how can I expect them to? This makes all the newcomers face the same pain and introduce more bugs into otherwise good code. +1 Change it now like ripping off a bandaid. Match C11/C++11 types and solve much pain past present and future in exchange for a few lashings for the remainder of the year. Thankfully stdint like types have existed for quite some times so protocol descriptions have been correct most of the time. -Jason On Fri, Jul 24, 2015 at 8:51 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 07/23/2015 04:29 AM, Nathaniel Smith wrote: > > Hi all, > > > > So one of the things exposed in the numpy namespace are objects called > > np.int > > np.float > > np.bool > > etc. > > > > These are commonly used -- in fact, just yesterday on another project > > I saw a senior person reviewing a pull request instruct a more junior > > person that they should use np.float instead of float or np.float64. > > But AFAICT everyone who is actually using them is doing this based on > > a very easy-to-fall-for misconception, i.e., that these objects have > > something to do with numpy. > > I don't see the issue. They are just aliases so how is np.float worse > than just float? > Too me this does not seem worth the bother of deprecation. > An argument could be made for deprecating creating dtypes from python > builtin types as they are ambiguous (C float != python float) and > platform dependent. E.g. dtype=int is just an endless source of bugs. > But this is also so invasive that the deprecation would never be > completed and just be a bother to everyone. > > So -1 from me. > > > > > > P.S.: using metamodule.py also gives us the option of making > > np.testing lazily imported, which last time this came up was > > benchmarked to improve numpy's import speed by ~35% [1] -- not too bad > > given that most production code will never touch np.testing. But this > > is just a teaser postscript; I'm not proposing that we actually do > > this at this time :-). > > > > [1] > http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063147.html > > > > I doubt these numbers from 2012 are still correct. When this was last > profiled last year the import there were two main offenders, add_docs > and np.polynomial. Both have been fixed in 1.9. I don't recall > np.testing even showing up. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Jul 31 03:38:57 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 31 Jul 2015 09:38:57 +0200 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: References: <55B25F1A.70107@googlemail.com> Message-ID: <55BB2611.10003@googlemail.com> On 31.07.2015 08:24, Jason Newton wrote: > Been using numpy in it's various forms since like 2005. burned on int, > int_ just today with boost.python / ndarray conversions and a number of > times before that. intc being C's int!? Didn't even know it existed > till today. This isn't the first time, esp with float. Bool is > actually expected for me and I'd prefer it stay 1 byte for storage > efficiency - I'll use a long if I want it machine word wide. A long is only machine word wide on posix, in windows its not. This nonsense is unfortunately also in numpy. It also affects dtype=int. The correct word size type is actually np.intp. btw. if something needs deprecating it is np.float128, this is the most confusing type name in all of numpy as its precision is actually a 80 bit in most cases (x86), 64 bit sometimes (arm) and very rarely actually 128 bit (sparc). From chris.barker at noaa.gov Fri Jul 31 11:53:03 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 31 Jul 2015 08:53:03 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: References: <55B25F1A.70107@googlemail.com> Message-ID: On Thu, Jul 30, 2015 at 11:24 PM, Jason Newton wrote: > This really needs changing though. scientific researchers don't catch > this subtlety and expect it to be just like the c and matlab types they > know a little about. > well, C types are a %&$ nightmare as well! In fact, one of the biggest issues comes from cPython's use of a C "long" for an integer -- which is not clearly defined. If you are writing code that needs any kind of binary compatibility, cross platform compatibility, and particularly if you want to be abel to distribute pre-compiled binaries of extensions, etc, then you'd better use well-defined types. numpy has had well-defined types for ages, but it is a shame that it's so easy to use the poorly-defined ones. I can't even keep it straight in all circumstances, how can I expect them > to? This makes all the newcomers face the same pain and introduce more > bugs into otherwise good code. > indeed. > +1 Change it now like ripping off a bandaid. Match C11/C++11 types and > solve much pain past present and future in exchange for a few lashings for > the remainder of the year. > Sorry -- I'm not sure what C11 types are -- is "int", "long", etc, deprecated? If so, then yes. What about Fortan -- I've been out of that loop for ages -- does semi-modern Fortran use well defined integer types? Is it possible to deprecate a bunch of the built-in numpy dtypes? Without annoying the heck out everyone -- because tehre is a LOT of code out there that just uses np.float, np.int, etc..... An argument could be made for deprecating creating dtypes from python >> builtin types as they are ambiguous (C float != python float) and >> platform dependent. E.g. dtype=int is just an endless source of bugs. >> But this is also so invasive that the deprecation would never be >> completed and just be a bother to everyone. >> > yeah, that is a big concern. :-( -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Jul 31 13:45:01 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 31 Jul 2015 17:45:01 +0000 (UTC) Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? References: <55B25F1A.70107@googlemail.com> Message-ID: <1281473095460055826.208120sturla.molden-gmail.com@news.gmane.org> Chris Barker wrote: > What about Fortan -- I've been out of that loop for ages -- does > semi-modern Fortran use well defined integer types? Modern Fortran is completely sane. INTEGER without kind number (Fortran 77) is the fastest integer on the CPU. On AMD64 that is 32 bit, because it is designed to use a 64 bit pointer with a 32 bit offset. (That is also why Microsoft decided to use a 32 bit long, because it by definition is the fastest integer of at least 32 bits. One can actually claim that the C standard is violated with a 64 bit long on AMD64.) Because of this we use a 32 bit interger in BLAS and LAPACK linked to NumPy and SciPy. The function KIND (Fortran 90) allows us to query the kind number of a given variable, e.g. to find out the size of INTEGER and REAL. The function SELECTED_INT_KIND (Fortran 90) returns the kind number of smallest integer with a specified range. The function SELECTED_REAL_KIND (Fortran 90) returns the kind number of smallest float with a given range and precision. THe returned kind number can be used for REAL and COMPLEX. KIND, SELECTED_INT_KIND and SELECTED_REAL_KIND will all return compile-time constants, and can be used to declare other variables if the return value is stored in a variable with the attribute PARAMETER. This allows te programmer to get the REAL, COMPLEX or INTEGER the algorithm needs numerically, without thinking about how big they need to be in bits. ISO_C_BINDING is a Fortran 2003 module which contains kind numbers corresponding to all C types, including size_t and void*, C structs, an attribute for using pass-by-value semantics, controlling the C name to avoid name mangling, as well as functions for converting between C and Fortran pointers. It allows portable interop between C and Fortran (either calling C from Fortran or calling Fortran from C). ISO_FORTRAN_ENV is a Fortran 2003 and 2008 module. In F2003 it contain kind numbers for integers with specified size: INT8, INT16, INT32, and INT64. In F2008 it also contains kind numbers for IEEE floating point types: REAL32, REAL64, and REAL128. The kind numbers for floating point types can also be used to declare complex numbers. So with modern Fortran we have a completely portable and unambiguous type system. C11/C++11 is sane as well, but not quite as sane as that of modern Fortran. Sturla From nickpapior at gmail.com Fri Jul 31 20:19:31 2015 From: nickpapior at gmail.com (Nick Papior) Date: Sat, 1 Aug 2015 02:19:31 +0200 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: References: <55B25F1A.70107@googlemail.com> Message-ID: -- Kind regards Nick Papior On 31 Jul 2015 17:53, "Chris Barker" wrote: > > On Thu, Jul 30, 2015 at 11:24 PM, Jason Newton wrote: >> >> This really needs changing though. scientific researchers don't catch this subtlety and expect it to be just like the c and matlab types they know a little about. > > > well, C types are a %&$ nightmare as well! In fact, one of the biggest issues comes from cPython's use of a C "long" for an integer -- which is not clearly defined. If you are writing code that needs any kind of binary compatibility, cross platform compatibility, and particularly if you want to be abel to distribute pre-compiled binaries of extensions, etc, then you'd better use well-defined types. > > numpy has had well-defined types for ages, but it is a shame that it's so easy to use the poorly-defined ones. > >> I can't even keep it straight in all circumstances, how can I expect them to? This makes all the newcomers face the same pain and introduce more bugs into otherwise good code. > > > indeed. > >> >> +1 Change it now like ripping off a bandaid. Match C11/C++11 types and solve much pain past present and future in exchange for a few lashings for the remainder of the year. > > > Sorry -- I'm not sure what C11 types are -- is "int", "long", etc, deprecated? If so, then yes. > > What about Fortan -- I've been out of that loop for ages -- does semi-modern Fortran use well defined integer types? Yes, this is much like the c equivalent, integer is int, real is float, for long and double constant castings are needed. > > Is it possible to deprecate a bunch of the built-in numpy dtypes? Without annoying the heck out everyone -- because tehre is a LOT of code out there that just uses np.float, np.int, etc..... > > >>> An argument could be made for deprecating creating dtypes from python >>> builtin types as they are ambiguous (C float != python float) and >>> platform dependent. E.g. dtype=int is just an endless source of bugs. >>> But this is also so invasive that the deprecation would never be >>> completed and just be a bother to everyone. > > > yeah, that is a big concern. :-( > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nevion at gmail.com Fri Jul 31 20:46:02 2015 From: nevion at gmail.com (Jason Newton) Date: Fri, 31 Jul 2015 17:46:02 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: References: <55B25F1A.70107@googlemail.com> Message-ID: On Fri, Jul 31, 2015 at 5:19 PM, Nick Papior wrote: > -- > > Kind regards Nick Papior > On 31 Jul 2015 17:53, "Chris Barker" wrote: > > > > On Thu, Jul 30, 2015 at 11:24 PM, Jason Newton wrote: > >> > >> This really needs changing though. scientific researchers don't catch > this subtlety and expect it to be just like the c and matlab types they > know a little about. > > > > > > well, C types are a %&$ nightmare as well! In fact, one of the biggest > issues comes from cPython's use of a C "long" for an integer -- which is > not clearly defined. If you are writing code that needs any kind of binary > compatibility, cross platform compatibility, and particularly if you want > to be abel to distribute pre-compiled binaries of extensions, etc, then > you'd better use well-defined types. > There was some truth to this but if you, like the majority of scientific researchers only produce code for x86 or x86_64 on windows and linux... as long as you aren't treating pointers as int's, everything behaves in accordance to general expectations. The standards did and still do allow for a bit of flux but things like OpenCL [ https://www.khronos.org/registry/cl/sdk/1.0/docs/man/xhtml/scalarDataTypes.html ] made this really strict so we stop writing ifdef's to deal with varying bitwidths and just implement the algorithms - which is typically a researcher?s top priority. I'd say I use the strongly defined types (e.g. int/float32) whenever doing protocol or communications work - it makes complete sense there. But often for computation, especially when interfacing with c extensions it makes more sense for the developer to use types/typenames that ought to match 1:1 with c in every case. -Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Jul 31 20:47:22 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 31 Jul 2015 17:47:22 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: <1281473095460055826.208120sturla.molden-gmail.com@news.gmane.org> References: <55B25F1A.70107@googlemail.com> <1281473095460055826.208120sturla.molden-gmail.com@news.gmane.org> Message-ID: <1051175736349575057@unknownmsgid> So one more bit of anecdotal evidence: I just today revived some Cython code I wrote a couple years ago and haven't tested since. It wraps a C library that uses a lot of "int" typed values. Turns out I was passing in numpy arrays that I had typed as "np.int". It worked OK two years ago when I was testing only on 32 bit pythons, but today I got a bunch of failed tests on 64 bit OS-X -- a np.int is now a C long! I really thought I knew better, even a couple years ago, but I guess it's just too easy to slip up there. Yeah to Cython for keeping types straight (I got a run-time error). And Yeah to me for having at least some basic tests. But Boo to numpy for a very easy to confuse type API. -Chris Sent from my iPhone > On Jul 31, 2015, at 10:45 AM, Sturla Molden wrote: > > Chris Barker wrote: > >> What about Fortan -- I've been out of that loop for ages -- does >> semi-modern Fortran use well defined integer types? > > Modern Fortran is completely sane. > > INTEGER without kind number (Fortran 77) is the fastest integer on the CPU. > On AMD64 that is 32 bit, because it is designed to use a 64 bit pointer > with a 32 bit offset. (That is also why Microsoft decided to use a 32 bit > long, because it by definition is the fastest integer of at least 32 bits. > One can actually claim that the C standard is violated with a 64 bit long > on AMD64.) Because of this we use a 32 bit interger in BLAS and LAPACK > linked to NumPy and SciPy. > > The function KIND (Fortran 90) allows us to query the kind number of a > given variable, e.g. to find out the size of INTEGER and REAL. > > The function SELECTED_INT_KIND (Fortran 90) returns the kind number of > smallest integer with a specified range. > > The function SELECTED_REAL_KIND (Fortran 90) returns the kind number of > smallest float with a given range and precision. THe returned kind number > can be used for REAL and COMPLEX. > > KIND, SELECTED_INT_KIND and SELECTED_REAL_KIND will all return compile-time > constants, and can be used to declare other variables if the return value > is stored in a variable with the attribute PARAMETER. This allows te > programmer to get the REAL, COMPLEX or INTEGER the algorithm needs > numerically, without thinking about how big they need to be in bits. > > ISO_C_BINDING is a Fortran 2003 module which contains kind numbers > corresponding to all C types, including size_t and void*, C structs, an > attribute for using pass-by-value semantics, controlling the C name to > avoid name mangling, as well as functions for converting between C and > Fortran pointers. It allows portable interop between C and Fortran (either > calling C from Fortran or calling Fortran from C). > > ISO_FORTRAN_ENV is a Fortran 2003 and 2008 module. In F2003 it contain kind > numbers for integers with specified size: INT8, INT16, INT32, and INT64. In > F2008 it also contains kind numbers for IEEE floating point types: REAL32, > REAL64, and REAL128. The kind numbers for floating point types can also be > used to declare complex numbers. > > So with modern Fortran we have a completely portable and unambiguous type > system. > > C11/C++11 is sane as well, but not quite as sane as that of modern Fortran. > > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Fri Jul 31 21:13:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 31 Jul 2015 18:13:19 -0700 Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? In-Reply-To: <55B25F1A.70107@googlemail.com> References: <55B25F1A.70107@googlemail.com> Message-ID: On Jul 24, 2015 08:55, "Julian Taylor" wrote: > > On 07/23/2015 04:29 AM, Nathaniel Smith wrote: > > Hi all, > > > > So one of the things exposed in the numpy namespace are objects called > > np.int > > np.float > > np.bool > > etc. > > > > These are commonly used -- in fact, just yesterday on another project > > I saw a senior person reviewing a pull request instruct a more junior > > person that they should use np.float instead of float or np.float64. > > But AFAICT everyone who is actually using them is doing this based on > > a very easy-to-fall-for misconception, i.e., that these objects have > > something to do with numpy. > > I don't see the issue. They are just aliases so how is np.float worse > than just float? Because np.float systematically confuses people in a way that plain float does not. Which is problematic given that we have a lot of users who aren't expert programmers and are easily confused. > Too me this does not seem worth the bother of deprecation. > An argument could be made for deprecating creating dtypes from python > builtin types as they are ambiguous (C float != python float) and > platform dependent. E.g. dtype=int is just an endless source of bugs. > But this is also so invasive that the deprecation would never be > completed and just be a bother to everyone. Yeah, I don't see any way to ever make dtype=int an error, though I can see an argument for making it unconditionally int64 or intp. That's a separate discussion... but every step we can make to simplify these names makes it easier to untangle the overall knot, IMHO. (E.g. if people have different expectations about what int and np.int should mean -- as they obviously do -- then changing the meaning of both of them is harder than deprecating one and then changing the other, so this deprecation puts us in a better position even if it doesn't immediately help much.) > So -1 from me. Do you really mean this as a true veto? While some of the thread has gotten a bit confused about how much of a change we're actually talking about, AFAICT everyone else is very much in favor of this deprecation, including testimony from multiple specific users who have gotten burned. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Jul 31 21:25:58 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 1 Aug 2015 01:25:58 +0000 (UTC) Subject: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.? References: <55B25F1A.70107@googlemail.com> <1281473095460055826.208120sturla.molden-gmail.com@news.gmane.org> <1051175736349575057@unknownmsgid> Message-ID: <1278336859460085079.353193sturla.molden-gmail.com@news.gmane.org> Chris Barker - NOAA Federal wrote: > Turns out I was passing in numpy arrays that I had typed as "np.int". > It worked OK two years ago when I was testing only on 32 bit pythons, > but today I got a bunch of failed tests on 64 bit OS-X -- a np.int is > now a C long! It has always been C long. It is the C long that varies between platforms. Sturla