From jtaylor.debian at googlemail.com Sun Feb 1 12:26:46 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 01 Feb 2015 18:26:46 +0100 Subject: [SciPy-Dev] ANN: NumPy 1.9.2 release candidate Message-ID: <54CE61D6.2090000@googlemail.com> Hi, We have finished the first release candidate of NumPy 1.9.2. The 1.9.2 release will as usual be a bugfix only release to the 1.9.x series. The tarballs and win32 binaries are available on sourceforge: https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ If no regressions show up the final release is planned next week. The upgrade is recommended for all users of the 1.9.x series. Following issues have been fixed: * gh-5316: fix too large dtype alignment of strings and complex types * gh-5424: fix ma.median when used on ndarrays * gh-5481: Fix astype for structured array fields of different byte order * gh-5155: Fix loadtxt with comments=None and a string None data * gh-4476: Masked array view fails if structured dtype has datetime component * gh-5388: Make RandomState.set_state and RandomState.get_state threadsafe * gh-5390: make seed, randint and shuffle threadsafe * gh-5374: Fixed incorrect assert_array_almost_equal_nulp documentation * gh-5393: Add support for ATLAS > 3.9.33. * gh-5313: PyArray_AsCArray caused segfault for 3d arrays * gh-5492: handle out of memory in rfftf * gh-4181: fix a few bugs in the random.pareto docstring * gh-5359: minor changes to linspace docstring * gh-4723: fix a compile issues on AIX Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ Cheers, The NumPy Developer team -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From matthew.brett at gmail.com Sun Feb 1 18:53:48 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 1 Feb 2015 15:53:48 -0800 Subject: [SciPy-Dev] ANN: NumPy 1.9.2 release candidate In-Reply-To: <54CE61D6.2090000@googlemail.com> References: <54CE61D6.2090000@googlemail.com> Message-ID: Hi, On Sun, Feb 1, 2015 at 9:26 AM, Julian Taylor wrote: > Hi, > > We have finished the first release candidate of NumPy 1.9.2. > The 1.9.2 release will as usual be a bugfix only release to the 1.9.x > series. > The tarballs and win32 binaries are available on sourceforge: > https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ > > If no regressions show up the final release is planned next week. > The upgrade is recommended for all users of the 1.9.x series. > > Following issues have been fixed: > * gh-5316: fix too large dtype alignment of strings and complex types > * gh-5424: fix ma.median when used on ndarrays > * gh-5481: Fix astype for structured array fields of different byte order > * gh-5155: Fix loadtxt with comments=None and a string None data > * gh-4476: Masked array view fails if structured dtype has datetime > component > * gh-5388: Make RandomState.set_state and RandomState.get_state threadsafe > * gh-5390: make seed, randint and shuffle threadsafe > * gh-5374: Fixed incorrect assert_array_almost_equal_nulp documentation > * gh-5393: Add support for ATLAS > 3.9.33. > * gh-5313: PyArray_AsCArray caused segfault for 3d arrays > * gh-5492: handle out of memory in rfftf > * gh-4181: fix a few bugs in the random.pareto docstring > * gh-5359: minor changes to linspace docstring > * gh-4723: fix a compile issues on AIX > > Source tarballs, windows installers and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ I built wheels for OSX testing, via the automated travis builders [1]. Install with: pip install -f http://wheels.scipy.org -U --pre numpy Scipy ecosystem tests (scipy, pandas, etc) running against the rc1 wheel at [2]. Cheers, Matthew [1] https://travis-ci.org/MacPython/numpy-wheels [2] https://travis-ci.org/MacPython/scipy-stack-osx-testing From davidmenhur at gmail.com Mon Feb 2 06:16:05 2015 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Mon, 2 Feb 2015 12:16:05 +0100 Subject: [SciPy-Dev] [Numpy-discussion] ANN: NumPy 1.9.2 release candidate In-Reply-To: References: <54CE61D6.2090000@googlemail.com> Message-ID: All tests passing on Fedora 21. Also, ATLAS 3.10 is correctly recognised. Thanks! np.show_config() atlas_3_10_blas_threads_info: libraries = ['tatlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"\\"3.10.1\\""')] language = c include_dirs = ['/usr/include'] lapack_opt_info: libraries = ['tatlas', 'tatlas', 'tatlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.10.1\\""')] language = f77 include_dirs = ['/usr/include'] blas_opt_info: libraries = ['tatlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"\\"3.10.1\\""')] language = c include_dirs = ['/usr/include'] openblas_info: NOT AVAILABLE openblas_lapack_info: NOT AVAILABLE atlas_3_10_threads_info: libraries = ['tatlas', 'tatlas', 'tatlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.10.1\\""')] language = f77 include_dirs = ['/usr/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE On 2 February 2015 at 00:53, Matthew Brett wrote: > Hi, > > On Sun, Feb 1, 2015 at 9:26 AM, Julian Taylor > wrote: >> Hi, >> >> We have finished the first release candidate of NumPy 1.9.2. >> The 1.9.2 release will as usual be a bugfix only release to the 1.9.x >> series. >> The tarballs and win32 binaries are available on sourceforge: >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ >> >> If no regressions show up the final release is planned next week. >> The upgrade is recommended for all users of the 1.9.x series. >> >> Following issues have been fixed: >> * gh-5316: fix too large dtype alignment of strings and complex types >> * gh-5424: fix ma.median when used on ndarrays >> * gh-5481: Fix astype for structured array fields of different byte order >> * gh-5155: Fix loadtxt with comments=None and a string None data >> * gh-4476: Masked array view fails if structured dtype has datetime >> component >> * gh-5388: Make RandomState.set_state and RandomState.get_state threadsafe >> * gh-5390: make seed, randint and shuffle threadsafe >> * gh-5374: Fixed incorrect assert_array_almost_equal_nulp documentation >> * gh-5393: Add support for ATLAS > 3.9.33. >> * gh-5313: PyArray_AsCArray caused segfault for 3d arrays >> * gh-5492: handle out of memory in rfftf >> * gh-4181: fix a few bugs in the random.pareto docstring >> * gh-5359: minor changes to linspace docstring >> * gh-4723: fix a compile issues on AIX >> >> Source tarballs, windows installers and release notes can be found at >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/ > > I built wheels for OSX testing, via the automated travis builders [1]. > > Install with: > > pip install -f http://wheels.scipy.org -U --pre numpy > > Scipy ecosystem tests (scipy, pandas, etc) running against the rc1 wheel at [2]. > > Cheers, > > Matthew > > [1] https://travis-ci.org/MacPython/numpy-wheels > [2] https://travis-ci.org/MacPython/scipy-stack-osx-testing > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From maniteja.modesty067 at gmail.com Mon Feb 2 08:30:36 2015 From: maniteja.modesty067 at gmail.com (Maniteja Nandana) Date: Mon, 2 Feb 2015 19:00:36 +0530 Subject: [SciPy-Dev] Regarding taking up project ideas and GSoC 2015 Message-ID: Hello everyone, I am a third year computer science undergraduate student, from BITS Pilani, India. First of all, I have cross posted to both the mailing lists, since there is a large overlap among contributors of the organisations. Please do mention if there is any objection to do so. I am writing this mail to introduce myself and to try to work on ideas that are hovering around, if possible contribute as much as I can to the community. Apologies in advance for the lengthy mail. I have been in active touch with numpy and scipy since December, though I have been following the mailing lists and discussions for quite some time. There is an ambience which makes me stick to the github page all the time, just so that I keep learning new things and as well as help others if possible. I think I have a working knowledge of git, though not at advanced level, which I learned when working on small patches to numpy and scipy, thanks for the help there. Though not greatly familiar with the structure of modules in the library, I have a basic understanding of the codebase, getting to know things as and when I have been digging at them, during various issues. I have recently looked into the Scipy 1.0 Roadmap, todo wiki and Project Ideas on github wiki. I was interested in the numpy idea for pythonic dtypes and also in the scipy interpolation module, which were close to my academic fields. I would also be eager to learn about the other ideas, but as they are not a part of my curriculum, I would be needing to spend some time to get an idea of them. Also there are various enhancement proposals, mentioned in issues and PRs like the elliptical integrals of third kind, which was brought up recently. It would be great to try to work on such ideas, but need your guidance. I am a pursing a computer science major, and completed courses in linear algebra, differential equations, computer graphics, probability and statistics, machine learning, information retrieval and data mining. I have beginner knowledge in image and signal processing. I was wondering if I could request the concerned people who work in these fields to help beginners like me to look into and learn about the concepts. It would be a really great learning experience to work under you guys, and start as early as possible, irrespective of whether it is a feasible SoC project, but only according to your convenience. I would be more than pleased and happy to be a long term contributor. My github profile is https://github.com/maniteja123. It is just a mix of my modest beginning to open source contribution and my academic projects. Waiting in anticipation for your response. Thanks for spending time to read along my lengthy mail. Cheers, N.Maniteja _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion SciPy-Dev mailing list SciPy-Dev at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Feb 9 04:48:50 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 09 Feb 2015 10:48:50 +0100 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: References: <54BBBA46.2070802@gmail.com> Message-ID: On 19/01/15 00:17, Sturla Molden wrote: > On 18/01/15 15:10, josef.pktd at gmail.com wrote: > > >> I don't know how well the current lstsq is doing since I never checked. >> Using scipy pinv (SVD based ?) for linear regression works pretty well >> in those cases. > > pinv uses SVD, but there is a big difference: You are refering to > using SVD on the scatter matrix X'X, whereas SciPy uses SVD on the > data matrix X for the least-squares solver. > I misunderstood Josef, he was talking about pinv(X) not pinv(X'X). So he was talking about dot(pinv(X),y) and not what I thought dot(dot(pinv(dot(X.T,X)),X.T),y) The latter would be the naive transcription from a linear regression textbook's inverse(X' X) X' y So what is the difference between lstsq(X,y) and dot(pinv(X),y)? It turns out: "almost nothing". If we look at the equations that LAPACK solves, then lstsq(X,y) with SVD actually forms pinv(X) internally in LAPACK and then matrix multiplies with y. But if we do dot(pinv(X),y) then this has a small amount of redundant computation in it: np.linalg.pinv(X) calls lstsq with X and an indentity matrix. So internally LAPACK forms pinv(x) and then multiplies pinv(X) with I to return pinv(X). Calling lstsq(X,y) instead of dot(pinv(X),y) avoids this one redundant matrix multiplication, but otherwise they are identical from a computational perspective. The dot product in the latter would be done internally in LAPACK in the former, calling the same BLAS function, so that is just some constant Python overhead. The only numerical difference then is the multiplication of pinv(X) with an identity matrix internally in LAPACK. Sturla From josef.pktd at gmail.com Mon Feb 9 08:29:29 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Feb 2015 08:29:29 -0500 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: References: <54BBBA46.2070802@gmail.com> Message-ID: On Mon, Feb 9, 2015 at 4:48 AM, Sturla Molden wrote: > On 19/01/15 00:17, Sturla Molden wrote: >> On 18/01/15 15:10, josef.pktd at gmail.com wrote: >> >> >>> I don't know how well the current lstsq is doing since I never checked. >>> Using scipy pinv (SVD based ?) for linear regression works pretty well >>> in those cases. >> >> pinv uses SVD, but there is a big difference: You are refering to >> using SVD on the scatter matrix X'X, whereas SciPy uses SVD on the >> data matrix X for the least-squares solver. >> > I misunderstood Josef, he was talking about pinv(X) not pinv(X'X). > > So he was talking about > > dot(pinv(X),y) > > and not what I thought > > dot(dot(pinv(dot(X.T,X)),X.T),y) > > The latter would be the naive transcription from a linear regression > textbook's > > inverse(X' X) X' y > > So what is the difference between lstsq(X,y) and dot(pinv(X),y)? It > turns out: "almost nothing". > > If we look at the equations that LAPACK solves, then lstsq(X,y) with SVD > actually forms pinv(X) internally in LAPACK and then matrix multiplies > with y. But if we do dot(pinv(X),y) then this has a small amount of > redundant computation in it: np.linalg.pinv(X) calls lstsq with X and an > indentity matrix. So internally LAPACK forms pinv(x) and then multiplies > pinv(X) with I to return pinv(X). Calling lstsq(X,y) instead of > dot(pinv(X),y) avoids this one redundant matrix multiplication, but > otherwise they are identical from a computational perspective. The dot > product in the latter would be done internally in LAPACK in the former, > calling the same BLAS function, so that is just some constant Python > overhead. The only numerical difference then is the multiplication of > pinv(X) with an identity matrix internally in LAPACK. in statsmodels we are using mostly np.linalg.pinv which is just an svd and a dot product (no identity matrix) https://github.com/numpy/numpy/blob/master/numpy/linalg/linalg.py#L1582 or, more precisely, IIRC, for OLS we use our copy of it that also returns the singular values https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tools/tools.py#L381 np.linalg.pinv was considerably faster than scipy.linalg.pinv when I checked it a few years ago (outdated information because code in scipy has changed). (However, there is still a little bit of slack in our usage of pinv/SVD.) Josef > > Sturla > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From sturla.molden at gmail.com Mon Feb 9 09:39:16 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 09 Feb 2015 15:39:16 +0100 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: References: <54BBBA46.2070802@gmail.com> Message-ID: On 09/02/15 14:29, josef.pktd at gmail.com wrote: > in statsmodels we are using mostly np.linalg.pinv which is just an svd > and a dot product > (no identity matrix) Yes. numpy.linalg.pinv just calls SVD and a dot product as you say, and there is no redundant computation. scipy.linalg.pinv calls lstsq with an identity matrix on RHS. > np.linalg.pinv was considerably faster than scipy.linalg.pinv when I > checked it a few years ago > (outdated information because code in scipy has changed). They are still different, it appears, but I am not sure which is faster. NumPy's pinv has larger Python overhead but does less computation in BLAS and LAPACK. Both will copy and transpose X if it is in C order. Sturla From ewm at redtetrahedron.org Mon Feb 9 09:49:40 2015 From: ewm at redtetrahedron.org (Eric Moore) Date: Mon, 9 Feb 2015 09:49:40 -0500 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: References: <54BBBA46.2070802@gmail.com> Message-ID: FWIW, this is also what scipy.linalg.pinv2 does. On Mon, Feb 9, 2015 at 9:39 AM, Sturla Molden wrote: > On 09/02/15 14:29, josef.pktd at gmail.com wrote: > > > in statsmodels we are using mostly np.linalg.pinv which is just an svd > > and a dot product > > (no identity matrix) > > Yes. > > numpy.linalg.pinv just calls SVD and a dot product as you say, and there > is no redundant computation. > > scipy.linalg.pinv calls lstsq with an identity matrix on RHS. > > > > np.linalg.pinv was considerably faster than scipy.linalg.pinv when I > > checked it a few years ago > > (outdated information because code in scipy has changed). > > They are still different, it appears, but I am not sure which is faster. > > NumPy's pinv has larger Python overhead but does less computation in > BLAS and LAPACK. > > Both will copy and transpose X if it is in C order. > > > Sturla > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Feb 9 10:12:54 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Feb 2015 15:12:54 +0000 (UTC) Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal References: <54BBBA46.2070802@gmail.com> Message-ID: <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> Eric Moore wrote: > FWIW, this is also what scipy.linalg.pinv2 does. Why is there two of them? Sturla From robert.kern at gmail.com Mon Feb 9 10:15:31 2015 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 9 Feb 2015 15:15:31 +0000 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> References: <54BBBA46.2070802@gmail.com> <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Feb 9, 2015 at 3:12 PM, Sturla Molden wrote: > > Eric Moore wrote: > > FWIW, this is also what scipy.linalg.pinv2 does. > > Why is there two of them? Two different methods of calculation, as the docstrings explain. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Feb 9 10:21:56 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Feb 2015 10:21:56 -0500 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: References: <54BBBA46.2070802@gmail.com> <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Feb 9, 2015 at 10:15 AM, Robert Kern wrote: > On Mon, Feb 9, 2015 at 3:12 PM, Sturla Molden > wrote: >> >> Eric Moore wrote: >> > FWIW, this is also what scipy.linalg.pinv2 does. >> >> Why is there two of them? > > Two different methods of calculation, as the docstrings explain. I thought there are 4 pinv in scipy (slow computer and cannot check right now) some discussion is in PR Josef > > -- > Robert Kern > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From sturla.molden at gmail.com Mon Feb 9 10:29:38 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Feb 2015 15:29:38 +0000 (UTC) Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal References: <54BBBA46.2070802@gmail.com> <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> Message-ID: <1575310027445188209.340151sturla.molden-gmail.com@news.gmane.org> Robert Kern wrote: > Two different methods of calculation, as the docstrings explain. Docstring is wrong. Both methods use SVD and it is the same method of computation. lstsq calls *gelss, which uses SVD internally: http://www.netlib.no/netlib/lapack/double/dgelss.f Sturla From sturla.molden at gmail.com Mon Feb 9 10:33:59 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Feb 2015 15:33:59 +0000 (UTC) Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal References: <54BBBA46.2070802@gmail.com> <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> <1575310027445188209.340151sturla.molden-gmail.com@news.gmane.org> Message-ID: <1155158812445188709.263473sturla.molden-gmail.com@news.gmane.org> Sturla Molden wrote: > Robert Kern wrote: > >> Two different methods of calculation, as the docstrings explain. > > Docstring is wrong. Both methods use SVD and it is the same method of > computation. Perhaps not wrong but at least misleading. The method of computation is the same, with the difference of less Python overhead in pinv compared to pinv2. Sturla From ewm at redtetrahedron.org Mon Feb 9 11:34:49 2015 From: ewm at redtetrahedron.org (Eric Moore) Date: Mon, 9 Feb 2015 11:34:49 -0500 Subject: [SciPy-Dev] Least-Squares Linear Solver ( scipy.linalg.lstsq ) not optimal In-Reply-To: <1155158812445188709.263473sturla.molden-gmail.com@news.gmane.org> References: <54BBBA46.2070802@gmail.com> <1641457780445187516.264089sturla.molden-gmail.com@news.gmane.org> <1575310027445188209.340151sturla.molden-gmail.com@news.gmane.org> <1155158812445188709.263473sturla.molden-gmail.com@news.gmane.org> Message-ID: In practice they behave differently, I suppose due to differences in the default handling of small singular values. On Mon, Feb 9, 2015 at 10:33 AM, Sturla Molden wrote: > Sturla Molden wrote: > > Robert Kern wrote: > > > >> Two different methods of calculation, as the docstrings explain. > > > > Docstring is wrong. Both methods use SVD and it is the same method of > > computation. > > Perhaps not wrong but at least misleading. The method of computation is the > same, with the difference of less Python overhead in pinv compared to > pinv2. > > Sturla > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohangoel0296 at gmail.com Sun Feb 15 15:30:22 2015 From: rohangoel0296 at gmail.com (Rohan Goel) Date: Mon, 16 Feb 2015 02:00:22 +0530 Subject: [SciPy-Dev] GSoC 2015 Message-ID: Dear Developers, I am Rohan Goel , Computer Science undergraduate at BITS Pilani , India and am interested in working for your organization in GSoC 2015. As I am a beginner in open source coding , it would be great help if you guide me where to start from. My primary language of interest is Python. *Thanking You,* *Rohan Goel* *Under Graduate, B.E. (Hons) Computer Science Birla Institute of Technology and Science, Pilani* *K.K Birla Goa Campus* *+91 7722047225 **| rohangoel0296 at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucente at pipeline.com Sun Feb 15 16:11:59 2015 From: rlucente at pipeline.com (Robert Lucente - Pipeline) Date: Sun, 15 Feb 2015 16:11:59 -0500 Subject: [SciPy-Dev] Total Unimodular Matrix Message-ID: <041c01d04964$0937e640$1ba7b2c0$@pipeline.com> I am embarrassed but I can't seem to find the call to determine if a matrix is totally unimodular? From robert.kern at gmail.com Sun Feb 15 16:36:46 2015 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 15 Feb 2015 21:36:46 +0000 Subject: [SciPy-Dev] Total Unimodular Matrix In-Reply-To: <041c01d04964$0937e640$1ba7b2c0$@pipeline.com> References: <041c01d04964$0937e640$1ba7b2c0$@pipeline.com> Message-ID: On Sun, Feb 15, 2015 at 9:11 PM, Robert Lucente - Pipeline < rlucente at pipeline.com> wrote: > > I am embarrassed but I can't seem to find the call to determine if a matrix > is totally unimodular? I don't think we have one. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmcgibbo at gmail.com Tue Feb 17 20:18:53 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Tue, 17 Feb 2015 17:18:53 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? Message-ID: Hey, tl;dir: thinking of writing a PR the following feature & looking for feedback first. In the L-BFGS-B minimizer, a low-memory approximation to the inverse hessian is used internally to determine the descent direction. Once the optimization terminates, none of this information is returned to the client though. I think it would be a nice to return a callable for the inverse hessian - vector product as a LinearOperator. The main use case for this that I have in mind is for computing standard errors in maximum likelihood statistical models. In this use case, you write down a likelihood function (and maybe its gradient), and pass it off to a minimizer to find the MLE. Getting standard errors in a scalar function of the parameters with the delta method , requires computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a vector (the gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is the inverse of the Hessian at the MLE. For large-scale optimization of a likelihood with L-BFGS-B, the hessian is too big to calculate (or invert) explicitly, but a low memory approximation to the inverse is already available inside the optimizer, so I think it makes sense to expose it. The factored form of the BFGS matrix is kind of harry, so the most reasonable route is to return it as a LinearOperator that computes approximate inverse hessian - vector products. This calculation can be done pretty efficiently, I think, with the routine from section 4 of [1]. Does this make sense to people? Is it a desirable feature? I'm thinking of writing a PR for it, but I wanted to check here first before diving down the rabbit hole. [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage ."*Math. Comput.* 35.151 (1980): 773-782. -Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Feb 17 21:05:38 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Feb 2015 21:05:38 -0500 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon wrote: > Hey, > > tl;dir: thinking of writing a PR the following feature & looking for > feedback first. > > In the L-BFGS-B minimizer, a low-memory approximation to the inverse > hessian is > used internally to determine the descent direction. Once the optimization > terminates, > none of this information is returned to the client though. I think it > would be a nice to > return a callable for the inverse hessian - vector product as a > LinearOperator. > > The main use case for this that I have in mind is for computing standard > errors in > maximum likelihood statistical models. In this use case, you write down a > likelihood > function (and maybe its gradient), and pass it off to a minimizer to find > the MLE. Getting > standard errors in a scalar function of the parameters with the delta > method , requires > computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a > vector (the > gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is > the inverse of > the Hessian at the MLE. > > For large-scale optimization of a likelihood with L-BFGS-B, the hessian is > too big to > calculate (or invert) explicitly, but a low memory approximation to the > inverse is > already available inside the optimizer, so I think it makes sense to > expose it. The > factored form of the BFGS matrix is kind of harry, so the most reasonable > route is > to return it as a LinearOperator that computes approximate inverse hessian > - vector > products. This calculation can be done pretty efficiently, I think, with > the routine from > section 4 of [1]. > > Does this make sense to people? Is it a desirable feature? I'm thinking of > writing a > PR for it, but I wanted to check here first before diving down the rabbit > hole. > > [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage > ."*Math. Comput.* > 35.151 (1980): 773-782. > > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. (I remember there was the suggestion once to get the Lagrange multipliers out of one of the constraint optimizers, but when whoever proposed that saw how inaccurate they were, he gave up on the idea.) If it's the only way to get an approximation in large problems, then it might still be worth it. Another question: How is the Hessian calculated if there are binding constraints? The Hessian won't give you the correct standard errors for a boundary solution. Josef > -Robert > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmcgibbo at gmail.com Tue Feb 17 21:20:18 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Tue, 17 Feb 2015 18:20:18 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. Yeah, that might be true. But the downstream approximations that go into calculating error bars from these hessians (that the likelihood is quadratic) can also be pretty bad approximations in many of the situations that they're commonly applied, so the whole enterprise is semi-quantitative at best. In my application, even with some tricks to calculate the Hessian faster, the optimizer can often converge to the MLE _much_ faster than I can calculate a single Hessian. So spending say, 1-10% of my computational effort finding the MLE and 90-99% calculating error bars seems silly when in the end I don't even have a lot of confidence in the approximations that go from hessian -> error bars, and I'm looking at them more from an order of magnitude perspective. As the the behavior for boundary solution, I'm honestly not sure. Obviously the true hessian isn't defined for certain (i,j) pairs, but for other directions that aren't at the boundary it should be okay. How this looks for the BFGS implicit inverse hessian I don't know. -Robert On Tue, Feb 17, 2015 at 6:05 PM, wrote: > > > On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon > wrote: > >> Hey, >> >> tl;dir: thinking of writing a PR the following feature & looking for >> feedback first. >> >> In the L-BFGS-B minimizer, a low-memory approximation to the inverse >> hessian is >> used internally to determine the descent direction. Once the optimization >> terminates, >> none of this information is returned to the client though. I think it >> would be a nice to >> return a callable for the inverse hessian - vector product as a >> LinearOperator. >> >> The main use case for this that I have in mind is for computing standard >> errors in >> maximum likelihood statistical models. In this use case, you write down a >> likelihood >> function (and maybe its gradient), and pass it off to a minimizer to find >> the MLE. Getting >> standard errors in a scalar function of the parameters with the delta >> method , requires >> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a >> vector (the >> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is >> the inverse of >> the Hessian at the MLE. >> >> For large-scale optimization of a likelihood with L-BFGS-B, the hessian >> is too big to >> calculate (or invert) explicitly, but a low memory approximation to the >> inverse is >> already available inside the optimizer, so I think it makes sense to >> expose it. The >> factored form of the BFGS matrix is kind of harry, so the most reasonable >> route is >> to return it as a LinearOperator that computes approximate inverse >> hessian - vector >> products. This calculation can be done pretty efficiently, I think, with >> the routine from >> section 4 of [1]. >> >> Does this make sense to people? Is it a desirable feature? I'm thinking >> of writing a >> PR for it, but I wanted to check here first before diving down the rabbit >> hole. >> >> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage >> ."*Math. Comput.* >> 35.151 (1980): 773-782. >> >> > My guess, without knowing what l-bfgs-b is exactly doing, is that the > approximation might not be very good. > (I remember there was the suggestion once to get the Lagrange multipliers > out of one of the constraint optimizers, but when whoever proposed that saw > how inaccurate they were, he gave up on the idea.) > > If it's the only way to get an approximation in large problems, then it > might still be worth it. > > Another question: How is the Hessian calculated if there are binding > constraints? > The Hessian won't give you the correct standard errors for a boundary > solution. > > Josef > > >> -Robert >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmcgibbo at gmail.com Tue Feb 17 21:27:10 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Tue, 17 Feb 2015 18:27:10 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: I should check the accuracy of the BFGS approx inv. hessian using the (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the accuracy of the low-memory version computed by L-BFGS. -Robert On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon wrote: > > My guess, without knowing what l-bfgs-b is exactly doing, is that the > approximation might not be very good. > > Yeah, that might be true. But the downstream approximations that go into > calculating error bars from these > hessians (that the likelihood is quadratic) can also be pretty bad > approximations in many of the situations that > they're commonly applied, so the whole enterprise is semi-quantitative at > best. > > In my application, even with some tricks to calculate the Hessian faster, > the optimizer can often converge to the > MLE _much_ faster than I can calculate a single Hessian. So spending say, > 1-10% of my computational effort > finding the MLE and 90-99% calculating error bars seems silly when in the > end I don't even have a lot of confidence > in the approximations that go from hessian -> error bars, and I'm looking > at them more from an order of magnitude > perspective. > > As the the behavior for boundary solution, I'm honestly not sure. > Obviously the true hessian isn't defined for certain > (i,j) pairs, but for other directions that aren't at the boundary it > should be okay. How this looks for the BFGS implicit > inverse hessian I don't know. > > -Robert > > > On Tue, Feb 17, 2015 at 6:05 PM, wrote: > >> >> >> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon >> wrote: >> >>> Hey, >>> >>> tl;dir: thinking of writing a PR the following feature & looking for >>> feedback first. >>> >>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse >>> hessian is >>> used internally to determine the descent direction. Once the >>> optimization terminates, >>> none of this information is returned to the client though. I think it >>> would be a nice to >>> return a callable for the inverse hessian - vector product as a >>> LinearOperator. >>> >>> The main use case for this that I have in mind is for computing standard >>> errors in >>> maximum likelihood statistical models. In this use case, you write down >>> a likelihood >>> function (and maybe its gradient), and pass it off to a minimizer to >>> find the MLE. Getting >>> standard errors in a scalar function of the parameters with the delta >>> method , requires >>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a >>> vector (the >>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is >>> the inverse of >>> the Hessian at the MLE. >>> >>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian >>> is too big to >>> calculate (or invert) explicitly, but a low memory approximation to the >>> inverse is >>> already available inside the optimizer, so I think it makes sense to >>> expose it. The >>> factored form of the BFGS matrix is kind of harry, so the most >>> reasonable route is >>> to return it as a LinearOperator that computes approximate inverse >>> hessian - vector >>> products. This calculation can be done pretty efficiently, I think, with >>> the routine from >>> section 4 of [1]. >>> >>> Does this make sense to people? Is it a desirable feature? I'm thinking >>> of writing a >>> PR for it, but I wanted to check here first before diving down the >>> rabbit hole. >>> >>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage >>> ."*Math. Comput.* >>> 35.151 (1980): 773-782. >>> >>> >> My guess, without knowing what l-bfgs-b is exactly doing, is that the >> approximation might not be very good. >> (I remember there was the suggestion once to get the Lagrange multipliers >> out of one of the constraint optimizers, but when whoever proposed that saw >> how inaccurate they were, he gave up on the idea.) >> >> If it's the only way to get an approximation in large problems, then it >> might still be worth it. >> >> Another question: How is the Hessian calculated if there are binding >> constraints? >> The Hessian won't give you the correct standard errors for a boundary >> solution. >> >> Josef >> >> >>> -Robert >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jascha at stanford.edu Tue Feb 17 21:54:25 2015 From: jascha at stanford.edu (Jascha Sohl-Dickstein) Date: Wed, 18 Feb 2015 02:54:25 +0000 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? References: Message-ID: This sounds practically like it's probably a really good test -- but just BTW, it's totally possible for the BFGS approximation to be either worse or better than the LBFGS approximation. The major cause of a difference in terms of accuracy is that the BFGS algorithms uses the full history of gradients and iterates, while LBFGS only uses the recent history of gradients and iterates (think it defaults to the last 10 steps for the scipy implementation). If the Hessian is constant, the BFGS approximation will be much better. If the Hessian changes significantly over the course of optimization, the BFGS approximation may actually turn out to be much worse. -jascha On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon wrote: > I should check the accuracy of the BFGS approx inv. hessian using the > (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. > The accuracy of that matrix is going to be no worse than the accuracy of > the low-memory version computed by L-BFGS. > > -Robert > > On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon > wrote: > >> > My guess, without knowing what l-bfgs-b is exactly doing, is that the >> approximation might not be very good. >> >> Yeah, that might be true. But the downstream approximations that go into >> calculating error bars from these >> hessians (that the likelihood is quadratic) can also be pretty bad >> approximations in many of the situations that >> they're commonly applied, so the whole enterprise is semi-quantitative at >> best. >> >> In my application, even with some tricks to calculate the Hessian >> faster, the optimizer can often converge to the >> MLE _much_ faster than I can calculate a single Hessian. So spending >> say, 1-10% of my computational effort >> finding the MLE and 90-99% calculating error bars seems silly when in >> the end I don't even have a lot of confidence >> in the approximations that go from hessian -> error bars, and I'm looking >> at them more from an order of magnitude >> perspective. >> >> As the the behavior for boundary solution, I'm honestly not sure. >> Obviously the true hessian isn't defined for certain >> (i,j) pairs, but for other directions that aren't at the boundary it >> should be okay. How this looks for the BFGS implicit >> inverse hessian I don't know. >> >> -Robert >> >> >> On Tue, Feb 17, 2015 at 6:05 PM, wrote: >> >>> >>> >>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon >>> wrote: >>> >>>> Hey, >>>> >>>> tl;dir: thinking of writing a PR the following feature & looking for >>>> feedback first. >>>> >>>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse >>>> hessian is >>>> used internally to determine the descent direction. Once the >>>> optimization terminates, >>>> none of this information is returned to the client though. I think it >>>> would be a nice to >>>> return a callable for the inverse hessian - vector product as a >>>> LinearOperator. >>>> >>>> The main use case for this that I have in mind is for computing >>>> standard errors in >>>> maximum likelihood statistical models. In this use case, you write down >>>> a likelihood >>>> function (and maybe its gradient), and pass it off to a minimizer to >>>> find the MLE. Getting >>>> standard errors in a scalar function of the parameters with the delta >>>> method , requires >>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a >>>> vector (the >>>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is >>>> the inverse of >>>> the Hessian at the MLE. >>>> >>>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian >>>> is too big to >>>> calculate (or invert) explicitly, but a low memory approximation to the >>>> inverse is >>>> already available inside the optimizer, so I think it makes sense to >>>> expose it. The >>>> factored form of the BFGS matrix is kind of harry, so the most >>>> reasonable route is >>>> to return it as a LinearOperator that computes approximate inverse >>>> hessian - vector >>>> products. This calculation can be done pretty efficiently, I think, >>>> with the routine from >>>> section 4 of [1]. >>>> >>>> Does this make sense to people? Is it a desirable feature? I'm thinking >>>> of writing a >>>> PR for it, but I wanted to check here first before diving down the >>>> rabbit hole. >>>> >>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage >>>> ."*Math. Comput.* >>>> 35.151 (1980): 773-782. >>>> >>>> >>> My guess, without knowing what l-bfgs-b is exactly doing, is that the >>> approximation might not be very good. >>> (I remember there was the suggestion once to get the Lagrange >>> multipliers out of one of the constraint optimizers, but when whoever >>> proposed that saw how inaccurate they were, he gave up on the idea.) >>> >>> If it's the only way to get an approximation in large problems, then it >>> might still be worth it. >>> >>> Another question: How is the Hessian calculated if there are binding >>> constraints? >>> The Hessian won't give you the correct standard errors for a boundary >>> solution. >>> >>> Josef >>> >>> >>>> -Robert >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Feb 17 22:02:39 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Feb 2015 22:02:39 -0500 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein wrote: > > This sounds practically like it's probably a really good test -- but just BTW, it's totally possible for the BFGS approximation to be either worse or better than the LBFGS approximation. The major cause of a difference in terms of accuracy is that the BFGS algorithms uses the full history of gradients and iterates, while LBFGS only uses the recent history of gradients and iterates (think it defaults to the last 10 steps for the scipy implementation). If the Hessian is constant, the BFGS approximation will be much better. If the Hessian changes significantly over the course of optimization, the BFGS approximation may actually turn out to be much worse. > > -jascha > > On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon wrote: >> >> I should check the accuracy of the BFGS approx inv. hessian using the (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the accuracy of the low-memory version computed by L-BFGS. >> >> -Robert >> >> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon wrote: >>> >>> > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>> >>> Yeah, that might be true. But the downstream approximations that go into calculating error bars from these >>> hessians (that the likelihood is quadratic) can also be pretty bad approximations in many of the situations that >>> they're commonly applied, so the whole enterprise is semi-quantitative at best. >>> >>> In my application, even with some tricks to calculate the Hessian faster, the optimizer can often converge to the >>> MLE _much_ faster than I can calculate a single Hessian. So spending say, 1-10% of my computational effort >>> finding the MLE and 90-99% calculating error bars seems silly when in the end I don't even have a lot of confidence >>> in the approximations that go from hessian -> error bars, and I'm looking at them more from an order of magnitude >>> perspective. >>> >>> >>> As the the behavior for boundary solution, I'm honestly not sure. Obviously the true hessian isn't defined for certain >>> (i,j) pairs, but for other directions that aren't at the boundary it should be okay. How this looks for the BFGS implicit >>> inverse hessian I don't know. >>> >>> -Robert >>> >>> >>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: >>>> >>>> >>>> >>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon wrote: >>>>> >>>>> Hey, >>>>> >>>>> tl;dir: thinking of writing a PR the following feature & looking for feedback first. >>>>> >>>>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse hessian is >>>>> used internally to determine the descent direction. Once the optimization terminates, >>>>> none of this information is returned to the client though. I think it would be a nice to >>>>> return a callable for the inverse hessian - vector product as a LinearOperator. >>>>> >>>>> The main use case for this that I have in mind is for computing standard errors in >>>>> maximum likelihood statistical models. In this use case, you write down a likelihood >>>>> function (and maybe its gradient), and pass it off to a minimizer to find the MLE. Getting >>>>> standard errors in a scalar function of the parameters with the delta method, requires >>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a vector (the >>>>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is the inverse of >>>>> the Hessian at the MLE. >>>>> >>>>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian is too big to >>>>> calculate (or invert) explicitly, but a low memory approximation to the inverse is >>>>> already available inside the optimizer, so I think it makes sense to expose it. The >>>>> factored form of the BFGS matrix is kind of harry, so the most reasonable route is >>>>> to return it as a LinearOperator that computes approximate inverse hessian - vector >>>>> products. This calculation can be done pretty efficiently, I think, with the routine from >>>>> section 4 of [1]. >>>>> >>>>> Does this make sense to people? Is it a desirable feature? I'm thinking of writing a >>>>> PR for it, but I wanted to check here first before diving down the rabbit hole. >>>>> >>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage."Math. Comput. >>>>> 35.151 (1980): 773-782. >>>>> >>>> >>>> My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>> (I remember there was the suggestion once to get the Lagrange multipliers out of one of the constraint optimizers, but when whoever proposed that saw how inaccurate they were, he gave up on the idea.) >>>> >>>> If it's the only way to get an approximation in large problems, then it might still be worth it. >>>> >>>> Another question: How is the Hessian calculated if there are binding constraints? >>>> The Hessian won't give you the correct standard errors for a boundary solution. >>>> >>>> Josef >>>> >>>>> >>>>> -Robert >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From josef.pktd at gmail.com Tue Feb 17 22:04:07 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Feb 2015 22:04:07 -0500 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: On Tue, Feb 17, 2015 at 10:02 PM, wrote: > On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein > wrote: >> >> This sounds practically like it's probably a really good test -- but just BTW, it's totally possible for the BFGS approximation to be either worse or better than the LBFGS approximation. The major cause of a difference in terms of accuracy is that the BFGS algorithms uses the full history of gradients and iterates, while LBFGS only uses the recent history of gradients and iterates (think it defaults to the last 10 steps for the scipy implementation). If the Hessian is constant, the BFGS approximation will be much better. If the Hessian changes significantly over the course of optimization, the BFGS approximation may actually turn out to be much worse. >> >> -jascha >> >> On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon wrote: >>> >>> I should check the accuracy of the BFGS approx inv. hessian using the (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the accuracy of the low-memory version computed by L-BFGS. >>> >>> -Robert >>> >>> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon wrote: >>>> >>>> > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>> >>>> Yeah, that might be true. But the downstream approximations that go into calculating error bars from these >>>> hessians (that the likelihood is quadratic) can also be pretty bad approximations in many of the situations that >>>> they're commonly applied, so the whole enterprise is semi-quantitative at best. >>>> >>>> In my application, even with some tricks to calculate the Hessian faster, the optimizer can often converge to the >>>> MLE _much_ faster than I can calculate a single Hessian. So spending say, 1-10% of my computational effort >>>> finding the MLE and 90-99% calculating error bars seems silly when in the end I don't even have a lot of confidence >>>> in the approximations that go from hessian -> error bars, and I'm looking at them more from an order of magnitude >>>> perspective. > > > > > > >>>> >>>> >>>> As the the behavior for boundary solution, I'm honestly not sure. Obviously the true hessian isn't defined for certain >>>> (i,j) pairs, but for other directions that aren't at the boundary it should be okay. How this looks for the BFGS implicit >>>> inverse hessian I don't know. >>>> >>>> -Robert >>>> >>>> >>>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon wrote: >>>>>> >>>>>> Hey, >>>>>> >>>>>> tl;dir: thinking of writing a PR the following feature & looking for feedback first. >>>>>> >>>>>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse hessian is >>>>>> used internally to determine the descent direction. Once the optimization terminates, >>>>>> none of this information is returned to the client though. I think it would be a nice to >>>>>> return a callable for the inverse hessian - vector product as a LinearOperator. >>>>>> >>>>>> The main use case for this that I have in mind is for computing standard errors in >>>>>> maximum likelihood statistical models. In this use case, you write down a likelihood >>>>>> function (and maybe its gradient), and pass it off to a minimizer to find the MLE. Getting >>>>>> standard errors in a scalar function of the parameters with the delta method, requires >>>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a vector (the >>>>>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is the inverse of >>>>>> the Hessian at the MLE. >>>>>> >>>>>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian is too big to >>>>>> calculate (or invert) explicitly, but a low memory approximation to the inverse is >>>>>> already available inside the optimizer, so I think it makes sense to expose it. The >>>>>> factored form of the BFGS matrix is kind of harry, so the most reasonable route is >>>>>> to return it as a LinearOperator that computes approximate inverse hessian - vector >>>>>> products. This calculation can be done pretty efficiently, I think, with the routine from >>>>>> section 4 of [1]. >>>>>> >>>>>> Does this make sense to people? Is it a desirable feature? I'm thinking of writing a >>>>>> PR for it, but I wanted to check here first before diving down the rabbit hole. >>>>>> >>>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage."Math. Comput. >>>>>> 35.151 (1980): 773-782. >>>>>> >>>>> >>>>> My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>>> (I remember there was the suggestion once to get the Lagrange multipliers out of one of the constraint optimizers, but when whoever proposed that saw how inaccurate they were, he gave up on the idea.) >>>>> >>>>> If it's the only way to get an approximation in large problems, then it might still be worth it. >>>>> >>>>> Another question: How is the Hessian calculated if there are binding constraints? >>>>> The Hessian won't give you the correct standard errors for a boundary solution. >>>>> >>>>> Josef >>>>> >>>>>> >>>>>> -Robert >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>> >>>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> From josef.pktd at gmail.com Tue Feb 17 22:04:14 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Feb 2015 22:04:14 -0500 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: On Tue, Feb 17, 2015 at 10:04 PM, wrote: > On Tue, Feb 17, 2015 at 10:02 PM, wrote: >> On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein >> wrote: >>> >>> This sounds practically like it's probably a really good test -- but just BTW, it's totally possible for the BFGS approximation to be either worse or better than the LBFGS approximation. The major cause of a difference in terms of accuracy is that the BFGS algorithms uses the full history of gradients and iterates, while LBFGS only uses the recent history of gradients and iterates (think it defaults to the last 10 steps for the scipy implementation). If the Hessian is constant, the BFGS approximation will be much better. If the Hessian changes significantly over the course of optimization, the BFGS approximation may actually turn out to be much worse. >>> >>> -jascha >>> >>> On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon wrote: >>>> >>>> I should check the accuracy of the BFGS approx inv. hessian using the (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the accuracy of the low-memory version computed by L-BFGS. >>>> >>>> -Robert >>>> >>>> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon wrote: >>>>> >>>>> > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>>> >>>>> Yeah, that might be true. But the downstream approximations that go into calculating error bars from these >>>>> hessians (that the likelihood is quadratic) can also be pretty bad approximations in many of the situations that >>>>> they're commonly applied, so the whole enterprise is semi-quantitative at best. >>>>> >>>>> In my application, even with some tricks to calculate the Hessian faster, the optimizer can often converge to the >>>>> MLE _much_ faster than I can calculate a single Hessian. So spending say, 1-10% of my computational effort >>>>> finding the MLE and 90-99% calculating error bars seems silly when in the end I don't even have a lot of confidence >>>>> in the approximations that go from hessian -> error bars, and I'm looking at them more from an order of magnitude >>>>> perspective. >> >> >> >> >> >> >>>>> >>>>> >>>>> As the the behavior for boundary solution, I'm honestly not sure. Obviously the true hessian isn't defined for certain >>>>> (i,j) pairs, but for other directions that aren't at the boundary it should be okay. How this looks for the BFGS implicit >>>>> inverse hessian I don't know. >>>>> >>>>> -Robert >>>>> >>>>> >>>>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon wrote: >>>>>>> >>>>>>> Hey, >>>>>>> >>>>>>> tl;dir: thinking of writing a PR the following feature & looking for feedback first. >>>>>>> >>>>>>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse hessian is >>>>>>> used internally to determine the descent direction. Once the optimization terminates, >>>>>>> none of this information is returned to the client though. I think it would be a nice to >>>>>>> return a callable for the inverse hessian - vector product as a LinearOperator. >>>>>>> >>>>>>> The main use case for this that I have in mind is for computing standard errors in >>>>>>> maximum likelihood statistical models. In this use case, you write down a likelihood >>>>>>> function (and maybe its gradient), and pass it off to a minimizer to find the MLE. Getting >>>>>>> standard errors in a scalar function of the parameters with the delta method, requires >>>>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a vector (the >>>>>>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is the inverse of >>>>>>> the Hessian at the MLE. >>>>>>> >>>>>>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian is too big to >>>>>>> calculate (or invert) explicitly, but a low memory approximation to the inverse is >>>>>>> already available inside the optimizer, so I think it makes sense to expose it. The >>>>>>> factored form of the BFGS matrix is kind of harry, so the most reasonable route is >>>>>>> to return it as a LinearOperator that computes approximate inverse hessian - vector >>>>>>> products. This calculation can be done pretty efficiently, I think, with the routine from >>>>>>> section 4 of [1]. >>>>>>> >>>>>>> Does this make sense to people? Is it a desirable feature? I'm thinking of writing a >>>>>>> PR for it, but I wanted to check here first before diving down the rabbit hole. >>>>>>> >>>>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage."Math. Comput. >>>>>>> 35.151 (1980): 773-782. >>>>>>> >>>>>> >>>>>> My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>>>> (I remember there was the suggestion once to get the Lagrange multipliers out of one of the constraint optimizers, but when whoever proposed that saw how inaccurate they were, he gave up on the idea.) >>>>>> >>>>>> If it's the only way to get an approximation in large problems, then it might still be worth it. >>>>>> >>>>>> Another question: How is the Hessian calculated if there are binding constraints? >>>>>> The Hessian won't give you the correct standard errors for a boundary solution. >>>>>> >>>>>> Josef >>>>>> >>>>>>> >>>>>>> -Robert >>>>>>> >>>>>>> _______________________________________________ >>>>>>> SciPy-Dev mailing list >>>>>>> SciPy-Dev at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-Dev mailing list >>>>>> SciPy-Dev at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> From rmcgibbo at gmail.com Tue Feb 17 22:06:52 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Tue, 17 Feb 2015 19:06:52 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: Oh right. Good point, Jascha. -Robert On Tue, Feb 17, 2015 at 7:04 PM, wrote: > On Tue, Feb 17, 2015 at 10:04 PM, wrote: > > On Tue, Feb 17, 2015 at 10:02 PM, wrote: > >> On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein > >> wrote: > >>> > >>> This sounds practically like it's probably a really good test -- but > just BTW, it's totally possible for the BFGS approximation to be either > worse or better than the LBFGS approximation. The major cause of a > difference in terms of accuracy is that the BFGS algorithms uses the full > history of gradients and iterates, while LBFGS only uses the recent history > of gradients and iterates (think it defaults to the last 10 steps for the > scipy implementation). If the Hessian is constant, the BFGS approximation > will be much better. If the Hessian changes significantly over the course > of optimization, the BFGS approximation may actually turn out to be much > worse. > >>> > >>> -jascha > >>> > >>> On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon > wrote: > >>>> > >>>> I should check the accuracy of the BFGS approx inv. hessian using the > (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. > The accuracy of that matrix is going to be no worse than the accuracy of > the low-memory version computed by L-BFGS. > >>>> > >>>> -Robert > >>>> > >>>> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon > wrote: > >>>>> > >>>>> > My guess, without knowing what l-bfgs-b is exactly doing, is that > the approximation might not be very good. > >>>>> > >>>>> Yeah, that might be true. But the downstream approximations that go > into calculating error bars from these > >>>>> hessians (that the likelihood is quadratic) can also be pretty bad > approximations in many of the situations that > >>>>> they're commonly applied, so the whole enterprise is > semi-quantitative at best. > >>>>> > >>>>> In my application, even with some tricks to calculate the Hessian > faster, the optimizer can often converge to the > >>>>> MLE _much_ faster than I can calculate a single Hessian. So spending > say, 1-10% of my computational effort > >>>>> finding the MLE and 90-99% calculating error bars seems silly when > in the end I don't even have a lot of confidence > >>>>> in the approximations that go from hessian -> error bars, and I'm > looking at them more from an order of magnitude > >>>>> perspective. > >> > >> > >> > >> > >> > >> > >>>>> > >>>>> > >>>>> As the the behavior for boundary solution, I'm honestly not sure. > Obviously the true hessian isn't defined for certain > >>>>> (i,j) pairs, but for other directions that aren't at the boundary it > should be okay. How this looks for the BFGS implicit > >>>>> inverse hessian I don't know. > >>>>> > >>>>> -Robert > >>>>> > >>>>> > >>>>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon < > rmcgibbo at gmail.com> wrote: > >>>>>>> > >>>>>>> Hey, > >>>>>>> > >>>>>>> tl;dir: thinking of writing a PR the following feature & looking > for feedback first. > >>>>>>> > >>>>>>> In the L-BFGS-B minimizer, a low-memory approximation to the > inverse hessian is > >>>>>>> used internally to determine the descent direction. Once the > optimization terminates, > >>>>>>> none of this information is returned to the client though. I think > it would be a nice to > >>>>>>> return a callable for the inverse hessian - vector product as a > LinearOperator. > >>>>>>> > >>>>>>> The main use case for this that I have in mind is for computing > standard errors in > >>>>>>> maximum likelihood statistical models. In this use case, you write > down a likelihood > >>>>>>> function (and maybe its gradient), and pass it off to a minimizer > to find the MLE. Getting > >>>>>>> standard errors in a scalar function of the parameters with the > delta method, requires > >>>>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) > is a vector (the > >>>>>>> gradient of the scalar function at the MLE) and the matrix > H^{-1}(x) is the inverse of > >>>>>>> the Hessian at the MLE. > >>>>>>> > >>>>>>> For large-scale optimization of a likelihood with L-BFGS-B, the > hessian is too big to > >>>>>>> calculate (or invert) explicitly, but a low memory approximation > to the inverse is > >>>>>>> already available inside the optimizer, so I think it makes sense > to expose it. The > >>>>>>> factored form of the BFGS matrix is kind of harry, so the most > reasonable route is > >>>>>>> to return it as a LinearOperator that computes approximate inverse > hessian - vector > >>>>>>> products. This calculation can be done pretty efficiently, I > think, with the routine from > >>>>>>> section 4 of [1]. > >>>>>>> > >>>>>>> Does this make sense to people? Is it a desirable feature? I'm > thinking of writing a > >>>>>>> PR for it, but I wanted to check here first before diving down the > rabbit hole. > >>>>>>> > >>>>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited > storage."Math. Comput. > >>>>>>> 35.151 (1980): 773-782. > >>>>>>> > >>>>>> > >>>>>> My guess, without knowing what l-bfgs-b is exactly doing, is that > the approximation might not be very good. > >>>>>> (I remember there was the suggestion once to get the Lagrange > multipliers out of one of the constraint optimizers, but when whoever > proposed that saw how inaccurate they were, he gave up on the idea.) > >>>>>> > >>>>>> If it's the only way to get an approximation in large problems, > then it might still be worth it. > >>>>>> > >>>>>> Another question: How is the Hessian calculated if there are > binding constraints? > >>>>>> The Hessian won't give you the correct standard errors for a > boundary solution. > >>>>>> > >>>>>> Josef > >>>>>> > >>>>>>> > >>>>>>> -Robert > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> SciPy-Dev mailing list > >>>>>>> SciPy-Dev at scipy.org > >>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> SciPy-Dev mailing list > >>>>>> SciPy-Dev at scipy.org > >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>>>> > >>>>> > >>>> > >>>> _______________________________________________ > >>>> SciPy-Dev mailing list > >>>> SciPy-Dev at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>> > >>> > >>> _______________________________________________ > >>> SciPy-Dev mailing list > >>> SciPy-Dev at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Feb 17 22:17:30 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Feb 2015 22:17:30 -0500 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: Sorry, I'm getting messed up with the silly line breaks that gmail makes, and hit send several times by accident. On Tue, Feb 17, 2015 at 10:04 PM, wrote: > On Tue, Feb 17, 2015 at 10:04 PM, wrote: >> On Tue, Feb 17, 2015 at 10:02 PM, wrote: >>> On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein >>> wrote: >>>> >>>> This sounds practically like it's probably a really good test -- but just BTW, it's totally possible for the BFGS approximation to be either worse or better than the LBFGS approximation. The major cause of a difference in terms of accuracy is that the BFGS algorithms uses the full history of gradients and iterates, while LBFGS only uses the recent history of gradients and iterates (think it defaults to the last 10 steps for the scipy implementation). If the Hessian is constant, the BFGS approximation will be much better. If the Hessian changes significantly over the course of optimization, the BFGS approximation may actually turn out to be much worse. >>>> >>>> -jascha >>>> >>>> On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon wrote: >>>>> >>>>> I should check the accuracy of the BFGS approx inv. hessian using the (dense) `hess_inv` that is computed and returned already by `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the accuracy of the low-memory version computed by L-BFGS. >>>>> >>>>> -Robert >>>>> >>>>> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon wrote: >>>>>> >>>>>> > My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>>>> >>>>>> Yeah, that might be true. But the downstream approximations that go into calculating error bars from these >>>>>> hessians (that the likelihood is quadratic) can also be pretty bad approximations in many of the situations that >>>>>> they're commonly applied, so the whole enterprise is semi-quantitative at best. >>>>>> >>>>>> In my application, even with some tricks to calculate the Hessian faster, the optimizer can often converge to the >>>>>> MLE _much_ faster than I can calculate a single Hessian. So spending say, 1-10% of my computational effort >>>>>> finding the MLE and 90-99% calculating error bars seems silly when in the end I don't even have a lot of confidence >>>>>> in the approximations that go from hessian -> error bars, and I'm looking at them more from an order of magnitude >>>>>> perspective. >>> In statsmodels we use l-bfgs-b without bounds as default optimizer in some models, but we still calculate the hessian ourselves using our own numerical derivatives functions. I don't think our ratio for the timing is so bad, but the hessian calculation can be slow. In most cases we managed to get analytical hessian, especially in cases where the "inner" part is linear (x * beta). If there is more structure to the likelihood function, then it would still be possible that using chain rule, or using automatic differentiation is fast and more accurate. Another common approach in econometrics, but not yet used much in statsmodels, is to to the outer product of gradients (OPG), which requires memory but should be reasonably fast to calculate. OPG is inv(x'x) where x is the derivative of the loglikelihood with respect to the parameters for each observation. My guess is that adding the inverse hessian operator to lbfgsb is worth a try, but whether it should be included in scipy will depend on how difficult the implementation is and on how good the approximation is, IMO. Josef >>> >>>>>> >>>>>> >>>>>> As the the behavior for boundary solution, I'm honestly not sure. Obviously the true hessian isn't defined for certain >>>>>> (i,j) pairs, but for other directions that aren't at the boundary it should be okay. How this looks for the BFGS implicit >>>>>> inverse hessian I don't know. >>>>>> >>>>>> -Robert >>>>>> >>>>>> >>>>>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon wrote: >>>>>>>> >>>>>>>> Hey, >>>>>>>> >>>>>>>> tl;dir: thinking of writing a PR the following feature & looking for feedback first. >>>>>>>> >>>>>>>> In the L-BFGS-B minimizer, a low-memory approximation to the inverse hessian is >>>>>>>> used internally to determine the descent direction. Once the optimization terminates, >>>>>>>> none of this information is returned to the client though. I think it would be a nice to >>>>>>>> return a callable for the inverse hessian - vector product as a LinearOperator. >>>>>>>> >>>>>>>> The main use case for this that I have in mind is for computing standard errors in >>>>>>>> maximum likelihood statistical models. In this use case, you write down a likelihood >>>>>>>> function (and maybe its gradient), and pass it off to a minimizer to find the MLE. Getting >>>>>>>> standard errors in a scalar function of the parameters with the delta method, requires >>>>>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) is a vector (the >>>>>>>> gradient of the scalar function at the MLE) and the matrix H^{-1}(x) is the inverse of >>>>>>>> the Hessian at the MLE. >>>>>>>> >>>>>>>> For large-scale optimization of a likelihood with L-BFGS-B, the hessian is too big to >>>>>>>> calculate (or invert) explicitly, but a low memory approximation to the inverse is >>>>>>>> already available inside the optimizer, so I think it makes sense to expose it. The >>>>>>>> factored form of the BFGS matrix is kind of harry, so the most reasonable route is >>>>>>>> to return it as a LinearOperator that computes approximate inverse hessian - vector >>>>>>>> products. This calculation can be done pretty efficiently, I think, with the routine from >>>>>>>> section 4 of [1]. >>>>>>>> >>>>>>>> Does this make sense to people? Is it a desirable feature? I'm thinking of writing a >>>>>>>> PR for it, but I wanted to check here first before diving down the rabbit hole. >>>>>>>> >>>>>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited storage."Math. Comput. >>>>>>>> 35.151 (1980): 773-782. >>>>>>>> >>>>>>> >>>>>>> My guess, without knowing what l-bfgs-b is exactly doing, is that the approximation might not be very good. >>>>>>> (I remember there was the suggestion once to get the Lagrange multipliers out of one of the constraint optimizers, but when whoever proposed that saw how inaccurate they were, he gave up on the idea.) >>>>>>> >>>>>>> If it's the only way to get an approximation in large problems, then it might still be worth it. >>>>>>> >>>>>>> Another question: How is the Hessian calculated if there are binding constraints? >>>>>>> The Hessian won't give you the correct standard errors for a boundary solution. >>>>>>> >>>>>>> Josef >>>>>>> >>>>>>>> >>>>>>>> -Robert >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> SciPy-Dev mailing list >>>>>>>> SciPy-Dev at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> SciPy-Dev mailing list >>>>>>> SciPy-Dev at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> From rmcgibbo at gmail.com Tue Feb 17 22:59:48 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Tue, 17 Feb 2015 19:59:48 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: My medium-sized models have ~10^4 variables, so even analytic manipulations of the Hessian get very unwieldy, and easily take a lot longer than the optimization. -Robert On Tue, Feb 17, 2015 at 7:17 PM, wrote: > Sorry, I'm getting messed up with the silly line breaks that gmail > makes, and hit send several times by accident. > > On Tue, Feb 17, 2015 at 10:04 PM, wrote: > > On Tue, Feb 17, 2015 at 10:04 PM, wrote: > >> On Tue, Feb 17, 2015 at 10:02 PM, wrote: > >>> On Tue, Feb 17, 2015 at 9:54 PM, Jascha Sohl-Dickstein > >>> wrote: > >>>> > >>>> This sounds practically like it's probably a really good test -- but > just BTW, it's totally possible for the BFGS approximation to be either > worse or better than the LBFGS approximation. The major cause of a > difference in terms of accuracy is that the BFGS algorithms uses the full > history of gradients and iterates, while LBFGS only uses the recent history > of gradients and iterates (think it defaults to the last 10 steps for the > scipy implementation). If the Hessian is constant, the BFGS approximation > will be much better. If the Hessian changes significantly over the course > of optimization, the BFGS approximation may actually turn out to be much > worse. > >>>> > >>>> -jascha > >>>> > >>>> On Tue Feb 17 2015 at 6:27:17 PM Robert McGibbon > wrote: > >>>>> > >>>>> I should check the accuracy of the BFGS approx inv. hessian using > the (dense) `hess_inv` that is computed and returned already by > `fmin_bfgs`. The accuracy of that matrix is going to be no worse than the > accuracy of the low-memory version computed by L-BFGS. > >>>>> > >>>>> -Robert > >>>>> > >>>>> On Tue, Feb 17, 2015 at 6:20 PM, Robert McGibbon > wrote: > >>>>>> > >>>>>> > My guess, without knowing what l-bfgs-b is exactly doing, is that > the approximation might not be very good. > >>>>>> > >>>>>> Yeah, that might be true. But the downstream approximations that go > into calculating error bars from these > >>>>>> hessians (that the likelihood is quadratic) can also be pretty bad > approximations in many of the situations that > >>>>>> they're commonly applied, so the whole enterprise is > semi-quantitative at best. > >>>>>> > >>>>>> In my application, even with some tricks to calculate the Hessian > faster, the optimizer can often converge to the > >>>>>> MLE _much_ faster than I can calculate a single Hessian. So > spending say, 1-10% of my computational effort > >>>>>> finding the MLE and 90-99% calculating error bars seems silly when > in the end I don't even have a lot of confidence > >>>>>> in the approximations that go from hessian -> error bars, and I'm > looking at them more from an order of magnitude > >>>>>> perspective. > >>> > > In statsmodels we use l-bfgs-b without bounds as default optimizer in > some models, but we still calculate the hessian ourselves using our > own numerical derivatives functions. > I don't think our ratio for the timing is so bad, but the hessian > calculation can be slow. In most cases we managed to get analytical > hessian, especially in cases where the "inner" part is linear (x * > beta). > > If there is more structure to the likelihood function, then it would > still be possible that using chain rule, or using automatic > differentiation is fast and more accurate. > > Another common approach in econometrics, but not yet used much in > statsmodels, is to to the outer product of gradients (OPG), which > requires memory but should be reasonably fast to calculate. > OPG is inv(x'x) where x is the derivative of the loglikelihood with > respect to the parameters for each observation. > > My guess is that adding the inverse hessian operator to lbfgsb is > worth a try, but whether it should be included in scipy will depend on > how difficult the implementation is and on how good the approximation > is, IMO. > > Josef > > > >>> > >>>>>> > >>>>>> > >>>>>> As the the behavior for boundary solution, I'm honestly not sure. > Obviously the true hessian isn't defined for certain > >>>>>> (i,j) pairs, but for other directions that aren't at the boundary > it should be okay. How this looks for the BFGS implicit > >>>>>> inverse hessian I don't know. > >>>>>> > >>>>>> -Robert > >>>>>> > >>>>>> > >>>>>> On Tue, Feb 17, 2015 at 6:05 PM, wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Feb 17, 2015 at 8:18 PM, Robert McGibbon < > rmcgibbo at gmail.com> wrote: > >>>>>>>> > >>>>>>>> Hey, > >>>>>>>> > >>>>>>>> tl;dir: thinking of writing a PR the following feature & looking > for feedback first. > >>>>>>>> > >>>>>>>> In the L-BFGS-B minimizer, a low-memory approximation to the > inverse hessian is > >>>>>>>> used internally to determine the descent direction. Once the > optimization terminates, > >>>>>>>> none of this information is returned to the client though. I > think it would be a nice to > >>>>>>>> return a callable for the inverse hessian - vector product as a > LinearOperator. > >>>>>>>> > >>>>>>>> The main use case for this that I have in mind is for computing > standard errors in > >>>>>>>> maximum likelihood statistical models. In this use case, you > write down a likelihood > >>>>>>>> function (and maybe its gradient), and pass it off to a minimizer > to find the MLE. Getting > >>>>>>>> standard errors in a scalar function of the parameters with the > delta method, requires > >>>>>>>> computing a quadratic form like g(x)' H^{-1}(x) g(x), where g(x) > is a vector (the > >>>>>>>> gradient of the scalar function at the MLE) and the matrix > H^{-1}(x) is the inverse of > >>>>>>>> the Hessian at the MLE. > >>>>>>>> > >>>>>>>> For large-scale optimization of a likelihood with L-BFGS-B, the > hessian is too big to > >>>>>>>> calculate (or invert) explicitly, but a low memory approximation > to the inverse is > >>>>>>>> already available inside the optimizer, so I think it makes sense > to expose it. The > >>>>>>>> factored form of the BFGS matrix is kind of harry, so the most > reasonable route is > >>>>>>>> to return it as a LinearOperator that computes approximate > inverse hessian - vector > >>>>>>>> products. This calculation can be done pretty efficiently, I > think, with the routine from > >>>>>>>> section 4 of [1]. > >>>>>>>> > >>>>>>>> Does this make sense to people? Is it a desirable feature? I'm > thinking of writing a > >>>>>>>> PR for it, but I wanted to check here first before diving down > the rabbit hole. > >>>>>>>> > >>>>>>>> [1] Nocedal, J. "Updating quasi-Newton matrices with limited > storage."Math. Comput. > >>>>>>>> 35.151 (1980): 773-782. > >>>>>>>> > >>>>>>> > >>>>>>> My guess, without knowing what l-bfgs-b is exactly doing, is that > the approximation might not be very good. > >>>>>>> (I remember there was the suggestion once to get the Lagrange > multipliers out of one of the constraint optimizers, but when whoever > proposed that saw how inaccurate they were, he gave up on the idea.) > >>>>>>> > >>>>>>> If it's the only way to get an approximation in large problems, > then it might still be worth it. > >>>>>>> > >>>>>>> Another question: How is the Hessian calculated if there are > binding constraints? > >>>>>>> The Hessian won't give you the correct standard errors for a > boundary solution. > >>>>>>> > >>>>>>> Josef > >>>>>>> > >>>>>>>> > >>>>>>>> -Robert > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> SciPy-Dev mailing list > >>>>>>>> SciPy-Dev at scipy.org > >>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> SciPy-Dev mailing list > >>>>>>> SciPy-Dev at scipy.org > >>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>>>>> > >>>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> SciPy-Dev mailing list > >>>>> SciPy-Dev at scipy.org > >>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>> > >>>> > >>>> _______________________________________________ > >>>> SciPy-Dev mailing list > >>>> SciPy-Dev at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev > >>>> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Feb 18 12:26:02 2015 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 18 Feb 2015 19:26:02 +0200 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: Hi, 18.02.2015, 03:18, Robert McGibbon kirjoitti: [clip] > In the L-BFGS-B minimizer, a low-memory approximation to the inverse > hessian is > used internally to determine the descent direction. Once the optimization > terminates, > none of this information is returned to the client though. I think it would > be a nice to > return a callable for the inverse hessian - vector product as a > LinearOperator. Depending on what the implicit storage format in lbfgsb is, the Python-side may already be implemented in scipy.optimize.nonlin.LowRankMatrix The maintenance problem here is that extracting the information requires mucking with undocumented implementation details of the algorithm, which may change the author releases yet another new version. However, the software is fairly mature, so with proper tests adding this could be possible. A completely separate question then is whether the approximation is useful for your purpose, and whether you could do better just by constructing the BFGS secant approximation yourself by recording the gradients evaluated during your optimization process. From raniere at ime.unicamp.br Thu Feb 19 07:04:09 2015 From: raniere at ime.unicamp.br (Raniere Silva) Date: Thu, 19 Feb 2015 10:04:09 -0200 Subject: [SciPy-Dev] Google Summer of Code and NumFOCUS Message-ID: <20150219120408.GB16143@pupunha> Hi, NumFOCUS has promotes and supports the ongoing research and development of open-source computing tools including SciPy. This year NumFOCUS want to try be a Google Summer of Code "umbrella" mentoring organization, Umbrella organizations are mentoring organizations accepted into the Google Summer of Code program that have other open source organizations working "under" them. Sometime organizations that work very closely or have very similar goals or communities may get put together under an "umbrella." Google stills expects all organizations under the umbrella, whether accepted into the program under their title or not, to adhere to all the rules and regulations of the program. From https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2015/help_page#umbrella_organization To help promote and support SciPy. We encourage SciPy to apply to Google Summer of Code under your own title and will be very happy if you can also do with us. If you are interested, please check https://github.com/swcarpentry/gsoc2015 and https://github.com/swcarpentry/gsoc2015/blob/master/CONTRIBUTING.md. If you have any question, please email me directly. Thanks in advance, Raniere -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From stefan.peterson at rubico.com Thu Feb 19 06:55:26 2015 From: stefan.peterson at rubico.com (stefan) Date: Thu, 19 Feb 2015 12:55:26 +0100 Subject: [SciPy-Dev] Slow moment function in scipy.stats Message-ID: <54E5CF2E.3040209@rubico.com> Hello list, First time poster here. Anyway, some time ago I noticed that the scipy skewness function was a major bottleneck in an algorithm of mine. Back then, I typed up my own replacement and thought no more about it. Today, for some unknown reason, I decided to dig a little deeper in this and found the major culprit to be the way moments are computed, specifically the use of np.power. Testing an alternative approach (missing some safeties, obviously) in IPython: In [1]: import numpy as np In [2]: import scipy.stats as spstat In [3]: def moment(x, mom=1, axis=0): ...: if mom == 1: ...: return np.float64(0.0) ...: else: ...: x_zero_mean = x - np.expand_dims(np.mean(x, axis), axis) ...: x_zero_mean_2 = x_zero_mean**2 ...: s = x_zero_mean_2.copy() ...: for k in range(1, mom // 2): ...: s *= x_zero_mean_2 ...: if mom % 2: ...: s *= x_zero_mean ...: return np.mean(s, axis) ...: In [4]: a = np.random.randn(25,1e5) In [5]: for ax in range(2): ...: for k in range(1, 8): ...: %timeit spstat.moment(a, k, ax) ...: 10000 loops, best of 3: 41.9 ?s per loop 10 loops, best of 3: 44 ms per loop 1 loops, best of 3: 233 ms per loop 1 loops, best of 3: 229 ms per loop 1 loops, best of 3: 232 ms per loop 1 loops, best of 3: 232 ms per loop 1 loops, best of 3: 236 ms per loop 100000 loops, best of 3: 4.14 ?s per loop 10 loops, best of 3: 43.3 ms per loop 1 loops, best of 3: 227 ms per loop 1 loops, best of 3: 225 ms per loop 1 loops, best of 3: 227 ms per loop 1 loops, best of 3: 232 ms per loop 1 loops, best of 3: 232 ms per loop In [6]: for ax in range(2): for k in range(1, 8): %timeit moment(a, k, ax) ...: 1000000 loops, best of 3: 458 ns per loop 10 loops, best of 3: 21.7 ms per loop 10 loops, best of 3: 26 ms per loop 10 loops, best of 3: 25.9 ms per loop 10 loops, best of 3: 30.4 ms per loop 10 loops, best of 3: 30.3 ms per loop 10 loops, best of 3: 33.1 ms per loop 1000000 loops, best of 3: 463 ns per loop 10 loops, best of 3: 21.2 ms per loop 10 loops, best of 3: 25.5 ms per loop 10 loops, best of 3: 25.5 ms per loop 10 loops, best of 3: 30.1 ms per loop 10 loops, best of 3: 30.1 ms per loop 10 loops, best of 3: 33.2 ms per loop In [7]: for ax in range(2): for k in range(1, 8): print(np.sum((spstat.moment(a, k, ax) - moment(a, k, ax))**2)) ...: 0.0 0.0 6.87146461986e-28 1.49841810127e-26 6.84527222501e-26 1.26529038747e-24 1.35165136907e-23 0.0 0.0 1.34532977463e-33 3.94430452611e-30 1.5467173476e-31 1.95637504495e-28 2.48413854543e-29 So there are some rounding errors, but they're hardly alarming. Is there another reason not to do it this way? Best Regards, Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Feb 20 04:05:50 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 20 Feb 2015 10:05:50 +0100 Subject: [SciPy-Dev] GSoC'15 prep - ideas page Message-ID: Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Feb 20 10:40:46 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 20 Feb 2015 16:40:46 +0100 Subject: [SciPy-Dev] GSoC 2015 In-Reply-To: References: Message-ID: On Sun, Feb 15, 2015 at 9:30 PM, Rohan Goel wrote: > Dear Developers, > > I am Rohan Goel , Computer Science undergraduate at BITS Pilani , India > and am interested in working for your organization in GSoC 2015. As I am a > beginner in open source coding , it would be great help if you guide me > where to start from. My primary language of interest is Python. > Hi Rohan, nice to see your interest in working on Scipy for this year's GSoC. Here are some guidelines and ideas to get you started: https://github.com/scipy/scipy/wiki/GSoC-project-ideas If you have specific questions or ideas, please don't hesitate to ask on this list. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Feb 20 10:53:05 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 20 Feb 2015 16:53:05 +0100 Subject: [SciPy-Dev] [Numpy-discussion] Google Summer of Code and NumFOCUS In-Reply-To: <20150219120315.GA16143@pupunha> References: <20150219120315.GA16143@pupunha> Message-ID: Hi Raniere, On Thu, Feb 19, 2015 at 1:03 PM, Raniere Silva wrote: > Hi, > > NumFOCUS has promotes and supports the ongoing research and development of > open-source computing tools including NumPy. > > This year NumFOCUS want to try be a Google Summer of Code > "umbrella" mentoring organization, > > Umbrella organizations are mentoring organizations accepted into the > Google > Summer of Code program that have other open source organizations > working > "under" them. Sometime organizations that work very closely or have > very > similar goals or communities may get put together under an "umbrella." > Google stills expects all organizations under the umbrella, whether > accepted > into the program under their title or not, to adhere to all the rules > and > regulations of the program. > > From > https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2015/help_page#umbrella_organization > > To help promote and support NumPy. > > We encourage NumPy to apply to Google Summer of Code under your own title > and will be very happy if you can also do with us. > If you are interested, please check > https://github.com/swcarpentry/gsoc2015 > and https://github.com/swcarpentry/gsoc2015/blob/master/CONTRIBUTING.md. > > If you have any question, please email me directly. > Thanks for the enthusiasm in getting Numfocus set up as a new mentoring org - I'm sure that this will be useful for the long term. I'm not sure if you're aware that Numpy, Scipy and many other scientific Python projects have been participating under the umbrella of the PSF for a number of years already (see https://wiki.python.org/moin/SummerOfCode/2014). For this year we'd like to keep it that way. I'll email you privately with more details and to see if I can help you in any way. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From raniere at ime.unicamp.br Fri Feb 20 11:19:00 2015 From: raniere at ime.unicamp.br (Raniere Silva) Date: Fri, 20 Feb 2015 14:19:00 -0200 Subject: [SciPy-Dev] [Numpy-discussion] Google Summer of Code and NumFOCUS In-Reply-To: References: <20150219120315.GA16143@pupunha> Message-ID: <20150220161900.GF12853@pupunha> Hi Ralf, > Thanks for the enthusiasm in getting Numfocus set up as a new mentoring org > - I'm sure that this will be useful for the long term. Thanks for your email. > I'm not sure if > you're aware that Numpy, Scipy and many other scientific Python projects > have been participating under the umbrella of the PSF for a number of years > already (see https://wiki.python.org/moin/SummerOfCode/2014). Yes, I'm aware of it. I only care that every project have the opportunity to participate at GSoC and participants have fun during it. =) > For this year we'd like to keep it that way. No problem for me. I hope that we can work together on this next year. Raniere -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From maniteja.modesty067 at gmail.com Sat Feb 21 14:52:03 2015 From: maniteja.modesty067 at gmail.com (Maniteja Nandana) Date: Sun, 22 Feb 2015 01:22:03 +0530 Subject: [SciPy-Dev] Regarding taking up project ideas and GSoC 2015 In-Reply-To: References: Message-ID: Hello everyone, Sorry for pinging the mailing list again, but wanted to seek guidance and advice on working on these : 1. levenberg-marquardt nonlinear least squares algorithm, which was worked in PR #90 . I have done some reading on Levenberg-Marquardt algorithm, though need to know suitable implementations. 2. cubic regular grid interpolation schemes on lines with #3233 . I have some basic knowledge of spline math, but need more insight into implementation issues about uniform and non-uniform spaced grids. I am interested to work on these topics, since they are related to my field of study. It would also be great to work on other topics like numerical differentiation, but as of now I have very less practical exposure to these fields and am keen to know more about them. Looking forward for your opinions and suggestions. Cheers, N.Maniteja _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Mon Feb 23 11:48:32 2015 From: larson.eric.d at gmail.com (Eric Larson) Date: Mon, 23 Feb 2015 08:48:32 -0800 Subject: [SciPy-Dev] Polyphase filtering and resampling Message-ID: Hey, Scipy's resampling algorithm currently makes use of a frequency-domain technique. I'm interested in potentially adding a time-domain polyphase resampling method. This is the same technique used by MATLAB. It requires implementation of an upfirdn-type function that upsamples, applies a FIR filter, then downsamples efficiently. There is already a SWIG-compiled BSD-licensed Python version available: https://code.google.com/p/upfirdn/ I have gotten permission from the author to port the code to scipy. Would upfirdn and the associated resampling method be useful to move into scipy? Cheers, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Feb 23 12:45:51 2015 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 23 Feb 2015 12:45:51 -0500 Subject: [SciPy-Dev] Polyphase filtering and resampling In-Reply-To: References: Message-ID: On Mon, Feb 23, 2015 at 11:48 AM, Eric Larson wrote: > Hey, > > Scipy's resampling algorithm currently makes use of a frequency-domain > technique. I'm interested in potentially adding a time-domain polyphase > resampling method. This is the same technique used by MATLAB. It requires > implementation of an upfirdn-type function that upsamples, applies a FIR > filter, then downsamples efficiently. There is already a SWIG-compiled > BSD-licensed Python version available: > > https://code.google.com/p/upfirdn/ > > I have gotten permission from the author to port the code to scipy. Would > upfirdn and the associated resampling method be useful to move into scipy? > > Yes, an "upfirdn" resampler would be great! Warren > Cheers, > Eric > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karandev43 at gmail.com Mon Feb 23 22:04:22 2015 From: karandev43 at gmail.com (Karan Dev) Date: Tue, 24 Feb 2015 08:34:22 +0530 Subject: [SciPy-Dev] Introduction: gsoc2015 Message-ID: Hi, I'm final year B.Tech. Computer Science student. I want to participate in gsoc2015 by contributing to the organization. I am comfortable in Python, C/C++, JavaScript. I was wondering if anyone could suggest some bugs that are suitable for a beginner in this project? Thank You. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Feb 24 00:47:19 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Feb 2015 06:47:19 +0100 Subject: [SciPy-Dev] scipy.signal function naming Message-ID: Hi, Historically many names in scipy.signal have followed Matlab, which typically chooses short but very nondescriptive names. I would prefer to not keep doing that but instead choose readable names that fit with the rest of Scipy and Python. Recent examples from PRs, with in brackets a proposed alternative: grpdelay (group_delay) - https://github.com/scipy/scipy/pull/4549 place (place_poles) - https://github.com/scipy/scipy/pull/4295 ctrb (controllalbility_matrix) - https://github.com/scipy/scipy/pull/4515 obsv (observability_matrix) - https://github.com/scipy/scipy/pull/4515 upfirdn (rate_resample ?) - just discussed on list Thoughts? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Feb 24 00:56:51 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Feb 2015 06:56:51 +0100 Subject: [SciPy-Dev] Introduction: gsoc2015 In-Reply-To: References: Message-ID: Hi Karan, On Tue, Feb 24, 2015 at 4:04 AM, Karan Dev wrote: > Hi, > > I'm final year B.Tech. Computer Science student. I want to participate in > gsoc2015 by contributing to the organization. I am comfortable in Python, > C/C++, JavaScript. I was wondering if anyone could suggest some bugs that > are suitable for a beginner in this project? > Welcome! There are some issues labeled "easy-fix" on https://github.com/scipy/scipy/issues that you could have a look at. Make sure to check that there's not already an open PR addressing the issue though (if there is, that will be visible in the issue). Alternatively, tell us if you're interested in a specific GSoC idea or a part of Scipy and we'll point you to some issues related to your interest. Cheers, Ralf > Thank You. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Feb 24 01:38:34 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Feb 2015 07:38:34 +0100 Subject: [SciPy-Dev] Regarding taking up project ideas and GSoC 2015 In-Reply-To: References: Message-ID: Hi Maniteja, On Sat, Feb 21, 2015 at 8:52 PM, Maniteja Nandana < maniteja.modesty067 at gmail.com> wrote: > Hello everyone, > > Sorry for pinging the mailing list again, but wanted to seek guidance and > advice on working on these : > > 1. levenberg-marquardt nonlinear least squares algorithm, which was > worked in PR #90 . I have done > some reading > on > Levenberg-Marquardt algorithm, though need to know suitable > implementations. > 2. cubic regular grid interpolation schemes on lines with #3233 > . I have some basic > knowledge of spline math, but need more insight into implementation issues > about uniform and non-uniform spaced grids. > > I am interested to work on these topics, since they are related to my > field of study. It would also be great to work on other topics like > numerical differentiation, but as of now I have very less practical > exposure to these fields and am keen to know more about them. > > Looking forward for your opinions and suggestions. > In general you are on the right track here - starting early and identifying the project ideas that interest you. I'm hoping that the people who came up with and potentially can mentor them (Pauli and Evgeni respectively in this case) can provide you with a few more details. I know you're already looking at related issues in the Scipy issue tracker, which is great. My only advice there would be to try to pick issues that are clearcut - if it requires back-and-forth discussion then it often takes a while to get anywhere due to the limited amount of developer resources that we have. Cheers, Ralf P.S. I'll start bothering potential mentors for all ideas offline for their avaialbility this year. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Feb 24 01:49:41 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Feb 2015 07:49:41 +0100 Subject: [SciPy-Dev] GSoC'15 - mentors & ideas Message-ID: Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers wrote: > Hi all, > > It's time to start preparing for this year's Google Summer of Code. There > is actually one urgent thing to be done (before 19.00 UTC today), which is > to get our ideas page in decent shape. It doesn't have to be final, but > there has to be enough on there for the organizers to judge it. This page > is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be > reworking it and linking it from the PSF page today, but if you already > have new ideas please add them there. See > https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this > year's template for adding a new idea. > The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. What we also need is mentors. All ideas already have a potential mentor listed, however some ideas are from last year and I'm not sure that all those mentors really are available this year. And more than one potential mentor per idea is always good. So can everyone please add/remove his or her name on that page? I'm happy to take care of most of the organizational aspects this year, however I'll be offline for two weeks in July and from the end of August onwards, so I'll some help in those periods. Any volunteers? Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Feb 24 02:17:48 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Feb 2015 08:17:48 +0100 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: <54E5CF2E.3040209@rubico.com> References: <54E5CF2E.3040209@rubico.com> Message-ID: Hi Stefan, On Thu, Feb 19, 2015 at 12:55 PM, stefan wrote: > Hello list, > > First time poster here. Anyway, some time ago I noticed that the scipy > skewness function was a major bottleneck in an algorithm of mine. Back > then, I typed up my own replacement and thought no more about it. Today, > for some unknown reason, I decided to dig a little deeper in this and found > the major culprit to be the way moments are computed, specifically the use > of np.power. > np.power is indeed slow, see for explanations: http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents > > Testing an alternative approach (missing some safeties, obviously) in > IPython: > > In [1]: import numpy as np > > In [2]: import scipy.stats as spstat > > In [3]: def moment(x, mom=1, axis=0): > ...: if mom == 1: > ...: return np.float64(0.0) > ...: else: > ...: x_zero_mean = x - np.expand_dims(np.mean(x, axis), axis) > ...: x_zero_mean_2 = x_zero_mean**2 > ...: s = x_zero_mean_2.copy() > ...: for k in range(1, mom // 2): > ...: s *= x_zero_mean_2 > ...: if mom % 2: > ...: s *= x_zero_mean > ...: return np.mean(s, axis) > ...: > > In [4]: a = np.random.randn(25,1e5) > > In [5]: for ax in range(2): > ...: for k in range(1, 8): > ...: %timeit spstat.moment(a, k, ax) > ...: > 10000 loops, best of 3: 41.9 ?s per loop > 10 loops, best of 3: 44 ms per loop > 1 loops, best of 3: 233 ms per loop > 1 loops, best of 3: 229 ms per loop > 1 loops, best of 3: 232 ms per loop > 1 loops, best of 3: 232 ms per loop > 1 loops, best of 3: 236 ms per loop > 100000 loops, best of 3: 4.14 ?s per loop > 10 loops, best of 3: 43.3 ms per loop > 1 loops, best of 3: 227 ms per loop > 1 loops, best of 3: 225 ms per loop > 1 loops, best of 3: 227 ms per loop > 1 loops, best of 3: 232 ms per loop > 1 loops, best of 3: 232 ms per loop > > In [6]: for ax in range(2): > for k in range(1, 8): > %timeit moment(a, k, ax) > ...: > 1000000 loops, best of 3: 458 ns per loop > 10 loops, best of 3: 21.7 ms per loop > 10 loops, best of 3: 26 ms per loop > 10 loops, best of 3: 25.9 ms per loop > 10 loops, best of 3: 30.4 ms per loop > 10 loops, best of 3: 30.3 ms per loop > 10 loops, best of 3: 33.1 ms per loop > 1000000 loops, best of 3: 463 ns per loop > 10 loops, best of 3: 21.2 ms per loop > 10 loops, best of 3: 25.5 ms per loop > 10 loops, best of 3: 25.5 ms per loop > 10 loops, best of 3: 30.1 ms per loop > 10 loops, best of 3: 30.1 ms per loop > 10 loops, best of 3: 33.2 ms per loop > > In [7]: for ax in range(2): > for k in range(1, 8): > print(np.sum((spstat.moment(a, k, ax) - moment(a, k, ax))**2)) > ...: > 0.0 > 0.0 > 6.87146461986e-28 > 1.49841810127e-26 > 6.84527222501e-26 > 1.26529038747e-24 > 1.35165136907e-23 > 0.0 > 0.0 > 1.34532977463e-33 > 3.94430452611e-30 > 1.5467173476e-31 > 1.95637504495e-28 > 2.48413854543e-29 > > So there are some rounding errors, but they're hardly alarming. Is there > another reason not to do it this way? > I'd say that replacing one call to np.power with 6 lines of code to achieve a ~10x speedup is a good tradeoff. Pull request is welcome:) Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Feb 24 04:57:46 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 24 Feb 2015 10:57:46 +0100 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: References: <54E5CF2E.3040209@rubico.com> Message-ID: <54EC4B1A.7040306@googlemail.com> On 02/24/2015 08:17 AM, Ralf Gommers wrote: > Hi Stefan, > > > On Thu, Feb 19, 2015 at 12:55 PM, stefan > wrote: > > Hello list, > > First time poster here. Anyway, some time ago I noticed that the > scipy skewness function was a major bottleneck in an algorithm of > mine. Back then, I typed up my own replacement and thought no more > about it. Today, for some unknown reason, I decided to dig a little > deeper in this and found the major culprit to be the way moments are > computed, specifically the use of np.power. > > > np.power is indeed slow, see for explanations: > http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining > http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents > > > I'd say that replacing one call to np.power with 6 lines of code to > achieve a ~10x speedup is a good tradeoff. Pull request is welcome:) > its faster but also less accurate, the reason pow is so slow is that it has an accuracy of 0.5 ulp regardless of input, which a multiplication does not. But the argument can certainly be made that this is irrelevant especially as the moment computation performed here is numerically not that stable in the first place (uncorrected loss of significance in the subtraction, mean has an error in the order of the array size ...) maybe numpy should have a special integer case in power for such situations? From maniteja.modesty067 at gmail.com Tue Feb 24 06:35:57 2015 From: maniteja.modesty067 at gmail.com (Maniteja Nandana) Date: Tue, 24 Feb 2015 17:05:57 +0530 Subject: [SciPy-Dev] Regarding taking up project ideas and GSoC 2015 In-Reply-To: References: Message-ID: Hi Ralf, On Tue, Feb 24, 2015 at 12:08 PM, Ralf Gommers wrote: > Hi Maniteja, > > In general you are on the right track here - starting early and > identifying the project ideas that interest you. I'm hoping that the people > who came up with and potentially can mentor them (Pauli and Evgeni > respectively in this case) can provide you with a few more details. > > Thanks for the reply, in meantime I am looking at the relevant background knowledge required to have a better understanding of the problems I know you're already looking at related issues in the Scipy issue tracker, > which is great. My only advice there would be to try to pick issues that > are clearcut - if it requires back-and-forth discussion then it often takes > a while to get anywhere due to the limited amount of developer resources > that we have. > I understand the situation and am trying to get a clear demarcation regarding the feasibility of my experience, with respect to the ideas and implementation. I will also try to look around for interesting problems, probably in the roadmap and put up a mail regarding the same. > Cheers, > Ralf > > P.S. I'll start bothering potential mentors for all ideas offline for > their avaialbility this year. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > Waiting in anticipation for suggestions. Always open to hear about new ideas. Happy to learn :) Cheers, Maniteja _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.peterson at rubico.com Tue Feb 24 06:42:36 2015 From: stefan.peterson at rubico.com (stefan) Date: Tue, 24 Feb 2015 12:42:36 +0100 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: References: Message-ID: <54EC63AC.1080406@rubico.com> Hi Ralf and Julian I have created a pull request (I hope, very limited experience with git and github). After doing so, I saw Julian's post. Indeed, pow is very accurate. I would argue that for all realistic uses of the moment function, the inaccuracies will be insignificant, but this is certainly up for discussion. BR, Stefan On 02/24/2015 08:17 AM, Ralf Gommers wrote: >/ Hi Stefan, />/ />/ />/ On Thu, Feb 19, 2015 at 12:55 PM, stefan />/ >> wrote: />/ />/ Hello list, />/ />/ First time poster here. Anyway, some time ago I noticed that the />/ scipy skewness function was a major bottleneck in an algorithm of />/ mine. Back then, I typed up my own replacement and thought no more />/ about it. Today, for some unknown reason, I decided to dig a little />/ deeper in this and found the major culprit to be the way moments are />/ computed, specifically the use of np.power. />/ />/ />/ np.power is indeed slow, see for explanations: />/ http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining />/ http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents />/ />/ />/ I'd say that replacing one call to np.power with 6 lines of code to />/ achieve a ~10x speedup is a good tradeoff. Pull request is welcome:) />/ / its faster but also less accurate, the reason pow is so slow is that it has an accuracy of 0.5 ulp regardless of input, which a multiplication does not. But the argument can certainly be made that this is irrelevant especially as the moment computation performed here is numerically not that stable in the first place (uncorrected loss of significance in the subtraction, mean has an error in the order of the array size ...) maybe numpy should have a special integer case in power for such situations? -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Tue Feb 24 09:29:58 2015 From: larson.eric.d at gmail.com (Eric Larson) Date: Tue, 24 Feb 2015 06:29:58 -0800 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: References: Message-ID: +1 for more readable names, and putting the MATLAB-equivalent names in the docstring. Usually when I'm looking for a MATLAB-equivalent function, I'll do something like google "python upfirdn" or "scipy upfirdn", so if "upfirdn" were in the docstring of "rate_resample" it would be also be found. Eric On Mon, Feb 23, 2015 at 9:47 PM, Ralf Gommers wrote: > Hi, > > Historically many names in scipy.signal have followed Matlab, which > typically chooses short but very nondescriptive names. I would prefer to > not keep doing that but instead choose readable names that fit with the > rest of Scipy and Python. Recent examples from PRs, with in brackets a > proposed alternative: > > grpdelay (group_delay) - https://github.com/scipy/scipy/pull/4549 > place (place_poles) - https://github.com/scipy/scipy/pull/4295 > ctrb (controllalbility_matrix) - https://github.com/scipy/scipy/pull/4515 > obsv (observability_matrix) - https://github.com/scipy/scipy/pull/4515 > upfirdn (rate_resample ?) - just discussed on list > > Thoughts? > > Ralf > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewm at redtetrahedron.org Tue Feb 24 09:40:19 2015 From: ewm at redtetrahedron.org (Eric Moore) Date: Tue, 24 Feb 2015 09:40:19 -0500 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: References: Message-ID: Yes. Absolutely +1. On Tue, Feb 24, 2015 at 12:47 AM, Ralf Gommers wrote: > Hi, > > Historically many names in scipy.signal have followed Matlab, which > typically chooses short but very nondescriptive names. I would prefer to > not keep doing that but instead choose readable names that fit with the > rest of Scipy and Python. Recent examples from PRs, with in brackets a > proposed alternative: > > grpdelay (group_delay) - https://github.com/scipy/scipy/pull/4549 > place (place_poles) - https://github.com/scipy/scipy/pull/4295 > ctrb (controllalbility_matrix) - https://github.com/scipy/scipy/pull/4515 > obsv (observability_matrix) - https://github.com/scipy/scipy/pull/4515 > upfirdn (rate_resample ?) - just discussed on list > > Thoughts? > > Ralf > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karandev43 at gmail.com Tue Feb 24 13:44:45 2015 From: karandev43 at gmail.com (Karan Dev) Date: Wed, 25 Feb 2015 00:14:45 +0530 Subject: [SciPy-Dev] Introduction: gsoc2015 In-Reply-To: References: Message-ID: I went through the GSoC 2015 project ideas and found https://github.com/sympy/sympy/wiki/GSoC-2015-Ideas#group-theory interesting. Please guide me for the next step. Also please provide me the code structure of Sympy to understand the code base. I am a newbie. On Tue, Feb 24, 2015 at 11:26 AM, Ralf Gommers wrote: > Hi Karan, > > > On Tue, Feb 24, 2015 at 4:04 AM, Karan Dev wrote: > >> Hi, >> >> I'm final year B.Tech. Computer Science student. I want to participate >> in gsoc2015 by contributing to the organization. I am comfortable in >> Python, C/C++, JavaScript. I was wondering if anyone could suggest some >> bugs that are suitable for a beginner in this project? >> > > Welcome! There are some issues labeled "easy-fix" on > https://github.com/scipy/scipy/issues that you could have a look at. Make > sure to check that there's not already an open PR addressing the issue > though (if there is, that will be visible in the issue). Alternatively, > tell us if you're interested in a specific GSoC idea or a part of Scipy and > we'll point you to some issues related to your interest. > > Cheers, > Ralf > > >> Thank You. >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucente at pipeline.com Tue Feb 24 14:11:19 2015 From: rlucente at pipeline.com (Robert Lucente) Date: Tue, 24 Feb 2015 14:11:19 -0500 (GMT-05:00) Subject: [SciPy-Dev] Slow moment function in scipy.stats Message-ID: <13734815.1424805079902.JavaMail.root@mswamui-cedar.atl.sa.earthlink.net> Why must it be an either or? Why not have 2 options specified via a parameter 1) Fast but inaccurate 2) Slow but more accurate From njs at pobox.com Tue Feb 24 14:31:56 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 24 Feb 2015 11:31:56 -0800 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: <13734815.1424805079902.JavaMail.root@mswamui-cedar.atl.sa.earthlink.net> References: <13734815.1424805079902.JavaMail.root@mswamui-cedar.atl.sa.earthlink.net> Message-ID: Options are really expensive: they double testing load, they complicate docs, they increase cognitive load for users, they constrain future api improvements, etc. They should only be provided when they provide a meaningful enough advantage to outweigh these costs. On Feb 24, 2015 11:11 AM, "Robert Lucente" wrote: > Why must it be an either or? Why not have 2 options specified via a > parameter > > 1) Fast but inaccurate > 2) Slow but more accurate > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Feb 24 15:21:41 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 24 Feb 2015 12:21:41 -0800 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: References: Message-ID: On Feb 23, 2015 9:47 PM, "Ralf Gommers" wrote: > > Hi, > > Historically many names in scipy.signal have followed Matlab, which typically chooses short but very nondescriptive names. I would prefer to not keep doing that but instead choose readable names that fit with the rest of Scipy and Python. Recent examples from PRs, with in brackets a proposed alternative: > > grpdelay (group_delay) - https://github.com/scipy/scipy/pull/4549 > place (place_poles) - https://github.com/scipy/scipy/pull/4295 > ctrb (controllalbility_matrix) - https://github.com/scipy/scipy/pull/4515 > obsv (observability_matrix) - https://github.com/scipy/scipy/pull/4515 > upfirdn (rate_resample ?) - just discussed on list +1 from me. Should anything be done about the existing functions that use the legacy naming scheme? Removing the old names would clearly be obnoxious, but as a gentler approach it might make sense to give them new Python-style names and document the old ones as being dispreferred legacy aliases (that nonetheless will remain supported indefinitely). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Feb 24 15:33:51 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Feb 2015 15:33:51 -0500 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: References: <13734815.1424805079902.JavaMail.root@mswamui-cedar.atl.sa.earthlink.net> Message-ID: On Tue, Feb 24, 2015 at 2:31 PM, Nathaniel Smith wrote: > Options are really expensive: they double testing load, they complicate > docs, they increase cognitive load for users, they constrain future api > improvements, etc. They should only be provided when they provide a > meaningful enough advantage to outweigh these costs. > > On Feb 24, 2015 11:11 AM, "Robert Lucente" wrote: >> >> Why must it be an either or? Why not have 2 options specified via a >> parameter >> >> 1) Fast but inaccurate >> 2) Slow but more accurate as Julian pointed out the power calculation is only a smaller source of numerical noise. The main precision problems comes from the sums. IIRC, there was a brief discussion to get additionally a better variance calculation to match Python stats package. Josef >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From rlucente at pipeline.com Tue Feb 24 16:00:46 2015 From: rlucente at pipeline.com (Robert Lucente) Date: Tue, 24 Feb 2015 16:00:46 -0500 (GMT-05:00) Subject: [SciPy-Dev] Slow moment function in scipy.stats Message-ID: <29485899.1424811647310.JavaMail.root@mswamui-cedar.atl.sa.earthlink.net> An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Feb 25 02:13:37 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Feb 2015 08:13:37 +0100 Subject: [SciPy-Dev] Slow moment function in scipy.stats In-Reply-To: <54EC63AC.1080406@rubico.com> References: <54EC63AC.1080406@rubico.com> Message-ID: On Tue, Feb 24, 2015 at 12:42 PM, stefan wrote: > Hi Ralf and Julian > > I have created a pull request (I hope, very limited experience with git and github). After doing so, I saw Julian's post. Indeed, pow is very accurate. I would argue that for all realistic uses of the moment function, the inaccuracies will be insignificant, but this is certainly up for discussion. > > I would agree - if you have to care about precision of some ulps for statistical functions like these then you are very likely using the wrong approach. BR, > Stefan > > On 02/24/2015 08:17 AM, Ralf Gommers wrote: > >* Hi Stefan, > * > > > >* On Thu, Feb 19, 2015 at 12:55 PM, stefan > *>* >> wrote: > *> >* Hello list, > *> >* First time poster here. Anyway, some time ago I noticed that the > *>* scipy skewness function was a major bottleneck in an algorithm of > *>* mine. Back then, I typed up my own replacement and thought no more > *>* about it. Today, for some unknown reason, I decided to dig a little > *>* deeper in this and found the major culprit to be the way moments are > *>* computed, specifically the use of np.power. > *> > >* np.power is indeed slow, see for explanations: > *>* http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining > *>* http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents > *> > >* I'd say that replacing one call to np.power with 6 lines of code to > *>* achieve a ~10x speedup is a good tradeoff. Pull request is welcome:) > *> > its faster but also less accurate, the reason pow is so slow is that it > has an accuracy of 0.5 ulp regardless of input, which a multiplication > does not. > But the argument can certainly be made that this is irrelevant > especially as the moment computation performed here is numerically not > that stable in the first place (uncorrected loss of significance in the > subtraction, mean has an error in the order of the array size ...) > > maybe numpy should have a special integer case in power for such situations? > > Probably not very high prio, but it would be nice to have that. Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Feb 25 02:26:56 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Feb 2015 08:26:56 +0100 Subject: [SciPy-Dev] Introduction: gsoc2015 In-Reply-To: References: Message-ID: On Tue, Feb 24, 2015 at 7:44 PM, Karan Dev wrote: > I went through the GSoC 2015 project ideas and found > https://github.com/sympy/sympy/wiki/GSoC-2015-Ideas#group-theory > interesting. Please guide me for the next step. > Also please provide me the code structure of Sympy to understand the code > base. > I am a newbie. > Hi Karan, SymPy is a different project than SciPy. You can ask for more details about this idea on the SymPy mailing list which is linked on the ideas page. And see sympy.org and https://github.com/sympy/sympy for the SymPy project overview and source code. Cheers, Ralf > On Tue, Feb 24, 2015 at 11:26 AM, Ralf Gommers > wrote: > >> Hi Karan, >> >> >> On Tue, Feb 24, 2015 at 4:04 AM, Karan Dev wrote: >> >>> Hi, >>> >>> I'm final year B.Tech. Computer Science student. I want to participate >>> in gsoc2015 by contributing to the organization. I am comfortable in >>> Python, C/C++, JavaScript. I was wondering if anyone could suggest some >>> bugs that are suitable for a beginner in this project? >>> >> >> Welcome! There are some issues labeled "easy-fix" on >> https://github.com/scipy/scipy/issues that you could have a look at. >> Make sure to check that there's not already an open PR addressing the issue >> though (if there is, that will be visible in the issue). Alternatively, >> tell us if you're interested in a specific GSoC idea or a part of Scipy and >> we'll point you to some issues related to your interest. >> >> Cheers, >> Ralf >> >> >>> Thank You. >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karandev43 at gmail.com Thu Feb 26 01:28:42 2015 From: karandev43 at gmail.com (Karan Dev) Date: Thu, 26 Feb 2015 11:58:42 +0530 Subject: [SciPy-Dev] Introduction: gsoc2015 In-Reply-To: References: Message-ID: apologies for the wrong message, I am just beginner and was exploring PSF projects, and got little confused. On Wed, Feb 25, 2015 at 12:56 PM, Ralf Gommers wrote: > > > On Tue, Feb 24, 2015 at 7:44 PM, Karan Dev wrote: > >> I went through the GSoC 2015 project ideas and found >> https://github.com/sympy/sympy/wiki/GSoC-2015-Ideas#group-theory >> interesting. Please guide me for the next step. >> Also please provide me the code structure of Sympy to understand the code >> base. >> I am a newbie. >> > > Hi Karan, SymPy is a different project than SciPy. You can ask for more > details about this idea on the SymPy mailing list which is linked on the > ideas page. And see sympy.org and https://github.com/sympy/sympy for the > SymPy project overview and source code. > > Cheers, > Ralf > > >> On Tue, Feb 24, 2015 at 11:26 AM, Ralf Gommers >> wrote: >> >>> Hi Karan, >>> >>> >>> On Tue, Feb 24, 2015 at 4:04 AM, Karan Dev wrote: >>> >>>> Hi, >>>> >>>> I'm final year B.Tech. Computer Science student. I want to participate >>>> in gsoc2015 by contributing to the organization. I am comfortable in >>>> Python, C/C++, JavaScript. I was wondering if anyone could suggest some >>>> bugs that are suitable for a beginner in this project? >>>> >>> >>> Welcome! There are some issues labeled "easy-fix" on >>> https://github.com/scipy/scipy/issues that you could have a look at. >>> Make sure to check that there's not already an open PR addressing the issue >>> though (if there is, that will be visible in the issue). Alternatively, >>> tell us if you're interested in a specific GSoC idea or a part of Scipy and >>> we'll point you to some issues related to your interest. >>> >>> Cheers, >>> Ralf >>> >>> >>>> Thank You. >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Thu Feb 26 06:13:32 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 26 Feb 2015 11:13:32 +0000 (UTC) Subject: [SciPy-Dev] scipy.signal function naming References: Message-ID: <56073342446641676.483298sturla.molden-gmail.com@news.gmane.org> Eric Larson wrote: > +1 for more readable names, and putting the MATLAB-equivalent names in the > docstring. I don't think it should be in the docstring because it might indicate we are making a plagiate of MATLAB, which in itself is an IP violation, even if nothing is copied. Instructions on porting from MATLAB to NumPy or SciPy could go in a book, pdf or a wiki page, but not the doctring. Sturla From robert.kern at gmail.com Thu Feb 26 06:31:42 2015 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Feb 2015 11:31:42 +0000 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: <56073342446641676.483298sturla.molden-gmail.com@news.gmane.org> References: <56073342446641676.483298sturla.molden-gmail.com@news.gmane.org> Message-ID: On Thu, Feb 26, 2015 at 11:13 AM, Sturla Molden wrote: > > Eric Larson wrote: > > > +1 for more readable names, and putting the MATLAB-equivalent names in the > > docstring. > > I don't think it should be in the docstring because it might indicate we > are making a plagiate of MATLAB, which in itself is an IP violation, even > if nothing is copied. Instructions on porting from MATLAB to NumPy or SciPy > could go in a book, pdf or a wiki page, but not the doctring. Noting equivalents from other popular systems is a perfectly reasonable thing to do in a docstring. It is a great aid to finding the right function for people who are familiar with those other systems and are trying to migrate, more so than books, PDFs or wiki pages. It does not indicate any kind of IP violation. We should not reduce the functionality of our documentation because a small number of people might be mistaken about that. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From tillsten at zedat.fu-berlin.de Thu Feb 26 07:25:24 2015 From: tillsten at zedat.fu-berlin.de (Till Stensitzki) Date: Thu, 26 Feb 2015 13:25:24 +0100 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: References: Message-ID: +1, i also would like to see some additional namespaces in signal. Maybe put window-functions and the whole filter design each in their own submodules? Right now it is a kitchen sink. Till Am 24.02.2015 um 06:47 schrieb Ralf Gommers: > Hi, > > Historically many names in scipy.signal have followed Matlab, which > typically chooses short but very nondescriptive names. I would prefer to > not keep doing that but instead choose readable names that fit with the > rest of Scipy and Python. Recent examples from PRs, with in brackets a > proposed alternative: > > grpdelay (group_delay) - https://github.com/scipy/scipy/pull/4549 > place (place_poles) - https://github.com/scipy/scipy/pull/4295 > ctrb (controllalbility_matrix) - https://github.com/scipy/scipy/pull/4515 > obsv (observability_matrix) - https://github.com/scipy/scipy/pull/4515 > upfirdn (rate_resample ?) - just discussed on list > > Thoughts? > > Ralf > > From irvin.probst at ensta-bretagne.fr Thu Feb 26 08:11:30 2015 From: irvin.probst at ensta-bretagne.fr (Irvin Probst) Date: Thu, 26 Feb 2015 14:11:30 +0100 Subject: [SciPy-Dev] scipy.signal function naming In-Reply-To: <56073342446641676.483298sturla.molden-gmail.com@news.gmane.org> References: <56073342446641676.483298sturla.molden-gmail.com@news.gmane.org> Message-ID: <1df3da2823566f1a313cb01865fff032@webmail.ensta-bretagne.fr> On Thu, 26 Feb 2015 11:13:32 +0000 (UTC), Sturla Molden wrote: > >> +1 for more readable names, and putting the MATLAB-equivalent names >> in the >> docstring. > > I don't think it should be in the docstring because it might indicate > we > are making a plagiate of MATLAB, which in itself is an IP violation, > even > if nothing is copied. Instructions on porting from MATLAB to NumPy or > SciPy > could go in a book, pdf or a wiki page, but not the doctring. If I may put in my two cents... My full time job is teaching and around 50% of this time is dedicated to using computers for solving scientific problems with my students. Four of five years ago we started to teach Numpy/Scipy as an alternative to Matlab and I can not stress enough how utterly important it is to provide a cookbook with instructions on how to get started for people accustomed to Matlab. I have very bad memories of a day when the Internet went down and http://wiki.scipy.org/NumPy_for_Matlab_Users was not available. So, I have no clue on what can be seen as MATLAB plagiarism or not, but from my user point of view if scipy chooses to have a different and more readable API you'll shoot yourself in the foot if you don't provide an easy way to find MATLAB<->Scipy equivalents in the docstring. -- Irvin From freddyrietdijk at fridh.nl Thu Feb 26 09:33:12 2015 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 26 Feb 2015 15:33:12 +0100 Subject: [SciPy-Dev] Spherical Harmonics and Condon-Shortley phase Message-ID: Hi, I'm working with auralization and Ambisonics, and the directivity patterns that are used with Ambisonics are spherical harmonics. Scipy has an implementation, scipy.special.sph_harm. Several definitions exist however for spherical harmonics, and the documentation does not specify which is implemented. A common definition that is used in quantum-mechanics includes the Condon-Shortley phase, which is a (-1)**m factor. http://en.wikipedia.org/wiki/Spherical_harmonics#Condon.E2.80.93Shortley_phase For my purpose, Ambisonics, I need spherical harmonics without this factor. I found the code, which uses external functions, quite difficult to read. I did see `(-1)**mp` but I'm not sure now whether this really is the CS phase or not. Who knows which definition is used in `sph_harm`? Frederik -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Thu Feb 26 11:12:51 2015 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 26 Feb 2015 17:12:51 +0100 Subject: [SciPy-Dev] ANN: SfePy 2015.1 Message-ID: <54EF4603.8090300@ntc.zcu.cz> I am pleased to announce release 2015.1 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - support for multiple fields in isogeometric analysis - redesigned handling of solver parameters - new modal analysis example For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Lubos Kejzlar, Vladimir Lukes From rmcgibbo at gmail.com Fri Feb 27 21:10:35 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Fri, 27 Feb 2015 18:10:35 -0800 Subject: [SciPy-Dev] Returning the (approximate) inverse hessian from L-BFGS-B as a LinearOperator? In-Reply-To: References: Message-ID: I opened #4572 on github for this. On Wed, Feb 18, 2015 at 9:26 AM, Pauli Virtanen wrote: > Hi, > > 18.02.2015, 03:18, Robert McGibbon kirjoitti: > [clip] > > In the L-BFGS-B minimizer, a low-memory approximation to the inverse > > hessian is > > used internally to determine the descent direction. Once the optimization > > terminates, > > none of this information is returned to the client though. I think it > would > > be a nice to > > return a callable for the inverse hessian - vector product as a > > LinearOperator. > > Depending on what the implicit storage format in lbfgsb is, the > Python-side may already be implemented in > scipy.optimize.nonlin.LowRankMatrix > > The maintenance problem here is that extracting the information requires > mucking with undocumented implementation details of the algorithm, which > may change the author releases yet another new version. > > However, the software is fairly mature, so with proper tests adding this > could be possible. > > A completely separate question then is whether the approximation is > useful for your purpose, and whether you could do better just by > constructing the BFGS secant approximation yourself by recording the > gradients evaluated during your optimization process. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n59_ru at hotmail.com Sat Feb 28 13:04:30 2015 From: n59_ru at hotmail.com (Nikolay Mayorov) Date: Sat, 28 Feb 2015 23:04:30 +0500 Subject: [SciPy-Dev] GSOC Optimization Project Message-ID: Hi! I want to clarify some things about GSOC project idea related to Levenberg-Marquardt algorithm. 1) Why do we want anything but current leastsq based on MINPACK? Looks like it is answered here: https://github.com/scipy/scipy/pull/90 "When we call python from FORTRAN, a lot of magic has to be done. This magic prevents us, for example, to properly pass exceptions through the FORTRANcode." Could you comment more on that perhaps? 2) What's wrong with https://github.com/scipy/scipy/pull/90? Why it is stalled? What do you expect from GSOC student to do better / different? Again partially answered in PR: "It's stalled: the algorithmic part is OK, the new interfaces proposed controversial.", "However, this could perhaps be extended to Levenberg-Marquardt supporting sparse Jacobians" 3) Based on 2: how GSOC student should proceed with interface issue? I mean there weren't any strong opinions and it was on the list for so long. I have no idea how to come up with a good solution all of a sudden. 4) Do you believe that code written during GSOC should be based on PR mentioned? --- That's what I come up so far about the work during GSOC: - Decide on interface part- Add new features to the PR from pv (probably just one of them): Sparse Jacobians support Constraints support- Implement a solid test suite --- I would appreciate your answers, Nikolay. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Feb 28 22:43:51 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 28 Feb 2015 22:43:51 -0500 Subject: [SciPy-Dev] NUFFT Message-ID: Advertised by Jake's blog post (*), NUFFT looks interesting https://github.com/dfm/python-nufft "...it's way faster than Lomb-Scargle!" something for scipy, or do we have to wait for Fortran 90? or maybe it's just a bit of cython. (*) https://jakevdp.github.io/blog/2015/02/24/optimizing-python-with-numpy-and-numba/ Josef