From bjorn.madsen at operationsresearchgroup.com Sun Jun 7 16:32:04 2015 From: bjorn.madsen at operationsresearchgroup.com (Bjorn Madsen) Date: Sun, 7 Jun 2015 21:32:04 +0100 Subject: [SciPy-User] 2D nparray lookup with wild cards - sane usage of 'eval' ? Message-ID: Hi Scipy users, I am looking for suggestions on the lookup function below. It uses 'eval' which has a reputation of "being evil" - however I can't find a better general way of implementing a lookup with wildcards than this. Any suggestions? def lookup2d(keys, array, wildcard='?', return_value=True): if len(keys)!=len(array[0]): raise valueError("Wrong dimensionality: Key must have same length as array[0]") # query constructor c = 'np.where( ' for idx,key in enumerate(keys): if key is not wildcard: c+= '(array[:,{idx}] == {key}) &'.format(idx=idx,key=key) c = c.rstrip('&') c += ' )' # query execution mask = eval(c) if return_value == True: return array[mask] elif return_value == False: return mask a = np.arange(1,13).reshape(3,4) # a = [[1,2,3,4],[5,6,7,8],[9,10,11,12]] a = np.concatenate( (a,a,a,a), axis=0) keys = [5,'?','?',8] print("a:\n", a) print("keys:", keys) print("Looking up keys in a:") print(lookup2d(keys, a, wildcard='?', return_value=False)) print(lookup2d(keys, a, wildcard='?', return_value=True)) -- Bjorn -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jun 7 16:57:00 2015 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 7 Jun 2015 21:57:00 +0100 Subject: [SciPy-User] 2D nparray lookup with wild cards - sane usage of 'eval' ? In-Reply-To: References: Message-ID: On Sun, Jun 7, 2015 at 9:32 PM, Bjorn Madsen < bjorn.madsen at operationsresearchgroup.com> wrote: > > Hi Scipy users, > I am looking for suggestions on the lookup function below. It uses 'eval' which has a reputation of "being evil" - however I can't find a better general way of implementing a lookup with wildcards than this. Any suggestions? > > def lookup2d(keys, array, wildcard='?', return_value=True): > if len(keys)!=len(array[0]): > raise valueError("Wrong dimensionality: Key must have same length as array[0]") > > # query constructor > c = 'np.where( ' > for idx,key in enumerate(keys): > if key is not wildcard: > c+= '(array[:,{idx}] == {key}) &'.format(idx=idx,key=key) > c = c.rstrip('&') > c += ' )' > > # query execution > mask = eval(c) > > if return_value == True: > return array[mask] > elif return_value == False: > return mask > > a = np.arange(1,13).reshape(3,4) > # a = [[1,2,3,4],[5,6,7,8],[9,10,11,12]] > a = np.concatenate( (a,a,a,a), axis=0) > > keys = [5,'?','?',8] > > print("a:\n", a) > print("keys:", keys) > print("Looking up keys in a:") > print(lookup2d(keys, a, wildcard='?', return_value=False)) > print(lookup2d(keys, a, wildcard='?', return_value=True)) Recall that True is the identity value for the & operation. def lookup2d(keys, array, wildcard='?', return_value=True): n = len(array[0]) assert len(keys) == n # I'm being lazy mask = np.ones(array.shape[:-1], dtype=bool) for idx, key in enumerate(keys): if key != wildcard: # don't use "is" for strings and integers mask &= (array[:, idx] == key) mask = np.where(mask) if return_value: return array[mask] else: return mask -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn.madsen at operationsresearchgroup.com Sun Jun 7 17:24:47 2015 From: bjorn.madsen at operationsresearchgroup.com (Bjorn Madsen) Date: Sun, 7 Jun 2015 22:24:47 +0100 Subject: [SciPy-User] 2D nparray lookup with wild cards - sane usage of 'eval' ? In-Reply-To: References: Message-ID: Thanks, - That will do just fine. PS. I've never seen the "&=" operator. Learned something new I guess. On 7 June 2015 at 21:57, Robert Kern wrote: > On Sun, Jun 7, 2015 at 9:32 PM, Bjorn Madsen < > bjorn.madsen at operationsresearchgroup.com> wrote: > > > > Hi Scipy users, > > I am looking for suggestions on the lookup function below. It uses > 'eval' which has a reputation of "being evil" - however I can't find a > better general way of implementing a lookup with wildcards than this. Any > suggestions? > > > > def lookup2d(keys, array, wildcard='?', return_value=True): > > if len(keys)!=len(array[0]): > > raise valueError("Wrong dimensionality: Key must have same > length as array[0]") > > > > # query constructor > > c = 'np.where( ' > > for idx,key in enumerate(keys): > > if key is not wildcard: > > c+= '(array[:,{idx}] == {key}) &'.format(idx=idx,key=key) > > c = c.rstrip('&') > > c += ' )' > > > > # query execution > > mask = eval(c) > > > > if return_value == True: > > return array[mask] > > elif return_value == False: > > return mask > > > > a = np.arange(1,13).reshape(3,4) > > # a = [[1,2,3,4],[5,6,7,8],[9,10,11,12]] > > a = np.concatenate( (a,a,a,a), axis=0) > > > > keys = [5,'?','?',8] > > > > print("a:\n", a) > > print("keys:", keys) > > print("Looking up keys in a:") > > print(lookup2d(keys, a, wildcard='?', return_value=False)) > > print(lookup2d(keys, a, wildcard='?', return_value=True)) > > Recall that True is the identity value for the & operation. > > def lookup2d(keys, array, wildcard='?', return_value=True): > n = len(array[0]) > assert len(keys) == n # I'm being lazy > mask = np.ones(array.shape[:-1], dtype=bool) > for idx, key in enumerate(keys): > if key != wildcard: # don't use "is" for strings and integers > mask &= (array[:, idx] == key) > mask = np.where(mask) > if return_value: > return array[mask] > else: > return mask > > -- > Robert Kern > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Bjorn Madsen *Researcher Complex Systems Research* Ph.: (+44) 0 7792 030 720 bjorn.madsen at operationsresearchgroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jslavin at cfa.harvard.edu Mon Jun 8 12:16:18 2015 From: jslavin at cfa.harvard.edu (Slavin, Jonathan) Date: Mon, 8 Jun 2015 12:16:18 -0400 Subject: [SciPy-User] problem using optimize.root Message-ID: Hi, I'm trying to solve a two-point boundary value problem using odeint in conjunction with root. I've written a function (the objective function) that takes as an argument the eigenvalue for the problem and returns the difference between the endpoint value for one of the variables and the desired value. So the function calls odeint with extra arguments that give the initial conditions at the left boundary as well as the eigenvalue. I can call this function perfectly well from the prompt and it behaves as expected but when I run it within the call to scipy.optimize.root I get the following error: ... /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.pyc in odeint(func, y0, t, args, Dfun, col_deriv, full_output, ml, mu, rtol, atol, tcrit, h0, hmax, hmin, ixpr, mxstep, mxhnil, mxordn, mxords, printmessg) 146 output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, ml, mu, 147 full_output, rtol, atol, tcrit, h0, hmax, hmin, --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) 149 if output[-1] < 0: 150 print(_msgs[output[-1]]) TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe' > /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.py(148)odeint() 147 full_output, rtol, atol, tcrit, h0, hmax, hmin, --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) 149 if output[-1] < 0: So when root calls the function, which in turn calls odeint, something that was previously of float type somehow gets the dtype of object. I've used pdb within ipython to try to find what is happening, but it seems that all the inputs to odeint are of the correct dtype. Any help would be appreciated. Regards, Jon -- ________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin at cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 fax: (617) 496-7577 USA ________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jslavin at cfa.harvard.edu Mon Jun 8 13:16:31 2015 From: jslavin at cfa.harvard.edu (Slavin, Jonathan) Date: Mon, 8 Jun 2015 13:16:31 -0400 Subject: [SciPy-User] problem using optimize.root In-Reply-To: References: Message-ID: Well, I still don't know what the problem was with using root the way I did, but I found a way around this issue by using newton instead. This seems to work just fine. Regards, Jon On Mon, Jun 8, 2015 at 12:16 PM, Slavin, Jonathan wrote: > Hi, > > I'm trying to solve a two-point boundary value problem using odeint in > conjunction with root. I've written a function (the objective function) > that takes as an argument the eigenvalue for the problem and returns the > difference between the endpoint value for one of the variables and the > desired value. So the function calls odeint with extra arguments that give > the initial conditions at the left boundary as well as the eigenvalue. I > can call this function perfectly well from the prompt and it behaves as > expected but when I run it within the call to scipy.optimize.root I get the > following error: > ... > /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.pyc > in odeint(func, y0, t, args, Dfun, col_deriv, full_output, ml, mu, rtol, > atol, tcrit, h0, hmax, hmin, ixpr, mxstep, mxhnil, mxordn, mxords, > printmessg) > 146 output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, > ml, mu, > 147 full_output, rtol, atol, tcrit, h0, > hmax, hmin, > --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) > 149 if output[-1] < 0: > 150 print(_msgs[output[-1]]) > > TypeError: Cannot cast array data from dtype('O') to dtype('float64') > according to the rule 'safe' > > > /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.py(148)odeint() > 147 full_output, rtol, atol, tcrit, h0, > hmax, hmin, > --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) > 149 if output[-1] < 0: > > So when root calls the function, which in turn calls odeint, something > that was previously of float type somehow gets the dtype of object. I've > used pdb within ipython to try to find what is happening, but it seems that > all the inputs to odeint are of the correct dtype. Any help would be > appreciated. > > Regards, > Jon > > -- > ________________________________________________________ > Jonathan D. Slavin Harvard-Smithsonian CfA > jslavin at cfa.harvard.edu 60 Garden Street, MS 83 > phone: (617) 496-7981 Cambridge, MA 02138-1516 > fax: (617) 496-7577 USA > ________________________________________________________ > > -- ________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin at cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 fax: (617) 496-7577 USA ________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Mon Jun 8 13:29:58 2015 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Mon, 8 Jun 2015 13:29:58 -0400 Subject: [SciPy-User] problem using optimize.root In-Reply-To: References: Message-ID: Hi Jon: Could it be because root feeds the func with a numpy.array whereas the newton with a float value. Help outputs: newton(func, x0, fprime=None, args=(), tol=1.48e-08, maxiter=50, fprime2=None) Find a zero using the Newton-Raphson or secant method. Find a zero of the function `func` given a nearby starting point `x0`. The Newton-Raphson method is used if the derivative `fprime` of `func` is provided, otherwise the secant method is used. If the second order derivate `fprime2` of `func` is provided, parabolic Halley's method is used. Parameters ---------- func : function The function whose zero is wanted. It must be a function of a single variable of the form f(x,a,b,c...), where a,b,c... are extra arguments that can be passed in the `args` parameter. x0 : float An initial estimate of the zero that should be somewhere near the actual zero. ..... root(fun, x0, args=(), method='hybr', jac=None, tol=None, callback=None, options=None) Find a root of a vector function. Parameters ---------- fun : callable A vector function to find a root of. x0 : ndarray Initial guess. Cheers 2015-06-08 13:16 GMT-04:00 Slavin, Jonathan : > Well, I still don't know what the problem was with using root the way I > did, but I found a way around this issue by using newton instead. This > seems to work just fine. > > Regards, > Jon > > On Mon, Jun 8, 2015 at 12:16 PM, Slavin, Jonathan > wrote: > >> Hi, >> >> I'm trying to solve a two-point boundary value problem using odeint in >> conjunction with root. I've written a function (the objective function) >> that takes as an argument the eigenvalue for the problem and returns the >> difference between the endpoint value for one of the variables and the >> desired value. So the function calls odeint with extra arguments that give >> the initial conditions at the left boundary as well as the eigenvalue. I >> can call this function perfectly well from the prompt and it behaves as >> expected but when I run it within the call to scipy.optimize.root I get the >> following error: >> ... >> /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.pyc >> in odeint(func, y0, t, args, Dfun, col_deriv, full_output, ml, mu, rtol, >> atol, tcrit, h0, hmax, hmin, ixpr, mxstep, mxhnil, mxordn, mxords, >> printmessg) >> 146 output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, >> ml, mu, >> 147 full_output, rtol, atol, tcrit, h0, >> hmax, hmin, >> --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) >> 149 if output[-1] < 0: >> 150 print(_msgs[output[-1]]) >> >> TypeError: Cannot cast array data from dtype('O') to dtype('float64') >> according to the rule 'safe' >> > >> /export/slavin/python/anaconda/lib/python2.7/site-packages/scipy/integrate/odepack.py(148)odeint() >> 147 full_output, rtol, atol, tcrit, h0, >> hmax, hmin, >> --> 148 ixpr, mxstep, mxhnil, mxordn, mxords) >> 149 if output[-1] < 0: >> >> So when root calls the function, which in turn calls odeint, something >> that was previously of float type somehow gets the dtype of object. I've >> used pdb within ipython to try to find what is happening, but it seems that >> all the inputs to odeint are of the correct dtype. Any help would be >> appreciated. >> >> Regards, >> Jon >> >> -- >> ________________________________________________________ >> Jonathan D. Slavin Harvard-Smithsonian CfA >> jslavin at cfa.harvard.edu 60 Garden Street, MS 83 >> phone: (617) 496-7981 Cambridge, MA 02138-1516 >> fax: (617) 496-7577 USA >> ________________________________________________________ >> >> > > > -- > ________________________________________________________ > Jonathan D. Slavin Harvard-Smithsonian CfA > jslavin at cfa.harvard.edu 60 Garden Street, MS 83 > phone: (617) 496-7981 Cambridge, MA 02138-1516 > fax: (617) 496-7577 USA > ________________________________________________________ > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Sasha -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjhelmus at gmail.com Tue Jun 9 16:38:08 2015 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Tue, 09 Jun 2015 15:38:08 -0500 Subject: [SciPy-User] ANN: Py-ART v1.4.0 released Message-ID: <55774EB0.3080705@gmail.com> I am happy to announce the release of Py-ART version 1.4.0. Py-ART is an open source Python module for reading, visualizing, correcting and analysis of weather radar data. Documentation : http://arm-doe.github.io/pyart/dev/index.html GitHub : https://github.com/ARM-DOE/pyart Pre-build conda binaries: https://binstar.org/jjhelmus/pyart/files?version=1.4.0 Version 1.4.0 is the result of 4 months of work by 7 contributors. Thanks to all contributors, especially those who have made their first contribution to Py-ART. Highlights of this release: * Support for reading and writing MDV Grid files. (thanks to Anderson Gama) * Support for reading GCPEX D3R files. (thanks to Steve Nesbitt) * Support for reading NEXRAD Level 3 files. * Optional loading of radar field data upon use rather than initial read. * Significantly faster gridding method, "map_gates_to_grid". * Improvements to the speed and bug fixes to the region based dealiasing algorithm. * Textures of differential phase fields. (thanks to Scott Collis) * Py-ART now can be used with Python 3.3 and 3.4 Cheers, - Jonathan Helmus From alan.isaac at gmail.com Wed Jun 10 10:29:50 2015 From: alan.isaac at gmail.com (Alan Isaac) Date: Wed, 10 Jun 2015 10:29:50 -0400 Subject: [SciPy-User] fsolve: Reference count error deteected: an attempt was made to deallocate 0 (?) Message-ID: I've used `fsolve` a lot and have never seen this before. Unfortunately, the code that produced it takes hours to run and is complex, so I cannot provide a minimal example. All my code is pure Python+numpy+scipy. Does this suggest a rarely-encountered reference-count bug somewhere in fsolve? Thanks, Alan Isaac -------------- next part -------------- A non-text attachment was scrubbed... Name: temp.png Type: image/png Size: 23970 bytes Desc: not available URL: From archibald at astron.nl Wed Jun 10 11:11:32 2015 From: archibald at astron.nl (Anne Archibald) Date: Wed, 10 Jun 2015 15:11:32 +0000 Subject: [SciPy-User] fsolve: Reference count error deteected: an attempt was made to deallocate 0 (?) In-Reply-To: References: Message-ID: That does seem suspicious. Perhaps the bug occurs in the path where it attempts to deal with a particular kind of failure to converge? FORTRAN is not exactly friendly to exception handling, and that's the kind of path that's hard to trigger. Maybe a carefully-crafted bogus objective function that triggers the "slow convergence" message? Just returning random or noisy data might be a start. Anne On Wed, Jun 10, 2015 at 4:30 PM Alan Isaac wrote: > I've used `fsolve` a lot and have never seen this before. > Unfortunately, the code that produced it takes hours to run and is > complex, so I cannot provide a minimal example. All my code is pure > Python+numpy+scipy. > > Does this suggest a rarely-encountered reference-count bug somewhere in > fsolve? > > Thanks, > Alan Isaac > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Thu Jun 11 17:18:10 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 11 Jun 2015 16:18:10 -0500 Subject: [SciPy-User] ANN: xray v0.5 Message-ID: I'm pleased to announce version 0.5 of xray, N-D labeled arrays and datasets in Python. xray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures. These data structures are based on the data model of the netCDF file format. Highlights of this release: * Support for parallel computation on arrays that don't fit in memory using dask.array (see http://continuum.io/blog/xray-dask for more details) * Support for multi-file datasets * assign and fillna methods, based on the pandas methods of the same name. * to_array and to_dataset methods for easier conversion between xray Dataset and DataArray objects. * Label based indexing with nearest neighbor lookups For more details, read the full release notes: http://xray.readthedocs.org/en/stable/whats-new.html Best, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From eckjoh2 at web.de Thu Jun 11 21:24:56 2015 From: eckjoh2 at web.de (eckjoh2 at web.de) Date: Thu, 11 Jun 2015 20:24:56 -0500 Subject: eckjoh2@web.de möchte Ihnen folgen. Annehmen? Message-ID: <0.0.1D8.A44.1D0A4ADAC6AD56C.2DC7@mail8.flipmailer.com> Hallo, eckjoh2 at web.de m?chte Ihnen folgen. ****** Ist eckjoh2 at web.de Ihr Freund? ****** Ja: http://invites.flipmailer.com/signup_e.html?fullname=Scipy+Users+List&email=scipy-user at scipy.org&invitername=Johannes&inviterid=38074537&userid=0&token=0&emailmasterid=eae34a94-6c4d-4f6f-9b49-8dc6691d18f6&from=eckjoh2 at web.de&template=invite_reg_a&test=AA&src=txt_yes Nein: http://invites.flipmailer.com/signup_e.html?fullname=Scipy+Users+List&email=scipy-user at scipy.org&invitername=Johannes&inviterid=38074537&userid=0&token=0&emailmasterid=eae34a94-6c4d-4f6f-9b49-8dc6691d18f6&from=eckjoh2 at web.de&template=invite_reg_a&test=AA&src=txt_no Klicken Sie hier, um sich aus allen derartigen E-Mails zu entfernen http://invites.flipmailer.com/uns_inviter.jsp?email=scipy-user at scipy.org&iid=eae34a94-6c4d-4f6f-9b49-8dc6691d18f6&from=eckjoh2 at web.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jerome.Kieffer at esrf.fr Sat Jun 13 04:53:06 2015 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Sat, 13 Jun 2015 10:53:06 +0200 Subject: [SciPy-User] [JOBs] Data analysis position at the European synchrotron Message-ID: <20150613105306.4f8b3ed9@patagonia> Dear Pythonistas, The European Synchrotron, ESRF, located in the French Alps, just got approved a large upgrade in which data-analysis is a key element. I am pleased to announce this strategy is built around Python and all code developed in this frame will be based around Python and made open-source. Feel free to distribute this around. 1 metadata manager position: http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=413 2 data scientist positions: http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=421 http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=414 3 software engineer positions http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=418 http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=417 http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=419 Other related data analysis positions http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=420 http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=411 Best regards, Jerome Kieffer From jeffreback at gmail.com Sat Jun 13 13:47:48 2015 From: jeffreback at gmail.com (Jeff Reback) Date: Sat, 13 Jun 2015 13:47:48 -0400 Subject: [SciPy-User] ANN: pandas v0.16.2 released Message-ID: Hello, We are proud to announce v0.16.2 of pandas, a minor release from 0.16.1. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was a release of 4 weeks with 105 commits by 32 authors encompassing 48 issues and 71 pull-requests. We recommend that all users upgrade to this version. *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with ?relational? or ?labeled? data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Highlights of this release include: - - A new *pipe* method, see here - Documentation on how to use numba with *pandas*, see here See the Whatsnew in v0.16.2 Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.9 macosx wheels are courtesy of Matthew Brett Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.16.2 release - Andrew Rosenfeld - Artemy Kolchinsky - Bernard Willers - Christer van der Meeren - Christian Hudon - Constantine Glen Evans - Daniel Julius Lasiman - Evan Wright - Francesco Brundu - Ga?tan de Menten - Jake VanderPlas - James Hiebert - Jeff Reback - Joris Van den Bossche - Justin Lecher - Ka Wo Chen - Kevin Sheppard - Mortada Mehyar - Morton Fox - Robin Wilson - Thomas Grainger - Tom Ajamian - Tom Augspurger - Yoshiki V?zquez Baeza - Younggun Kim - austinc - behzad nouri - jreback - lexual - rekcahpassyla - scls19fr - sinhrks -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.slater at wright.edu Mon Jun 15 15:59:46 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Mon, 15 Jun 2015 15:59:46 -0400 Subject: [SciPy-User] Installing scipy on MacOS X Message-ID: I'm working towards developing for scipy (staying away from compiled code) and I'm being tripped up by an inability to install scipy (16.0) in a virtualenv on MacOS X. I've tried different variants of python (MacPorts, Homebrew, Official distribution) (2.7.10, 3.4), and have tried multiple fortran compilers (starting with form the recommended site, then from elsewhere). I get the same error message regardless. ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status error: Command "/usr/local/bin/gfortran -Wall -g -L/usr/local/lib build/temp.macosx-10.6-intel-3.4/scipy/cluster/_vq.o -L/opt/local/lib -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/5.1.0 -Lbuild/temp.macosx-10.6-intel-3.4 -ltatlas -ltatlas -ltatlas -lgfortran -o build/lib.macosx-10.6-intel-3.4/scipy/cluster/_vq.so" failed with exit status 1 This does remind me why I left Fortran years ago, but isn't entertaining in the least. Can anyone walk me through a successful process? I'd greatly appreciate the help. FWIW: I do have it "working" within MacPorts, but that doesn't suit my expectation of development. I'd like to be able to merge my code into scipy and submit a pull request, eventually. Best Regards, Joe From ralf.gommers at gmail.com Mon Jun 15 16:07:26 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 15 Jun 2015 22:07:26 +0200 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: On Mon, Jun 15, 2015 at 9:59 PM, Slater Joseph C , PhD, PE < joseph.slater at wright.edu> wrote: > I'm working towards developing for scipy (staying away from compiled code) > and I'm being tripped up by an inability to install scipy (16.0) in a > virtualenv on MacOS X. I've tried different variants of python (MacPorts, > Homebrew, Official distribution) (2.7.10, 3.4), and have tried multiple > fortran compilers (starting with form the recommended site, then from > elsewhere). I get the same error message regardless. > > ld: symbol(s) not found for architecture x86_64 > collect2: error: ld returned 1 exit status > error: Command "/usr/local/bin/gfortran -Wall -g -L/usr/local/lib > build/temp.macosx-10.6-intel-3.4/scipy/cluster/_vq.o -L/opt/local/lib > -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/5.1.0 > -Lbuild/temp.macosx-10.6-intel-3.4 -ltatlas -ltatlas -ltatlas -lgfortran -o > build/lib.macosx-10.6-intel-3.4/scipy/cluster/_vq.so" failed with exit > status 1 > This is trying to use ATLAS, do you have that installed? I'd expect all those source builds to use Accelerate. Can you: - give us the full build command you used - put the complete build log in a gist and link to it - check that ``python -c "import numpy; numpy.test('full')`` passes I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. Ralf > > This does remind me why I left Fortran years ago, but isn't entertaining > in the least. Can anyone walk me through a successful process? I'd greatly > appreciate the help. > > FWIW: I do have it "working" within MacPorts, but that doesn't suit my > expectation of development. I'd like to be able to merge my code into scipy > and submit a pull request, eventually. > > Best Regards, > Joe > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jun 15 16:11:22 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 15 Jun 2015 13:11:22 -0700 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: On Mon, Jun 15, 2015 at 1:07 PM, Ralf Gommers wrote: > > > On Mon, Jun 15, 2015 at 9:59 PM, Slater Joseph C , PhD, PE > wrote: >> >> I'm working towards developing for scipy (staying away from compiled code) >> and I'm being tripped up by an inability to install scipy (16.0) in a >> virtualenv on MacOS X. I've tried different variants of python (MacPorts, >> Homebrew, Official distribution) (2.7.10, 3.4), and have tried multiple >> fortran compilers (starting with form the recommended site, then from >> elsewhere). I get the same error message regardless. >> >> ld: symbol(s) not found for architecture x86_64 >> collect2: error: ld returned 1 exit status >> error: Command "/usr/local/bin/gfortran -Wall -g -L/usr/local/lib >> build/temp.macosx-10.6-intel-3.4/scipy/cluster/_vq.o -L/opt/local/lib >> -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/5.1.0 >> -Lbuild/temp.macosx-10.6-intel-3.4 -ltatlas -ltatlas -ltatlas -lgfortran -o >> build/lib.macosx-10.6-intel-3.4/scipy/cluster/_vq.so" failed with exit >> status 1 > > > This is trying to use ATLAS, do you have that installed? I'd expect all > those source builds to use Accelerate. > > Can you: > - give us the full build command you used > - put the complete build log in a gist and link to it > - check that ``python -c "import numpy; numpy.test('full')`` passes > > I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. There should always be a working from-scratch build procedure in https://github.com/MacPython/scipy-wheels/blob/master/.travis.yml It's in .travis.yml format, and it does some post-build processing that is irrelevant for your case, but at least it is a build that works on a default OSX system. Cheers, Matthew From joseph.slater at wright.edu Mon Jun 15 17:13:11 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Mon, 15 Jun 2015 17:13:11 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: > On Jun 15, 2015, at 4:07 PM, Ralf Gommers wrote: > >> > > This is trying to use ATLAS, do you have that installed? I'd expect all those source builds to use Accelerate. I did not install it myself. My understanding from reading the docs is that it would build without is (using Accelerate). Do I have to do something to force it to recognize/use Accelerate? Likely I do, but shouldn't have to from what I understand. > > Can you: > - give us the full build command you used python setup.py install > - put the complete build log in a gist and link to it First time I've seen gist. Cool! https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-scipy-install-log-txt > - check that ``python -c "import numpy; numpy.test('full')`` passes Some failures. Log in the https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt numpy was installed in the virtualenv using pip. sudo -H pip3 install numpy matplotlib no errors resulted from that. > > I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. I'm now at 5.10 . I've been interchanging variables, blindly (due to lack of knowledge). Thanks, Joe > > Ralf > From guziy.sasha at gmail.com Mon Jun 15 20:34:20 2015 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Mon, 15 Jun 2015 20:34:20 -0400 Subject: [SciPy-User] =?iso-8859-1?q?Re=A0=3A__Installing_scipy_on_MacOS_X?= Message-ID: <0b4l9qv0p0vu5dx3pst1330c.1434414860463@email.android.com> Hi Sorry for the trivial questions: Have you activated your virtualenv before using pip? What is the output of pip list? Activate tour virtualenv and to install using pip install scipy. Cheers 15 juin 2015, ? 17:13, "Slater Joseph C , PhD, PE" a ?crit?: > On Jun 15, 2015, at 4:07 PM, Ralf Gommers wrote: > >> > > This is trying to use ATLAS, do you have that installed? I'd expect all those source builds to use Accelerate. I did not install it myself. My understanding from reading the docs is that it would build without is (using Accelerate). Do I have to do something to force it to recognize/use Accelerate? Likely I do, but shouldn't have to from what I understand. > > Can you: > - give us the full build command you used python setup.py install > - put the complete build log in a gist and link to it First time I've seen gist. Cool! https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-scipy-install-log-txt > - check that ``python -c "import numpy; numpy.test('full')`` passes Some failures. Log in the https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt numpy was installed in the virtualenv using pip. sudo -H pip3 install numpy matplotlib no errors resulted from that. > > I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. I'm now at 5.10 . I've been interchanging variables, blindly (due to lack of knowledge). Thanks, Joe > > Ralf > _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From joseph.slater at wright.edu Mon Jun 15 21:05:03 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Mon, 15 Jun 2015 21:05:03 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: > On Jun 15, 2015, at 4:11 PM, Matthew Brett wrote: > > On Mon, Jun 15, 2015 at 1:07 PM, Ralf Gommers wrote: >> >> >> On Mon, Jun 15, 2015 at 9:59 PM, Slater Joseph C , PhD, PE >> wrote: >>> > > There should always be a working from-scratch build procedure in > https://github.com/MacPython/scipy-wheels/blob/master/.travis.yml > > It's in .travis.yml format, and it does some post-build processing > that is irrelevant for your case, but at least it is a build that > works on a default OSX system. > Well, I was able to create a fork (my first), add the .travis.yml file, mod, push, and see that it all built on travis-ci.org, which seems pretty magical, but for the fact that I have no clue how to build on my own system using the .travis.yml file. Is there a doc on how to use it locally? Joe From joseph.slater at wright.edu Mon Jun 15 21:07:21 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Mon, 15 Jun 2015 21:07:21 -0400 Subject: [SciPy-User] =?utf-8?q?Re=C2=A0=3A__Installing_scipy_on_MacOS_X?= In-Reply-To: <0b4l9qv0p0vu5dx3pst1330c.1434414860463@email.android.com> References: <0b4l9qv0p0vu5dx3pst1330c.1434414860463@email.android.com> Message-ID: > On Jun 15, 2015, at 8:34 PM, Oleksandr Huziy wrote: > > Hi > Sorry for the trivial questions: Not at all. A presumption of incompetence on my part may not be far off. I've just started this "journey". > > Have you activated your virtualenv before using pip? Yes. > What is the output of pip list? (scipy-dev)[jslater at Norma scipy-dev]$ pip list Cython (0.22) delocate (0.6.2) matplotlib (1.4.3) nose (1.3.7) numpy (1.9.2) pip (7.0.3) pyparsing (2.0.3) python-dateutil (2.4.2) pytz (2015.4) setuptools (17.0) sh (1.11) six (1.9.0) wheel (0.24.0) > Activate tour virtualenv and to install using pip install scipy. That, however, will undermine my intent, which is to be working with the developer version. I'm not the best in Python... but I'm sufficiently capable, and more so in my field, to extend it. There is another gentleman working on extending signal/spectral overlapping what I'm doing. It makes more sense for me to be working in concert rather than as another disjointed signal processing effort. Nevertheless, I've installed it. It was successful. However, I really need to be using my fork, if anyone can help me figure out what I've done wrong... probably something harebrained. Joe > > Cheers > > > > 15 juin 2015, ? 17:13, "Slater Joseph C , PhD, PE" a ?crit : > > >> On Jun 15, 2015, at 4:07 PM, Ralf Gommers wrote: >> >>> >> >> This is trying to use ATLAS, do you have that installed? I'd expect all those source builds to use Accelerate. > > I did not install it myself. My understanding from reading the docs is that it would build without is (using Accelerate). Do I have to do something to force it to recognize/use Accelerate? Likely I do, but shouldn't have to from what I understand. > >> >> Can you: >> - give us the full build command you used > > python setup.py install > >> - put the complete build log in a gist and link to it > > First time I've seen gist. Cool! > > https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-scipy-install-log-txt > > >> - check that ``python -c "import numpy; numpy.test('full')`` passes > > Some failures. Log in the https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt > > numpy was installed in the virtualenv using pip. > sudo -H pip3 install numpy matplotlib > no errors resulted from that. > >> >> I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. > > I'm now at 5.10 . I've been interchanging variables, blindly (due to lack of knowledge). > > Thanks, > Joe > > >> >> Ralf >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From matthew.brett at gmail.com Mon Jun 15 21:13:12 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 15 Jun 2015 18:13:12 -0700 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: Hi, On Mon, Jun 15, 2015 at 6:05 PM, Slater Joseph C , PhD, PE wrote: > >> On Jun 15, 2015, at 4:11 PM, Matthew Brett wrote: >> >> On Mon, Jun 15, 2015 at 1:07 PM, Ralf Gommers wrote: >>> >>> >>> On Mon, Jun 15, 2015 at 9:59 PM, Slater Joseph C , PhD, PE >>> wrote: >>>> >> >> There should always be a working from-scratch build procedure in >> https://github.com/MacPython/scipy-wheels/blob/master/.travis.yml >> >> It's in .travis.yml format, and it does some post-build processing >> that is irrelevant for your case, but at least it is a build that >> works on a default OSX system. >> > > Well, I was able to create a fork (my first), add the .travis.yml file, mod, push, and see that it all built on travis-ci.org, which seems pretty magical, but for the fact that I have no clue how to build on my own system using the .travis.yml file. Is there a doc on how to use it locally? Ah - no - I only pointed that out in case it was obvious how to read the recipe there. Here's the relevant bits: """ - sudo installer -pkg archives/gfortran-4.2.3.pkg -target / - pip install cython - pip install numpy==$NUMPY_VERSION - cd scipy - export CC=clang - export CXX=clang++ """ which mean, respectively: * Install gfortran 4.2.3 dual arch build from pkg installer (in fact archived at https://github.com/MacPython/scipy-wheels/blob/master/archives/gfortran-4.2.3.pkg for posterity and this repo); * pip install cython numpy * cd scipy * export CC=clang * export CXX=clang++ * python setup.py install # or whatever local install method you like The repo just assures us that that this still works, and builds our wheels for us when we release... Cheers, Matthew From ralf.gommers at gmail.com Tue Jun 16 01:29:18 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jun 2015 07:29:18 +0200 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: On Mon, Jun 15, 2015 at 11:13 PM, Slater Joseph C , PhD, PE < joseph.slater at wright.edu> wrote: > > > On Jun 15, 2015, at 4:07 PM, Ralf Gommers > wrote: > > > >> > > > > This is trying to use ATLAS, do you have that installed? I'd expect all > those source builds to use Accelerate. > > I did not install it myself. My understanding from reading the docs is > that it would build without is (using Accelerate). Do I have to do > something to force it to recognize/use Accelerate? Likely I do, but > shouldn't have to from what I understand. > > > > > Can you: > > - give us the full build command you used > > python setup.py install > > > - put the complete build log in a gist and link to it > > First time I've seen gist. Cool! > > > https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-scipy-install-log-txt > > > > - check that ``python -c "import numpy; numpy.test('full')`` passes > > Some failures. Log in the > https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt > The numpy.f2py tests are failing with the same error as your scipy build, so this is due to an incorrectly installed numpy or the wrong gfortran. Once that works, scipy build probably also works. > numpy was installed in the virtualenv using pip. > sudo -H pip3 install numpy matplotlib > no errors resulted from that. > Never use sudo to install something, leads to annoying issues later on. > > > > > I've been using Homebrew and gfortran 4.9.1 for a year or so without > issues. > > I'm now at 5.10 . I've been interchanging variables, blindly (due to lack > of knowledge). > That's not a stable release right? I would use either 4.2.3 (the one Matthew linked to) or if you use Homebrew then the 4.9.1 that it packages is fine as well. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Jun 16 07:50:09 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 16 Jun 2015 11:50:09 +0000 (UTC) Subject: [SciPy-User] Installing scipy on MacOS X References: Message-ID: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> "Slater Joseph C , PhD, PE" wrote: > I'm working towards developing for scipy (staying away from compiled > code) and I'm being tripped up by an inability to install scipy (16.0) in > a virtualenv on MacOS X. Well, this is what I do... Install Xcode from AppStore. Install Xcode command line utilities using the secret handshake $ /usr/bin/xcode-select --install Install gfortran 4.9 from the gfortran wiki into /usr/local/gfortran. This also gives us version 4.9 of gcc and g++. https://gcc.gnu.org/wiki/GFortranBinaries#MacOS Edit .bash_profile to contain this: export PATH=/usr/local/gfortran/bin:$PATH export CXX=clang++ # or export CXX=g++ export CC=clang # or export CC=gcc export FC=gfortran If the python binary was built with llvm-gcc-4.2.1 you should also have this: export CFLAGS="-mmacosx-version-min=10.6" export CXXFLAGS="-stdlib=libstdc++ -mmacosx-version-min=10.6" This is to make sure you link the correct CRT. -stdlib=libstdc++ is only required for clang++. You don't need it if you use g++. After activating the venv, install Cython and Nose $ pip install cython $ pip install nose Build and install Numpy against Accelerate. Pull the source from Github. Use a stabile maintenance branch of numpy, not the master. $ python setup.py build_ext $ python setup.py install Run the test suite and make sure NumPy works correctly. Now we have a venv for development of SciPy. Make a backup of it for later use. To build scipy for scipy for development: First fork scipy on Github. Clone your forked repo into a local repo, add it as origin. Add https://github.com/scipy/scipy.git as upstream. Building SciPy against Accelerate is simply: $ python setup.py build_ext --inplace Now you can run the tests without installing SciPy into your venv. If your development branch works correctly, fetch upstream/master, rebase your development branch on upstream/master, build and run tests again, push to origin, open a PR on Github, wait for Travis CI to timeout, then complain and have someone restart Travis ;) Sturla From joseph.slater at wright.edu Tue Jun 16 09:53:43 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Tue, 16 Jun 2015 09:53:43 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: <0E3BE7B6-123E-4026-8561-21DCEC4FD5E6@wright.edu> > On Jun 16, 2015, at 1:29 AM, Ralf Gommers wrote: > > > > On Mon, Jun 15, 2015 at 11:13 PM, Slater Joseph C , PhD, PE > wrote: > > > On Jun 15, 2015, at 4:07 PM, Ralf Gommers > wrote: > > > >> > > > - check that ``python -c "import numpy; numpy.test('full')`` passes > > Some failures. Log in the https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt > > The numpy.f2py tests are failing with the same error as your scipy build, so this is due to an incorrectly installed numpy or the wrong gfortran. Once that works, scipy build probably also works. So here I'm a bit lost as to how to install numpy to make it work completely. I've had, nominally, two installations I've tried to get this to work with. One is the MacPorts version, which installs numpy "fine". However, the tests fail there as well. > > > numpy was installed in the virtualenv using pip. > sudo -H pip3 install numpy matplotlib > no errors resulted from that. > > Never use sudo to install something, leads to annoying issues later on. I thought it was necessary in order to install into the site-packages directory. > > > > > > I've been using Homebrew and gfortran 4.9.1 for a year or so without issues. > > I'm now at 5.10 . I've been interchanging variables, blindly (due to lack of knowledge). > > That's not a stable release right? I would use either 4.2.3 (the one Matthew linked to) or if you use Homebrew then the 4.9.1 that it packages is fine as well. I only tried the 5.1 that one time. Tried the 4.2.3 in various configurations since (without luck). -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.slater at wright.edu Tue Jun 16 11:31:17 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Tue, 16 Jun 2015 11:31:17 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: Message-ID: > On Jun 15, 2015, at 9:13 PM, Matthew Brett wrote: > > Hi, > > On Mon, Jun 15, 2015 at 6:05 PM, Slater Joseph C , PhD, PE > wrote: > > The repo just assures us that that this still works, and builds our > wheels for us when we release... > > Cheers, > > Matthew That will be very helpful if I ever get my system to comply. Unfortunately, the build still failed for me. I think I'm running in circles now. A diff between the results of this run and the one I posted yesterday is substantial. I might need to wipe all my python installs clean and start fresh, perhaps? Happy to if it has a chance of working. I don't have anything special that reinstalling wouldn't restore (via macports). I'm not sure I'm using macports for much more than python and python packages anyway. Joe From joseph.slater at wright.edu Tue Jun 16 11:35:26 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Tue, 16 Jun 2015 11:35:26 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> Message-ID: <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> > On Jun 16, 2015, at 7:50 AM, Sturla Molden wrote: > > "Slater Joseph C , PhD, PE" wrote: >> I'm working towards developing for scipy (staying away from compiled >> code) and I'm being tripped up by an inability to install scipy (16.0) in >> a virtualenv on MacOS X. > > Well, this is what I do... > > Install Xcode from AppStore. already had been done > > Install Xcode command line utilities using the secret handshake > $ /usr/bin/xcode-select --install already had been done (required by macports) > > Install gfortran 4.9 from the gfortran wiki into /usr/local/gfortran. This > also gives us version 4.9 of gcc and g++. > https://gcc.gnu.org/wiki/GFortranBinaries#MacOS > > Edit .bash_profile to contain this: > > export PATH=/usr/local/gfortran/bin:$PATH > export CXX=clang++ # or export CXX=g++ > export CC=clang # or export CC=gcc > export FC=gfortran I'm missing this. Will add this and try when I get back. > > If the python binary was built with llvm-gcc-4.2.1 you should also have > this: How do I determine this? Not sure what direction I should go. I can wipe macports. Not sure if I should try to remove the python.org version instead. What would you suggest? > > export CFLAGS="-mmacosx-version-min=10.6" > export CXXFLAGS="-stdlib=libstdc++ -mmacosx-version-min=10.6" > > This is to make sure you link the correct CRT. -stdlib=libstdc++ is only > required for clang++. You don't need it if you use g++. > > After activating the venv, install Cython and Nose > > $ pip install cython > $ pip install nose > > Build and install Numpy against Accelerate. Not sure how to do this. The the Macports version, I've been using their "port install". For the python.org distribution, I'd been installing using pip, which does it's magic without intervention, as I understand. > Pull the source from Github. > Use a stabile maintenance branch of numpy, not the master. > > $ python setup.py build_ext > $ python setup.py install > > Run the test suite and make sure NumPy works correctly. > > Now we have a venv for development of SciPy. Make a backup of it for later > use. > > To build scipy for scipy for development: > > First fork scipy on Github. Clone your forked repo into a local repo, add > it as origin. Add https://github.com/scipy/scipy.git as upstream. > > Building SciPy against Accelerate is simply: > > $ python setup.py build_ext --inplace > > Now you can run the tests without installing SciPy into your venv. > > If your development branch works correctly, fetch upstream/master, rebase > your development branch on upstream/master, build and run tests again, push > to origin, open a PR on Github, wait for Travis CI to timeout, then > complain and have someone restart Travis ;) > > Sturla > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From sturla.molden at gmail.com Tue Jun 16 12:18:55 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 16 Jun 2015 16:18:55 +0000 (UTC) Subject: [SciPy-User] Installing scipy on MacOS X References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> Message-ID: <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> "Slater Joseph C , PhD, PE" wrote: > How do I determine this? Good question. It usually is. > Not sure what direction I should go. I can wipe macports. Not sure if I > should try to remove the python.org version instead. What would you suggest? I don't use Macports or Homebrew, nor the python.org version. Right now I have Anaconda, I used to have Enthought Canopy. Get one of those or build your own Python into a local folder. >> Build and install Numpy against Accelerate. > Not sure how to do this. > The the Macports version, I've been using their "port install". For the > python.org distribution, I'd been installing using pip, which does it's > magic without intervention, as I understand. You can "pip install numpy" if the wheel is built against Accelerate. Ask Matthew Brett. Otherwise see below: > Pull the source from Github. >> Use a stabile maintenance branch of numpy, not the master. >> >> $ python setup.py build_ext >> $ python setup.py install Sturla From sturla.molden at gmail.com Tue Jun 16 12:25:32 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 16 Jun 2015 16:25:32 +0000 (UTC) Subject: [SciPy-User] Installing scipy on MacOS X References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> Message-ID: <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> Sturla Molden wrote: > You can "pip install numpy" if the wheel is built against Accelerate. Ask > Matthew Brett. As I understand it, if NumPy is built against ATLAS, then SciPy will try to do them same. And then you will get an error when it is not found. I was just wondering if that was the kind of ATLAS related you got, which is why I said build NumPy from source. Sturla From matthew.brett at gmail.com Tue Jun 16 16:12:14 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 16 Jun 2015 13:12:14 -0700 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Tue, Jun 16, 2015 at 9:18 AM, Sturla Molden wrote: > "Slater Joseph C , PhD, PE" wrote: > >> How do I determine this? > > Good question. It usually is. > >> Not sure what direction I should go. I can wipe macports. Not sure if I >> should try to remove the python.org version instead. What would you suggest? > > I don't use Macports or Homebrew, nor the python.org version. > > Right now I have Anaconda, I used to have Enthought Canopy. Get one of > those or build your own Python into a local folder. > > >>> Build and install Numpy against Accelerate. > >> Not sure how to do this. >> The the Macports version, I've been using their "port install". For the >> python.org distribution, I'd been installing using pip, which does it's >> magic without intervention, as I understand. > > You can "pip install numpy" if the wheel is built against Accelerate. Ask > Matthew Brett. I use homebrew a lot, but actually not for Python. I use Python.org Python. If you are getting stuck try nuking the Python.org Python directory, something like: rm -rf /Library/Frameworks/Python.framework/Versions/2.7 Then reinstall Python.org Python from the download, make sure that this Python and pip are on the path: $ which python /Library/Frameworks/Python.framework/Versions/2.7/bin/python $ which pip /Library/Frameworks/Python.framework/Versions/2.7/bin/pip Then make sure you have the dual-arch gfortran installed (from the link in the thread above) and: pip install numpy # should install the standard Accelerate-linked wheel pip install cython export CC=clang export CXX=clang++ At that point, you should be able to build scipy OK - let us know if not. Cheers, Matthew From chris.barker at noaa.gov Tue Jun 16 18:07:37 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 16 Jun 2015 15:07:37 -0700 Subject: [SciPy-User] Re : Installing scipy on MacOS X Message-ID: came into this in the middle, but I see near the end of your error log: ld: symbol(s) not found for architecture x86_64 which implies that you may have a mixture of 32 bit and 64 bit stuff in here. So you need to make sure that you are doing all one or the other, you probably want 64 bit everything. Take a look at the build commands that get generated, and see what's there like: --arch i386 and --arch x86-64 (may have those wrong, but you get the idea) Also -- if you have particular excutabl eor shared lib: file the_lib_file.so should tell you which architecture(s) it is built for. -CHB On Mon, Jun 15, 2015 at 6:07 PM, Slater Joseph C , PhD, PE < joseph.slater at wright.edu> wrote: > > > On Jun 15, 2015, at 8:34 PM, Oleksandr Huziy > wrote: > > > > Hi > > Sorry for the trivial questions: > > Not at all. A presumption of incompetence on my part may not be far off. > I've just started this "journey". > > > > Have you activated your virtualenv before using pip? > > Yes. > > > What is the output of pip list? > > (scipy-dev)[jslater at Norma scipy-dev]$ pip list > Cython (0.22) > delocate (0.6.2) > matplotlib (1.4.3) > nose (1.3.7) > numpy (1.9.2) > pip (7.0.3) > pyparsing (2.0.3) > python-dateutil (2.4.2) > pytz (2015.4) > setuptools (17.0) > sh (1.11) > six (1.9.0) > wheel (0.24.0) > > > Activate tour virtualenv and to install using pip install scipy. > > That, however, will undermine my intent, which is to be working with the > developer version. I'm not the best in Python... but I'm sufficiently > capable, and more so in my field, to extend it. There is another gentleman > working on extending signal/spectral overlapping what I'm doing. It makes > more sense for me to be working in concert rather than as another > disjointed signal processing effort. > > Nevertheless, I've installed it. It was successful. However, I really need > to be using my fork, if anyone can help me figure out what I've done > wrong... probably something harebrained. > > Joe > > > > > > > > > Cheers > > > > > > > > 15 juin 2015, ? 17:13, "Slater Joseph C , PhD, PE" < > joseph.slater at wright.edu> a ?crit : > > > > > >> On Jun 15, 2015, at 4:07 PM, Ralf Gommers > wrote: > >> > >>> > >> > >> This is trying to use ATLAS, do you have that installed? I'd expect all > those source builds to use Accelerate. > > > > I did not install it myself. My understanding from reading the docs is > that it would build without is (using Accelerate). Do I have to do > something to force it to recognize/use Accelerate? Likely I do, but > shouldn't have to from what I understand. > > > >> > >> Can you: > >> - give us the full build command you used > > > > python setup.py install > > > >> - put the complete build log in a gist and link to it > > > > First time I've seen gist. Cool! > > > > > https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-scipy-install-log-txt > > > > > >> - check that ``python -c "import numpy; numpy.test('full')`` passes > > > > Some failures. Log in the > https://gist.github.com/josephcslater/0930741203fd02dbcc0f#file-gistfile1-txt > > > > numpy was installed in the virtualenv using pip. > > sudo -H pip3 install numpy matplotlib > > no errors resulted from that. > > > >> > >> I've been using Homebrew and gfortran 4.9.1 for a year or so without > issues. > > > > I'm now at 5.10 . I've been interchanging variables, blindly (due to > lack of knowledge). > > > > Thanks, > > Joe > > > > > >> > >> Ralf > >> > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.slater at wright.edu Tue Jun 16 21:12:29 2015 From: joseph.slater at wright.edu (Slater Joseph C , PhD, PE) Date: Tue, 16 Jun 2015 21:12:29 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> Message-ID: > On Jun 16, 2015, at 12:25 PM, Sturla Molden wrote: > > Sturla Molden wrote: > >> You can "pip install numpy" if the wheel is built against Accelerate. Ask >> Matthew Brett. > > As I understand it, if NumPy is built against ATLAS, then SciPy will try to > do them same. And then you will get an error when it is not found. I was > just wondering if that was the kind of ATLAS related you got, which is why > I said build NumPy from source. > > I'll try doing this next. A lot of directions/support, taking me a long time to iterate around them. What python do you have installed? I'm happy to exactly copy what anyone else has that works. > Sturla > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From matthew.brett at gmail.com Tue Jun 16 21:17:54 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 16 Jun 2015 18:17:54 -0700 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Tue, Jun 16, 2015 at 6:12 PM, Slater Joseph C , PhD, PE wrote: >> On Jun 16, 2015, at 12:25 PM, Sturla Molden wrote: >> >> Sturla Molden wrote: >> >>> You can "pip install numpy" if the wheel is built against Accelerate. Ask >>> Matthew Brett. >> >> As I understand it, if NumPy is built against ATLAS, then SciPy will try to >> do them same. And then you will get an error when it is not found. I was >> just wondering if that was the kind of ATLAS related you got, which is why >> I said build NumPy from source. >> >> > > I'll try doing this next. A lot of directions/support, taking me a long time to iterate around them. What python do you have installed? I'm happy to exactly copy what anyone else has that works. Did you try with Python.org Python using the instructions in my last email? I think those instructions should work. If not - what error do you get? You might also consider checking which gfortran you have with $ which gfortran and $ gfortran --version Cheers, Matthew From sturla.molden at gmail.com Tue Jun 16 21:56:16 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 17 Jun 2015 01:56:16 +0000 (UTC) Subject: [SciPy-User] Installing scipy on MacOS X References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> Message-ID: <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> "Slater Joseph C , PhD, PE" wrote: > I'll try doing this next. A lot of directions/support, taking me a long > time to iterate around them. What python do you have installed? I'm happy > to exactly copy what anyone else has that works. I have Anaconda from continuum.io (Python 3.4). I use gfortran binaries from the gfortran wiki, not the dual architecture build that Matthew Brett recommends. With Anaconda you can create an environment for developing SciPy like so (from the top of my head): $ conda create --mkdir --prefix ~/scipydev-env python=3.4 numpy=1.9 cython nose pyflakes pep8 Sturla From chris.barker at noaa.gov Wed Jun 17 14:29:39 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 17 Jun 2015 11:29:39 -0700 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Jun 16, 2015 at 6:56 PM, Sturla Molden wrote: > "Slater Joseph C , PhD, PE" wrote: > > > I'll try doing this next. A lot of directions/support, taking me a long > > time to iterate around them. What python do you have installed? I'm happy > > to exactly copy what anyone else has that works. > > > I have Anaconda from continuum.io (Python 3.4). > which is 64 bit only. > I use gfortran binaries from the gfortran wiki, not the dual architecture > build that Matthew Brett recommends. > which will get you around the dual architecture complications... I take it Anaconda doesn't provide gfortran -- too bad, that would be nice. However, there are a couple on binstar -- maybe worth a shot? $ binstar search gfortran Using binstar api site https://api.anaconda.org Run 'binstar show ' to get more details: Packages: Name | Version | Package Types | Platforms ------------------------- | ------ | --------------- | --------------- OpenMDAO/libgfortran | 4.8.3 | conda | linux-32, osx-64 dfroger/gfortran | 4.8.2 | conda | osx-64 : The GNU Compiler Collection dfroger/libgfortran | 4.8.2 | conda | osx-64 : The GNU Compiler Collection Found 3 packages -Chris > > With Anaconda you can create an environment for developing SciPy like so > (from the top of my head): > > > $ conda create --mkdir --prefix ~/scipydev-env python=3.4 numpy=1.9 cython > nose pyflakes pep8 > > > Sturla > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Wed Jun 17 15:02:47 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 17 Jun 2015 21:02:47 +0200 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> Message-ID: On 17/06/15 20:29, Chris Barker wrote: > I have Anaconda from continuum.io (Python 3.4). > > > which is 64 bit only. > > I use gfortran binaries from the gfortran wiki, not the dual > architecture > build that Matthew Brett recommends. > > > which will get you around the dual architecture complications... Yes. That is good enough for SciPy development and for my personal use. I can understand why Matthew Brett wants a dual architecture gfortran for his official binary wheels. But that is a totally different matter. > I take it Anaconda doesn't provide gfortran -- too bad, that would be nice. They don't, but they should :) The gfortran maintainers does provide binaries: https://gcc.gnu.org/wiki/GFortranBinaries It supports Fortran, C, C++, Objective C and Objective-C++. It has OpenMP and it builds OpenBLAS, NumPy and SciPy without any fuzz. That is all I personally care about. Sturla From sturla.molden at gmail.com Wed Jun 17 15:08:51 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 17 Jun 2015 21:08:51 +0200 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> Message-ID: On 17/06/15 21:02, Sturla Molden wrote: >> which will get you around the dual architecture complications... > > Yes. > > That is good enough for SciPy development and for my personal use. And for publishing on AppStore it would also be a different story. Sturla From sturla.molden at gmail.com Wed Jun 17 15:14:04 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 17 Jun 2015 21:14:04 +0200 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> <1089033638456196768.726447sturla.molden-gmail.com@news.gmane.org> Message-ID: On 17/06/15 21:02, Sturla Molden wrote: >> I take it Anaconda doesn't provide gfortran -- too bad, that would be nice. > > They don't, but they should :) > > The gfortran maintainers does provide binaries: > > https://gcc.gnu.org/wiki/GFortranBinaries > > It supports Fortran, C, C++, Objective C and Objective-C++. It has > OpenMP and it builds OpenBLAS, NumPy and SciPy without any fuzz. That is > all I personally care about. Also I have used it with Enthought Canopy (Python 2.7, 64 bit) and it works ok for building NumPy and SciPy. Sturla From rob.clewley at gmail.com Fri Jun 19 11:57:07 2015 From: rob.clewley at gmail.com (Rob Clewley) Date: Fri, 19 Jun 2015 11:57:07 -0400 Subject: [SciPy-User] Installing scipy on MacOS X In-Reply-To: <9C1688E9-53E4-473C-91A6-38E3B88D4558@gmail.com> References: <1152062515456142807.267474sturla.molden-gmail.com@news.gmane.org> <919F99EC-DE78-49A6-AD08-13A1093D278C@wright.edu> <859528827456163473.424506sturla.molden-gmail.com@news.gmane.org> <563855820456164510.253832sturla.molden-gmail.com@news.gmane.org> <9C1688E9-53E4-473C-91A6-38E3B88D4558@gmail.com> Message-ID: Hi Joe, > Getting this working also fixed compilation issues I was having with installing PyDSTool and running the examples. Rob Clewley was subject to my haplessness in April. He is probably too kind to say it, but I'm sure he was at least a bit happy that I "went away". He may suffer for my return! Hopefully I turn around and start adding value to the community soon. I reapply appreciate all the help that was heaped on me (I was feverishly trying to try all of it, it came so fast). It is precisely the reason I "jumped communities" from being a longtime Matlab user/developer/advocate. I certainly appreciate all of the community's help as I continue to learn: Yes, and I've confirmed in the meantime that you basically will not be able to use PyDSTool's C ODE integrators on a 64 bit Windows platform. Numpy and scipy in Anaconda or the other pre-made packages were not built in a way that makes them compatible with compilers other than MVSC, so the usual cygwin/mingw route with gcc will not work. The only solution, AFAIK, is that you buy MVSC. There are a few more tech details posted on the sourceforge forum, but that's the short version. Sadly, I still have to recommend the fully 32-bit install of the scipy stack for use with PyDSTool, or any software that needs to compile using distutils. -Rob From carsonc at gmail.com Mon Jun 22 10:50:56 2015 From: carsonc at gmail.com (Cantwell Carson) Date: Mon, 22 Jun 2015 10:50:56 -0400 Subject: [SciPy-User] Increasing the maximum number of points in the spatial.Delaunay triangulation Message-ID: I am working on an image processing algorithm that relies on the Delaunay triangulation of a 3d image. Unfortunately, I need to work on images with more than 2**24 points that go into the Delaunay triangulation. This is problematic because the Qhull routine labels the input vertices with a 24-bit label. I have tried editing the source code for qhull here in libqhull.h, lines 352, 380 and 663 - 664: unsigned id:32; /* unique identifier, =>room for 8 flags, bit field matches qh.ridge_id */ ... unsigned id:32; /* unique identifier, bit field matches qh.vertex_id */ ... unsigned ridge_id:32; /* ID of next, new ridge from newridge() */ unsigned vertex_id:32; /* ID of next, new vertex from newvertex() */ poly.c, lines 569, 1019 - 1023: long ridgeid; ... if (qh ridge_id == 0xFFFFFFFF) { qh_fprintf(qh ferr, 7074, "\ qhull warning: more than %d ridges. ID field overflows and two ridges\n\ may have the same identifier. Otherwise output ok.\n", 0xFFFFFFFF); and poly2.c, lines 2277 - 2279: if (qh vertex_id == 0xFFFFFFFF) { qh_fprintf(qh ferr, 6159, "qhull error: more than %d vertices. ID field overflows and two vertices\n\ may have the same identifier. Vertices will not be sorted correctly.\n", 0xFFFFFFFF); This compiles fine, but so far, this produces a memory leak (as far as I can tell) when I put more than 2**24 points into the algorithm. I have emailed Brad Barber, who indicated that this changing the ridgeid field was possible. I am also working on a divide and conquer approach, but am having trouble deleting the old Delaunay object before instantiating the next one. I can send that code along if there is no good way to increase the number of points in the algorithm. I think it would be useful to have an algorithm that can accept more points, given the increased speed and processing power of modern computers. Thanks for your help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Jun 27 10:11:59 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 27 Jun 2015 10:11:59 -0400 Subject: [SciPy-User] notebook crash with fmin_lbfgsb Message-ID: Just a general question: Has anybody experience ipython notebook crashes with fmin_lbfgsb? Is it just my setup or is this more general? When I ran an estimation that uses fmin_lbfgsb for the optimization, then the notebook kernel shuts down and restarts. (Windows 8.1, winpython 3.4, previous release of ipython, I think) 2015-06-27 09:39:42.956 [NotebookApp] Saving notebook at /Untitled0.ipynb RUNNING THE L-BFGS-B CODE * * * Machine precision = 2.220D-16 N = 5 M = 12 forrtl: severe (47): write to READONLY file, unit 8, file M:\josef_new\eclipse_w s\statsmodels\statsmodels_py34\examples\notebooks\iterate.dat Image PC Routine Line Source libifcoremd.dll 0000000009154E2C Unknown Unknown Unknown _lbfgsb.pyd 00007FFC31529C01 Unknown Unknown Unknown _lbfgsb.pyd 00007FFC31524AD3 Unknown Unknown Unknown _lbfgsb.pyd 00007FFC3152465C Unknown Unknown Unknown _lbfgsb.pyd 00007FFC315224BF Unknown Unknown Unknown python34.dll 0000000073C28771 Unknown Unknown Unknown python34.dll 0000000073CE4074 Unknown Unknown Unknown ... python34.dll 0000000073BCD4D3 Unknown Unknown Unknown python.exe 000000001C1911AE Unknown Unknown Unknown KERNEL32.DLL 00007FFC56E313D2 Unknown Unknown Unknown ntdll.dll 00007FFC59335444 Unknown Unknown Unknown 2015-06-27 09:40:04.227 [NotebookApp] KernelRestarter: restarting kernel (1/5) WARNING:root:kernel a46450ce-69fe-4034-9869-4f1f3c543390 restarted AFAIR, I didn't have problems with spyder AFAIR, setting iprint=0 avoided the crash in the previous example, but doens't seem to fix it in my current example. I switched to bfgs, which works well in my example (aside: example is statsmodels ARIMA which uses fmin_lbfgsb by default) Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Mon Jun 29 04:15:13 2015 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Mon, 29 Jun 2015 18:15:13 +1000 Subject: [SciPy-User] Why is Ward limited to Euclidean/full data matrix? Message-ID: The Wikipedia page seems to suggest that Ward's method can be implemented just from the distance matrix. Why is it restricted to using the full dataset and only Euclidean distance? https://en.wikipedia.org/wiki/Ward's_method#Lance.E2.80.93Williams_algorithms -------------- next part -------------- An HTML attachment was scrubbed... URL: From lorenzo.isella at gmail.com Mon Jun 29 11:03:09 2015 From: lorenzo.isella at gmail.com (Lorenzo Isella) Date: Mon, 29 Jun 2015 17:03:09 +0200 Subject: [SciPy-User] Quick Implementation of Matrix via a Sum Message-ID: <20150629150309.GA4854@localhost.localdomain> Dear All, My scipy is unbelievably rusty and I am struggling with a simple problem. Please consider the following quantity (which I write a la LaTex) D_{n,r}=\frac{1}{n} \sum_{j=1}^r frac{1}{n-j+1} for r=1,2,...n How can I implement this efficiently as a numpy array? I am struggling with newaxis, but I am not going very far. Any suggestion is welcome. Cheers Lorenzo From guziy.sasha at gmail.com Mon Jun 29 11:45:19 2015 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Mon, 29 Jun 2015 11:45:19 -0400 Subject: [SciPy-User] Quick Implementation of Matrix via a Sum In-Reply-To: <20150629150309.GA4854@localhost.localdomain> References: <20150629150309.GA4854@localhost.localdomain> Message-ID: Hi Lorenzo: Is this what you want? https://github.com/guziy/PyNotebooks/blob/master/np_demo1.ipynb Cheers 2015-06-29 11:03 GMT-04:00 Lorenzo Isella : > Dear All, > My scipy is unbelievably rusty and I am struggling with a simple > problem. > Please consider the following quantity (which I write a la LaTex) > > D_{n,r}=\frac{1}{n} \sum_{j=1}^r frac{1}{n-j+1} for r=1,2,...n > > How can I implement this efficiently as a numpy array? > I am struggling with newaxis, but I am not going very far. > Any suggestion is welcome. > Cheers > > Lorenzo > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Sasha -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkhilmer at chemistry.montana.edu Mon Jun 29 11:55:31 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Mon, 29 Jun 2015 09:55:31 -0600 Subject: [SciPy-User] Quick Implementation of Matrix via a Sum In-Reply-To: References: <20150629150309.GA4854@localhost.localdomain> Message-ID: Or extended to handle a vector of n: 1/n * (1/(n[:,numpy.newaxis]-j+1)).sum(axis=1) On Mon, Jun 29, 2015 at 9:45 AM, Oleksandr Huziy wrote: > Hi Lorenzo: > > Is this what you want? > > https://github.com/guziy/PyNotebooks/blob/master/np_demo1.ipynb > > Cheers > > 2015-06-29 11:03 GMT-04:00 Lorenzo Isella : > >> Dear All, >> My scipy is unbelievably rusty and I am struggling with a simple >> problem. >> Please consider the following quantity (which I write a la LaTex) >> >> D_{n,r}=\frac{1}{n} \sum_{j=1}^r frac{1}{n-j+1} for r=1,2,...n >> >> How can I implement this efficiently as a numpy array? >> I am struggling with newaxis, but I am not going very far. >> Any suggestion is welcome. >> Cheers >> >> Lorenzo >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > > -- > Sasha > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lorenzo.isella at gmail.com Mon Jun 29 14:46:19 2015 From: lorenzo.isella at gmail.com (Lorenzo Isella) Date: Mon, 29 Jun 2015 20:46:19 +0200 Subject: [SciPy-User] Quick Implementation of Matrix via a Sum In-Reply-To: References: Message-ID: <20150629184619.GA1316@argo> >Hi Lorenzo: > >Is this what you want? > >https://github.com/guziy/PyNotebooks/blob/master/np_demo1.ipynb > >Cheers > >2015-06-29 11:03 GMT-04:00 Lorenzo Isella : > >> Dear All, >> My scipy is unbelievably rusty and I am struggling with a simple >> problem. >> Please consider the following quantity (which I write a la LaTex) >> >> D_{n,r}=\frac{1}{n} \sum_{j=1}^r frac{1}{n-j+1} for r=1,2,...n >> >> How can I implement this efficiently as a numpy array? >> I am struggling with newaxis, but I am not going very far. >> Any suggestion is welcome. >> Cheers >> >> Lorenzo >> Perfect! Thanks. Lorenzo From nitinchandra1 at gmail.com Mon Jun 29 14:59:28 2015 From: nitinchandra1 at gmail.com (nitin chandra) Date: Tue, 30 Jun 2015 00:29:28 +0530 Subject: [SciPy-User] Thinking in HDF5 Message-ID: Hello All, I have been on this for a long time and have done a lot of research / reading on this topic, HDF5. To use it for development of in-house software for design calculation of various alignments. A lot many time the basic alignment information is picked from a map / satellite imageries. Kilometre, ground level, geographic feature on the proposed route, etc. , in a csv file. Any where between 7-9 files are generated for each alignment. 1. How to read / write data into the HDF5 file ? What after creating root ? 2. In a regular situation, I have 2 input file with 3 column csv data, and 2 output files. How to do that within the same root ? Will be working in Python and c / c++. This is just the basic. If I can do this much, I can do the rest, hopefully :) Thanks to All Nitin PS :- Will post in boost-user list too. From kevin.gullikson at gmail.com Mon Jun 29 15:07:45 2015 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Mon, 29 Jun 2015 14:07:45 -0500 Subject: [SciPy-User] Thinking in HDF5 In-Reply-To: References: Message-ID: Check out h5py . It should work for reading/writing hdf5 files for you. Kevin Gullikson PhD Candidate University of Texas Astronomy RLM 15.310E On Mon, Jun 29, 2015 at 1:59 PM, nitin chandra wrote: > Hello All, > > I have been on this for a long time and have done a lot of research / > reading on this topic, HDF5. > > To use it for development of in-house software for design calculation > of various alignments. > > A lot many time the basic alignment information is picked from a map / > satellite imageries. Kilometre, ground level, geographic feature on > the proposed route, etc. , in a csv file. > > Any where between 7-9 files are generated for each alignment. > > 1. How to read / write data into the HDF5 file ? What after creating root ? > > 2. In a regular situation, I have 2 input file with 3 column csv data, > and 2 output files. > How to do that within the same root ? > > Will be working in Python and c / c++. > > This is just the basic. If I can do this much, I can do the rest, > hopefully :) > > Thanks to All > > Nitin > > PS :- Will post in boost-user list too. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathanrocher at gmail.com Mon Jun 29 15:40:55 2015 From: jonathanrocher at gmail.com (Jonathan Rocher) Date: Mon, 29 Jun 2015 14:40:55 -0500 Subject: [SciPy-User] Thinking in HDF5 In-Reply-To: References: Message-ID: Indeed h5py is a good library that is a direct wrapper of the C++ library and will expose most of the bells and whistles. I also recommend you take a look at pytables which is really nice to use, maybe a little more pythonic, and it integrates with numexpr for out of core computations. HTH, Jonathan On Mon, Jun 29, 2015 at 2:07 PM, Kevin Gullikson wrote: > Check out h5py . It should work for reading/writing > hdf5 files for you. > > > Kevin Gullikson > PhD Candidate > University of Texas Astronomy > RLM 15.310E > > On Mon, Jun 29, 2015 at 1:59 PM, nitin chandra > wrote: > >> Hello All, >> >> I have been on this for a long time and have done a lot of research / >> reading on this topic, HDF5. >> >> To use it for development of in-house software for design calculation >> of various alignments. >> >> A lot many time the basic alignment information is picked from a map / >> satellite imageries. Kilometre, ground level, geographic feature on >> the proposed route, etc. , in a csv file. >> >> Any where between 7-9 files are generated for each alignment. >> >> 1. How to read / write data into the HDF5 file ? What after creating root >> ? >> >> 2. In a regular situation, I have 2 input file with 3 column csv data, >> and 2 output files. >> How to do that within the same root ? >> >> Will be working in Python and c / c++. >> >> This is just the basic. If I can do this much, I can do the rest, >> hopefully :) >> >> Thanks to All >> >> Nitin >> >> PS :- Will post in boost-user list too. >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Jonathan Rocher Austin TX, USA http://www.linkedin.com/in/jonathanrocher ------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From maximilian.albert at gmail.com Mon Jun 29 19:48:35 2015 From: maximilian.albert at gmail.com (Maximilian Albert) Date: Tue, 30 Jun 2015 00:48:35 +0100 Subject: [SciPy-User] Thinking in HDF5 In-Reply-To: References: Message-ID: I agree with previous answers, h5py and pytables are good choices. If you have the option of working a bit higher-level then definitely check out 'odo' as well (which is part of the blaze ecosystem): http://blaze.pydata.org/en/latest/csv.html#migrate-to-binary-storage-formats http://odo.readthedocs.org/en/latest/uri.html http://odo.readthedocs.org/en/latest/hdf5.html I haven't used it in conjunction with HDF5 files yet, but used it to convert CSV files to SQLite databases, and was totally blown away by how easy it makes the conversion. Cheers, Max 2015-06-29 20:40 GMT+01:00 Jonathan Rocher : > Indeed h5py is a good library that is a direct > wrapper of the C++ library and will expose most of the bells and whistles. > I also recommend you take a look at pytables > which is really nice to use, maybe a little more pythonic, and it > integrates with numexpr for out of > core computations. > > HTH, > Jonathan > > On Mon, Jun 29, 2015 at 2:07 PM, Kevin Gullikson < > kevin.gullikson at gmail.com> wrote: > >> Check out h5py . It should work for >> reading/writing hdf5 files for you. >> >> >> Kevin Gullikson >> PhD Candidate >> University of Texas Astronomy >> RLM 15.310E >> >> On Mon, Jun 29, 2015 at 1:59 PM, nitin chandra >> wrote: >> >>> Hello All, >>> >>> I have been on this for a long time and have done a lot of research / >>> reading on this topic, HDF5. >>> >>> To use it for development of in-house software for design calculation >>> of various alignments. >>> >>> A lot many time the basic alignment information is picked from a map / >>> satellite imageries. Kilometre, ground level, geographic feature on >>> the proposed route, etc. , in a csv file. >>> >>> Any where between 7-9 files are generated for each alignment. >>> >>> 1. How to read / write data into the HDF5 file ? What after creating >>> root ? >>> >>> 2. In a regular situation, I have 2 input file with 3 column csv data, >>> and 2 output files. >>> How to do that within the same root ? >>> >>> Will be working in Python and c / c++. >>> >>> This is just the basic. If I can do this much, I can do the rest, >>> hopefully :) >>> >>> Thanks to All >>> >>> Nitin >>> >>> PS :- Will post in boost-user list too. >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Jonathan Rocher > Austin TX, USA > http://www.linkedin.com/in/jonathanrocher > ------------------------------------------------------------- > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nitinchandra1 at gmail.com Tue Jun 30 11:01:21 2015 From: nitinchandra1 at gmail.com (nitin chandra) Date: Tue, 30 Jun 2015 20:31:21 +0530 Subject: [SciPy-User] Thinking in HDF5 In-Reply-To: References: Message-ID: Thank you All, Just going through the links and suggestion. Shall buzz the list as and when I get buzzeedd :) .. Thanks Nitin On 30 June 2015 at 05:18, Maximilian Albert wrote: > I agree with previous answers, h5py and pytables are good choices. > > If you have the option of working a bit higher-level then definitely check > out 'odo' as well (which is part of the blaze ecosystem): > > > http://blaze.pydata.org/en/latest/csv.html#migrate-to-binary-storage-formats > http://odo.readthedocs.org/en/latest/uri.html > http://odo.readthedocs.org/en/latest/hdf5.html > > I haven't used it in conjunction with HDF5 files yet, but used it to convert > CSV files to SQLite databases, and was totally blown away by how easy it > makes the conversion. > > Cheers, > Max > > > 2015-06-29 20:40 GMT+01:00 Jonathan Rocher : >> >> Indeed h5py is a good library that is a direct wrapper of the C++ library >> and will expose most of the bells and whistles. I also recommend you take a >> look at pytables which is really nice to use, maybe a little more pythonic, >> and it integrates with numexpr for out of core computations. >> >> HTH, >> Jonathan >> >> On Mon, Jun 29, 2015 at 2:07 PM, Kevin Gullikson >> wrote: >>> >>> Check out h5py. It should work for reading/writing hdf5 files for you. >>> >>> >>> Kevin Gullikson >>> PhD Candidate >>> University of Texas Astronomy >>> RLM 15.310E >>> >>> On Mon, Jun 29, 2015 at 1:59 PM, nitin chandra >>> wrote: >>>> >>>> Hello All, >>>> >>>> I have been on this for a long time and have done a lot of research / >>>> reading on this topic, HDF5. >>>> >>>> To use it for development of in-house software for design calculation >>>> of various alignments. >>>> >>>> A lot many time the basic alignment information is picked from a map / >>>> satellite imageries. Kilometre, ground level, geographic feature on >>>> the proposed route, etc. , in a csv file. >>>> >>>> Any where between 7-9 files are generated for each alignment. >>>> >>>> 1. How to read / write data into the HDF5 file ? What after creating >>>> root ? >>>> >>>> 2. In a regular situation, I have 2 input file with 3 column csv data, >>>> and 2 output files. >>>> How to do that within the same root ? >>>> >>>> Will be working in Python and c / c++. >>>> >>>> This is just the basic. If I can do this much, I can do the rest, >>>> hopefully :) >>>> >>>> Thanks to All >>>> >>>> Nitin >>>> >>>> PS :- Will post in boost-user list too. >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> >> >> -- >> Jonathan Rocher >> Austin TX, USA >> http://www.linkedin.com/in/jonathanrocher >> ------------------------------------------------------------- >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user >