From thomas.robitaille at gmail.com Mon Mar 2 10:23:29 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 2 Mar 2015 16:23:29 +0100 Subject: [SciPy-User] Scientific Python Survey Message-ID: Hi everyone, If you use scientific Python packages for your research/work, I would appreciate if you could take a few minutes to fill out the following survey: http://goo.gl/PXzFAk The aim of this survey is to find out what versions of Python and various scientific Python packages people are using, and how people typically install packages, in order to determine how developers can better meet the needs of the Scientific Python community (for example, a common question is which version of Numpy need to be supported by packages). This is a follow-up to a similar survey which I did back in 2012 and which provided very interesting results that you can read about here: http://astrofrog.github.io/blog/2013/01/13/what-python-installations-are-scientists-using Please feel free to forward this survey to people in your own scientific Python communities! I will publish the results online in a few weeks. Thanks! Tom From alan.isaac at gmail.com Mon Mar 2 10:41:26 2015 From: alan.isaac at gmail.com (Alan G Isaac) Date: Mon, 02 Mar 2015 10:41:26 -0500 Subject: [SciPy-User] Scientific Python Survey In-Reply-To: References: Message-ID: <54F484A6.2040008@gmail.com> On 3/2/2015 10:23 AM, Thomas Robitaille wrote: > http://goo.gl/PXzFAk A comment on the instructions: Windows requires double quotes to give python commands at a DOS prompt. Alan Isaac From moorepants at gmail.com Mon Mar 2 11:33:42 2015 From: moorepants at gmail.com (Jason Moore) Date: Mon, 2 Mar 2015 08:33:42 -0800 Subject: [SciPy-User] Scientific Python Survey In-Reply-To: References: Message-ID: These days with conda, it is so easy to have numerous versions of all these packages. For example, I have conda environments for different projects and have a range of numpy versions used in each, from 1.6.1 to 1.9.1. What version would you expect us to enter if we use lots of versions in production scenarios? Jason moorepants.info +01 530-601-9791 On Mon, Mar 2, 2015 at 7:23 AM, Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > Hi everyone, > > If you use scientific Python packages for your research/work, I would > appreciate if you could take a few minutes to fill out the following > survey: > > http://goo.gl/PXzFAk > > The aim of this survey is to find out what versions of Python and > various scientific Python packages people are using, and how people > typically install packages, in order to determine how developers can > better meet the needs of the Scientific Python community (for example, > a common question is which version of Numpy need to be supported by > packages). > > This is a follow-up to a similar survey which I did back in 2012 and > which provided very interesting results that you can read about here: > > > http://astrofrog.github.io/blog/2013/01/13/what-python-installations-are-scientists-using > > Please feel free to forward this survey to people in your own > scientific Python communities! > > I will publish the results online in a few weeks. > > Thanks! > Tom > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Mon Mar 2 11:41:35 2015 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Mon, 2 Mar 2015 11:41:35 -0500 Subject: [SciPy-User] Scientific Python Survey In-Reply-To: References: Message-ID: Hi Jason: I was curious if it is possible to use pip with conda ? Myself I am able to install anything with pip or setup.py and never use conda, but for others condo could be a good alternative. But if the people to whom I could recommend conda would need something beyond, will they be able to use pip to install something else? Thanks 2015-03-02 11:33 GMT-05:00 Jason Moore : > These days with conda, it is so easy to have numerous versions of all > these packages. For example, I have conda environments for different > projects and have a range of numpy versions used in each, from 1.6.1 to > 1.9.1. > > What version would you expect us to enter if we use lots of versions in > production scenarios? > > > Jason > moorepants.info > +01 530-601-9791 > > On Mon, Mar 2, 2015 at 7:23 AM, Thomas Robitaille < > thomas.robitaille at gmail.com> wrote: > >> Hi everyone, >> >> If you use scientific Python packages for your research/work, I would >> appreciate if you could take a few minutes to fill out the following >> survey: >> >> http://goo.gl/PXzFAk >> >> The aim of this survey is to find out what versions of Python and >> various scientific Python packages people are using, and how people >> typically install packages, in order to determine how developers can >> better meet the needs of the Scientific Python community (for example, >> a common question is which version of Numpy need to be supported by >> packages). >> >> This is a follow-up to a similar survey which I did back in 2012 and >> which provided very interesting results that you can read about here: >> >> >> http://astrofrog.github.io/blog/2013/01/13/what-python-installations-are-scientists-using >> >> Please feel free to forward this survey to people in your own >> scientific Python communities! >> >> I will publish the results online in a few weeks. >> >> Thanks! >> Tom >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Sasha -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyrille.rossant at gmail.com Mon Mar 2 11:42:42 2015 From: cyrille.rossant at gmail.com (Cyrille Rossant) Date: Mon, 2 Mar 2015 17:42:42 +0100 Subject: [SciPy-User] Scientific Python Survey In-Reply-To: References: Message-ID: Hi, pip works just fine with conda: http://conda.pydata.org/docs/faq.html#installation Cyrille 2015-03-02 17:41 GMT+01:00 Oleksandr Huziy : > Hi Jason: > > I was curious if it is possible to use pip with conda ? > Myself I am able to install anything with pip or setup.py and never use > conda, but for others condo could be a good alternative. But if the people > to whom I could recommend conda would need something beyond, will they be > able to use pip to install something else? > > Thanks > > 2015-03-02 11:33 GMT-05:00 Jason Moore : > >> These days with conda, it is so easy to have numerous versions of all >> these packages. For example, I have conda environments for different >> projects and have a range of numpy versions used in each, from 1.6.1 to >> 1.9.1. >> >> What version would you expect us to enter if we use lots of versions in >> production scenarios? >> >> >> Jason >> moorepants.info >> +01 530-601-9791 >> >> On Mon, Mar 2, 2015 at 7:23 AM, Thomas Robitaille >> wrote: >>> >>> Hi everyone, >>> >>> If you use scientific Python packages for your research/work, I would >>> appreciate if you could take a few minutes to fill out the following >>> survey: >>> >>> http://goo.gl/PXzFAk >>> >>> The aim of this survey is to find out what versions of Python and >>> various scientific Python packages people are using, and how people >>> typically install packages, in order to determine how developers can >>> better meet the needs of the Scientific Python community (for example, >>> a common question is which version of Numpy need to be supported by >>> packages). >>> >>> This is a follow-up to a similar survey which I did back in 2012 and >>> which provided very interesting results that you can read about here: >>> >>> >>> http://astrofrog.github.io/blog/2013/01/13/what-python-installations-are-scientists-using >>> >>> Please feel free to forward this survey to people in your own >>> scientific Python communities! >>> >>> I will publish the results online in a few weeks. >>> >>> Thanks! >>> Tom >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > > -- > Sasha > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From njs at pobox.com Mon Mar 2 17:28:06 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 2 Mar 2015 14:28:06 -0800 Subject: [SciPy-User] Congratulations to Chris Barker... Message-ID: ...on the acceptance of his PEP! PEP 485 adds a math.isclose function to the standard library, encouraging people to do numerically more reasonable floating point comparisons. The PEP: https://www.python.org/dev/peps/pep-0485/ The pronouncement: http://thread.gmane.org/gmane.comp.python.devel/151776/focus=151778 -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Mon Mar 2 17:40:28 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 2 Mar 2015 14:40:28 -0800 Subject: [SciPy-User] Scientific Python Survey In-Reply-To: References: Message-ID: Hi, On Mon, Mar 2, 2015 at 8:33 AM, Jason Moore wrote: > These days with conda, it is so easy to have numerous versions of all these > packages. For example, I have conda environments for different projects and > have a range of numpy versions used in each, from 1.6.1 to 1.9.1. > > What version would you expect us to enter if we use lots of versions in > production scenarios? Yes, likewise with virtualenvs on OSX - I just chose my two most common virtualenvs (up-to-date numpy, scipy, matplotlib) with 2.7.9 and 3.4.2, Cheers, Matthew From shoyer at gmail.com Tue Mar 3 14:01:48 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 3 Mar 2015 11:01:48 -0800 Subject: [SciPy-User] ANN: xray v0.4 released Message-ID: I'm pleased to announce a major release of xray, v0.4. xray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures. Our goal is to provide a pandas-like and pandas-compatible toolkit for analytics on multi-dimensional arrays, rather than the tabular data for which pandas excels. Our approach adopts the Common Data Model for self-describing scientific data in widespread use in the Earth sciences: xray.Dataset is an in-memory representation of a netCDF file. Documentation: http://xray.readthedocs.org/ GitHub: https://github.com/xray/xray Highlights of this release: * Automatic alignment of index labels in arithmetic and when combining arrays or datasets. * Aggregations like mean now skip missing values by default. * Relaxed equality rules in concat and merge for variables with equal value(s) but different shapes. * New drop method for dropping variables or index labels. * Support for reindexing with a fill method like pandas. For more details, read the full release notes: http://xray.readthedocs.org/en/stable/whats-new.html Best, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaakko.luttinen at aalto.fi Thu Mar 5 03:51:23 2015 From: jaakko.luttinen at aalto.fi (Jaakko Luttinen) Date: Thu, 5 Mar 2015 10:51:23 +0200 Subject: [SciPy-User] ANN: BayesPy 0.3 released Message-ID: <54F8190B.2020605@aalto.fi> Dear all, I am pleased to announce that BayesPy 0.3 has been released. BayesPy provides tools for variational Bayesian inference. The user can easily constuct conjugate exponential family models from nodes and run approximate posterior inference. BayesPy aims to be efficient and flexible enough for experts but also accessible for casual users. ----------------------------------------------------------------------- This release adds several state-of-the-art VB features. Below is a list of significant new features in this release: * Gradient-based optimization of the nodes by using either the Euclidean or Riemannian/natural gradient. This enables, for instance, the Riemannian conjugate gradient method. * Collapsed variational inference to improve the speed of learning. * Stochastic variational inference to improve scalability. * Pattern search to improve the speed of learning. * Deterministic annealing to improve robustness against initializations. * Gaussian Markov chains can use input signals. More details about the new features can be found here: http://www.bayespy.org/user_guide/advanced.html ------------------------------------------------------ PyPI: https://pypi.python.org/pypi/bayespy/0.3 Git repository: https://github.com/bayespy/bayespy Documentation: http://www.bayespy.org/ Best regards, Jaakko From rob.clewley at gmail.com Fri Mar 6 11:17:09 2015 From: rob.clewley at gmail.com (Rob Clewley) Date: Fri, 6 Mar 2015 11:17:09 -0500 Subject: [SciPy-User] [ANN] GSOC project for data / geometry visual analysis needs student with technical matplotlib proficiency Message-ID: Hi, One final reminder that I am a mentor for two Google Summer of Code projects that involve extensive matplotlib GUI development for python scientific software. The INCF (incf.org) is sponsoring two GSOC projects that will directly benefit PyDSTool (http://pydstool.sf.net), a math modeling toolbox for science and engineering that some of you will know. One, if not both, of the projects will involve significant technical usage of MPL's 2D GUI widgets but, more excitingly, to build new types of model visualization tools over MPL. MPL has been chosen to maximize platform independence and minimize reliance on additional user installation of third party libraries, and is well suited for the prototyping of application front ends for research code. http://incf.org/gsoc/2015/proposals/#-span--span----nbsp---span---span--span-neuroscience-model-exploration-and-development-tools-for-pydstool--span- High level knowledge of math and the theoretical principles of cell biology are beneficial but not required. Applications can begin right away! Read more about the process here: http://www.google-melange.com/gsoc/homepage/google/gsoc2015 Thanks, Rob From bhmerchant at gmail.com Sun Mar 8 17:40:47 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Sun, 8 Mar 2015 14:40:47 -0700 Subject: [SciPy-User] Package recommendations for *efficiently* making good looking animations, or alternatively, how can I "predraw" matplotlib animation frames? Message-ID: Hi all, Right now, I am at the simulation stage in my modelling. I need to explore possible behaviours, in some parameter sets. In order to do this exploration, it is helpful to visualize the data via animation. I started with matplotlib, but am disappointed with how long it takes to produce an animation using my data. It is also difficult for me to navigate matplotlib's documentation. Anyway, let me explain what I am doing right now. Currently, my custom matplotlib animation (i.e. a class that inherit's from animation.TimedAnimation) class's `_draw_frame` method does this: 1. get line data for timestep `n` 2. update data of each line of interest in the axes with `set_data` (lets say for the sake of an example (`ax_line1.set_data(datax, datay`, where `ax_line1, = ax.plot(...)`) 3. make a list of each line that was updated, and set `self._drawn_artists` to that (so in the example if two lines of interest are there, then we update them `self._drawn_artists = [ax_line1, ax_line2]` Is there a way I can set the data in multiple copies of the line artists before `_draw_frame` is called? Then, all that `_draw_frame` has to do is fetch the predrawn lines, and update the drawn artist list with that. That means that I set the artist data in parallel, if I have the data already, and thus produce an animation faster? I tried doing `copy.deepcopy` of the artists, but got an error telling me that I cannot do so (`TransformationNode` has to be frozen). What other options do I have? At this point, I am totally open to exploring other packages apart from matplotlib, that may be recommended. Kind regards, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulkilgo at gmail.com Sun Mar 8 19:38:13 2015 From: paulkilgo at gmail.com (Paul Kilgo) Date: Sun, 08 Mar 2015 23:38:13 +0000 Subject: [SciPy-User] Package recommendations for *efficiently* making good looking animations, or alternatively, how can I "predraw" matplotlib animation frames? In-Reply-To: References: Message-ID: I can't comment much about matplotlib, but you might take a look at Vispy when exploring your options. It seems more geared toward interactive visualization which I think is what you're going for. It's pretty young and the documentation is mostly for reference, but they have a nice gallery of examples. I found it fairly easy to get up and going just from that. That said, I only used their "lower-level" gloo API which I would say requires a fair amount of knowledge of OpenGL and GLSL. It looks like they are working on a higher-level API (they call it "visuals"). I didn't touch this at all. I bet there is someone subscribed to this list who knows a lot more than me about it. :) On Sun, Mar 8, 2015 at 5:40 PM Brian Merchant wrote: > Hi all, > > Right now, I am at the simulation stage in my modelling. I need to explore > possible behaviours, in some parameter sets. In order to do this > exploration, it is helpful to visualize the data via animation. > > I started with matplotlib, but am disappointed with how long it takes to > produce an animation using my data. It is also difficult for me to navigate > matplotlib's documentation. Anyway, let me explain what I am doing right > now. > > Currently, my custom matplotlib animation (i.e. a class that inherit's > from > > animation.TimedAnimation) class's `_draw_frame` method does this: > > 1. get line data for timestep `n` > 2. update data of each line of interest in the axes with `set_data` (lets > say for the sake of an example (`ax_line1.set_data(datax, datay`, where > `ax_line1, = ax.plot(...)`) > 3. make a list of each line that was updated, and set > `self._drawn_artists` to that (so in the example if two lines of interest > are there, then we update them `self._drawn_artists = [ax_line1, ax_line2]` > > Is there a way I can set the data in multiple copies of the line artists > before `_draw_frame` is called? Then, all that `_draw_frame` has to do is > fetch the predrawn lines, and update the drawn artist list with that. > > That means that I set the artist data in parallel, if I have the data > already, and thus produce an animation faster? I tried doing > `copy.deepcopy` of the artists, but got an error telling me that I cannot > do so (`TransformationNode` has to be frozen). What other options do I have? > > At this point, I am totally open to exploring other packages apart from > matplotlib, that may be recommended. > > Kind regards, > Brian > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Mar 9 00:57:51 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 9 Mar 2015 05:57:51 +0100 Subject: [SciPy-User] Package recommendations for *efficiently* making good looking animations, or alternatively, how can I "predraw" matplotlib animation frames? In-Reply-To: References: Message-ID: On Mon, Mar 9, 2015 at 12:38 AM, Paul Kilgo wrote: > I can't comment much about matplotlib, but you might take a look at Vispy > when exploring your options. It seems more geared toward interactive > visualization which I think is what you're going for. It's pretty young and > the documentation is mostly for reference, but they have a nice gallery of > examples. I found it fairly easy to get up and going just from that. > > That said, I only used their "lower-level" gloo API which I would say > requires a fair amount of knowledge of OpenGL and GLSL. It looks like they > are working on a higher-level API (they call it "visuals"). I didn't touch > this at all. > > I bet there is someone subscribed to this list who knows a lot more than > me about it. :) > Probably. Asking on the Matplotlib list will likely generate more useful suggestions than this list though. Ralf > On Sun, Mar 8, 2015 at 5:40 PM Brian Merchant > wrote: > >> Hi all, >> >> Right now, I am at the simulation stage in my modelling. I need to >> explore possible behaviours, in some parameter sets. In order to do this >> exploration, it is helpful to visualize the data via animation. >> >> I started with matplotlib, but am disappointed with how long it takes to >> produce an animation using my data. It is also difficult for me to navigate >> matplotlib's documentation. Anyway, let me explain what I am doing right >> now. >> >> Currently, my custom matplotlib animation (i.e. a class that inherit's >> from >> >> animation.TimedAnimation) class's `_draw_frame` method does this: >> >> 1. get line data for timestep `n` >> 2. update data of each line of interest in the axes with `set_data` >> (lets say for the sake of an example (`ax_line1.set_data(datax, datay`, >> where `ax_line1, = ax.plot(...)`) >> 3. make a list of each line that was updated, and set >> `self._drawn_artists` to that (so in the example if two lines of interest >> are there, then we update them `self._drawn_artists = [ax_line1, ax_line2]` >> >> Is there a way I can set the data in multiple copies of the line artists >> before `_draw_frame` is called? Then, all that `_draw_frame` has to do is >> fetch the predrawn lines, and update the drawn artist list with that. >> >> That means that I set the artist data in parallel, if I have the data >> already, and thus produce an animation faster? I tried doing >> `copy.deepcopy` of the artists, but got an error telling me that I cannot >> do so (`TransformationNode` has to be frozen). What other options do I have? >> >> At this point, I am totally open to exploring other packages apart from >> matplotlib, that may be recommended. >> >> Kind regards, >> Brian >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyrille.rossant at gmail.com Mon Mar 9 07:34:40 2015 From: cyrille.rossant at gmail.com (Cyrille Rossant) Date: Mon, 9 Mar 2015 12:34:40 +0100 Subject: [SciPy-User] Package recommendations for *efficiently* making good looking animations, or alternatively, how can I "predraw" matplotlib animation frames? In-Reply-To: References: Message-ID: 2015-03-09 0:38 GMT+01:00 Paul Kilgo : > I can't comment much about matplotlib, but you might take a look at Vispy > when exploring your options. It seems more geared toward interactive > visualization which I think is what you're going for. Yes. Feel free to detail your use-case on https://groups.google.com/forum/#!forum/vispy and we'll see whether/how VisPy can help you. Cyrille From bhmerchant at gmail.com Tue Mar 10 01:27:49 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Mon, 9 Mar 2015 22:27:49 -0700 Subject: [SciPy-User] Should one NOT start with Cython first? Message-ID: Recently, this question was asked on stackoverflow:Cython: when using typed memoryviews, are Cython users supposed to implement their own library of ?vector? functions? The current answer suggests that one should not write as much of the program as possible using Cython. Rather, one should start with Python, and then only use Cython for bottlenecks. Is this the right mentality to have when using Cython for scientific projects? Why not start Cython first from the get-go? -------------- next part -------------- An HTML attachment was scrubbed... URL: From moorepants at gmail.com Tue Mar 10 02:41:37 2015 From: moorepants at gmail.com (Jason Moore) Date: Mon, 9 Mar 2015 23:41:37 -0700 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: Message-ID: Most code in a typical high level scientific projects is not CPU intensive, so it is often best to have a mixture of Python and Cython to maximize both development speed and performance. Most people tend to follow the pattern: 1) write in Python 2) profile 3) rewrite slow bits in with Cython. But that doesn't mean you have to. You can write everything in Cython if you want. One disadvantage to that is that you loose the rapid iterative development ability you get with pure Python. If only a small portion of your code is Cython you can usually get by with compiling it only occasionally as opposed to every time you make a change. One potential reason to write it all in Cython is if you want to share binary forms of your code instead of Python source code. Jason moorepants.info +01 530-601-9791 On Mon, Mar 9, 2015 at 10:27 PM, Brian Merchant wrote: > Recently, this question was asked on stackoverflow:Cython: when using > typed memoryviews, are Cython users supposed to implement their own library > of ?vector? functions? > > > The current answer suggests that one should not write as much of the > program as possible using Cython. Rather, one should start with Python, and > then only use Cython for bottlenecks. > > Is this the right mentality to have when using Cython for scientific > projects? Why not start Cython first from the get-go? > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Tue Mar 10 10:10:18 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2015 10:10:18 -0400 Subject: [SciPy-User] soft limiter function Message-ID: I'm looking for a parameterized set of functions, similar to logistic, where a parameter determines the 'sharpness' of the transition from the linear region to the flat region. I need to keep all the same scaling and derivative near the origin - so like a family of logistic functions that would overlay near the origin, but would become increasingly sharp limiters as the parameter was varied. In the limit, would approach the ideal limiter x |x<1| y = { 1 x > 1 -1 x < -1 -- Those who fail to understand recursion are doomed to repeat it From newville at cars.uchicago.edu Tue Mar 10 11:22:37 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Tue, 10 Mar 2015 10:22:37 -0500 Subject: [SciPy-User] soft limiter function In-Reply-To: References: Message-ID: Neal, On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker wrote: > I'm looking for a parameterized set of functions, similar to logistic, > where > a parameter determines the 'sharpness' of the transition from the linear > region to the flat region. I need to keep all the same scaling and > derivative near the origin - so like a family of logistic functions that > would overlay near the origin, but would become increasingly sharp limiters > as the parameter was varied. In the limit, would approach the ideal > limiter > > x |x<1| > y = { 1 x > 1 > -1 x < -1 > This might be too simplistic, but have you considered the "classic" step-like functions (here, going from 0 to 1, but not necessarily at x=+/-1): arctan: y(a) = 0.5 + arctan(a) / pi error fcn: y(a) = 0.5 * (1 + erf(a)) logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) where a = (x-x0)/sigma ? That gives you a knob (sigma) to control the sharpness of the step. --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Tue Mar 10 11:30:03 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2015 11:30:03 -0400 Subject: [SciPy-User] soft limiter function References: Message-ID: Matt Newville wrote: > Neal, > > > On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker wrote: > >> I'm looking for a parameterized set of functions, similar to logistic, >> where >> a parameter determines the 'sharpness' of the transition from the linear >> region to the flat region. I need to keep all the same scaling and >> derivative near the origin - so like a family of logistic functions that >> would overlay near the origin, but would become increasingly sharp >> limiters >> as the parameter was varied. In the limit, would approach the ideal >> limiter >> >> x |x<1| >> y = { 1 x > 1 >> -1 x < -1 >> > > This might be too simplistic, but have you considered the "classic" > step-like functions (here, going from 0 to 1, but not necessarily at > x=+/-1): > > arctan: y(a) = 0.5 + arctan(a) / pi > error fcn: y(a) = 0.5 * (1 + erf(a)) > logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) > > where a = (x-x0)/sigma ? That gives you a knob (sigma) to control the > sharpness of the step. > > --Matt Thanks, but I also need the derivative near the origin to be 1 - cannot change the steepness near the origin -- Those who fail to understand recursion are doomed to repeat it From sturla.molden at gmail.com Tue Mar 10 11:47:09 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Mar 2015 15:47:09 +0000 (UTC) Subject: [SciPy-User] Should one NOT start with Cython first? References: Message-ID: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> "Premature optimization is the root of all evil in computer programming." Here are my two cents: - Most code in scientific programming is not CPU bound. You don't get a faster harddrive or faster network connection from using Cython. This accounts for the majority (often 80 to 90 %) of the code we write. - Even if the code is CPU bound the bottleneck might be in a library like BLAS or LAPACK, for which using Cython might win you nothing. - Even in the cases where Cython helps, it might not be worth the effort: Python code which is fast enough is still fast enough, even if Cython code is faster. - In scientific computing we usually care more about correctness than speed. Speed is never important befre you have code which is actually correct. - Optimizing NumPy code with Numba is often easier than using Cython. - When programming Cython I often find I want to use C or Fortran instead. - Bottlenecks are often obscured. Use a profiler to locate them. - The best way to deal with a bottleneck is very often to use a better algorithm ot datastructure, not move from Python to C or Cython. So should you use Cython from the start? If you are using Cython to optimize for speed, the answer in my opinion is "no". Make a working Python prototype and then use a profiler to find the bottlenecks. If you are using Cython to wrap native code (C, C++, Fortran) then just go ahead and use it from the beginning. Regards, Sturla Brian Merchant wrote: > Recently, this question was asked on stackoverflow:Cython: when using typed > memoryviews, are Cython users supposed to implement their own library of > ?vector? functions? > < href="http://stackoverflow.com/questions/28948175/cython-when-using-typed-memoryviews-are-cython-users-supposed-to-implement-the/28948871#28948871">http://stackoverflow.com/questions/28948175/cython-when-using-typed-memoryviews-are-cython-users-supposed-to-implement-the/28948871#28948871> > > The current answer suggests that one should not write as much of the > program as possible using Cython. Rather, one should start with Python, and > then only use Cython for bottlenecks. > > Is this the right mentality to have when using Cython for scientific > projects? Why not start Cython first from the get-go? > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user From alimanfoo at googlemail.com Tue Mar 10 11:55:07 2015 From: alimanfoo at googlemail.com (Alistair Miles) Date: Tue, 10 Mar 2015 15:55:07 +0000 Subject: [SciPy-User] np.corrcoef ddof is redundant? Message-ID: Hi, I'm trying to calculate correlation coefficients and looking at the np.corrcoef function. It has bias and ddof arguments, however when I try different values of ddof with test data the results are always the same, i.e., changing ddof has no effect. From some back-of-the-envelope algebra I reckon the n/(n-ddof) normalisations should get cancelled out when calculating correlation coefficients from a covariance matrix, and therefore the ddof (and bias) arguments to np.corrcoef are redundant. I'd be very grateful if someone could verify this is true or tell me if I've missed something. Thanks, Alistair -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Web: http://purl.org/net/aliman Email: alimanfoo at gmail.com Tel: +44 (0)1865 287721 -------------- next part -------------- An HTML attachment was scrubbed... URL: From athanastasiou at gmail.com Tue Mar 10 12:14:50 2015 From: athanastasiou at gmail.com (Athanasios Anastasiou) Date: Tue, 10 Mar 2015 16:14:50 +0000 Subject: [SciPy-User] soft limiter function In-Reply-To: References: Message-ID: Hello Neal I was going to go with Matt's suggestion as well but then I started thinking "Why does he want the derivative to be 1 near the origin?" and so I concluded that you are probably trying to do something like audio compression or limiting. The rest of my response is governed by this assumption. If that's wrong, stop reading :D The compressor's curve is the "sum" of two curves. The linear part is 1:1, while the limiting part, past the threshold point becomes 1:2, 1:4 and in general it changes slope depending on how hard you want the compressor to be. So, at its very minimum, you can define a function that accepts two parameters (threshold and compression ratio) and from these it works out the slopes of two curves. In your case it is even easier, you know you want 1:1 up to the threshold, so all you have to do is have one parameter that determines the slope of the limiter. (example: http://en.wikipedia.org/wiki/Dynamic_range_compression#mediaviewer/File:Audio-level-compresion-diagram-01.svg). That's the basic idea. Having said this, such a "kink" will be audible as high frequency noise. So you want to make the transition as soft as possible. You can check the responses of popular diodes used for this, they are all based on some exp() function and the parameters for it are available in the datasheets. The other thing you can be doing is follow the original idea of setting up two lines of different slopes and then interpolating them via a spline. By controlling the "strength" of the node of the spline you can be changing the transition to avoid that "kink". Finally, you can map the curve of an existing compressor that you like, in the steady state, to see the kinds of curves it uses for different settings and then interpolate through that family of curves through a single parameter. Hope this helps. All the best Athanasios Anastasiou P.S. You can still add a low pass filter right after the comp curve to reduce the audible effect of the "kink" much more easily. On Tue, Mar 10, 2015 at 3:30 PM, Neal Becker wrote: > Matt Newville wrote: > > > Neal, > > > > > > On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker > wrote: > > > >> I'm looking for a parameterized set of functions, similar to logistic, > >> where > >> a parameter determines the 'sharpness' of the transition from the linear > >> region to the flat region. I need to keep all the same scaling and > >> derivative near the origin - so like a family of logistic functions that > >> would overlay near the origin, but would become increasingly sharp > >> limiters > >> as the parameter was varied. In the limit, would approach the ideal > >> limiter > >> > >> x |x<1| > >> y = { 1 x > 1 > >> -1 x < -1 > >> > > > > This might be too simplistic, but have you considered the "classic" > > step-like functions (here, going from 0 to 1, but not necessarily at > > x=+/-1): > > > > arctan: y(a) = 0.5 + arctan(a) / pi > > error fcn: y(a) = 0.5 * (1 + erf(a)) > > logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) > > > > where a = (x-x0)/sigma ? That gives you a knob (sigma) to control > the > > sharpness of the step. > > > > --Matt > > Thanks, but I also need the derivative near the origin to be 1 - cannot > change the steepness near the origin > > -- > Those who fail to understand recursion are doomed to repeat it > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Tue Mar 10 12:16:35 2015 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Tue, 10 Mar 2015 12:16:35 -0400 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: Message-ID: It does change for me, though very little.... x = np.random.randn(50) y = x * x * x * x for ddof in range(20): print "ddof = {}; r = {:.20f}".format(ddof, np.corrcoef(x, y, ddof=ddof)[0, 1]) ddof = 0; r = 0.27115960925626320099 ddof = 1; r = 0.27115960925626320099 ddof = 2; r = 0.27115960925626314548 ddof = 3; r = 0.27115960925626320099 ddof = 4; r = 0.27115960925626320099 ddof = 5; r = 0.27115960925626314548 ddof = 6; r = 0.27115960925626320099 ddof = 7; r = 0.27115960925626320099 ddof = 8; r = 0.27115960925626320099 ddof = 9; r = 0.27115960925626320099 ddof = 10; r = 0.27115960925626314548 ddof = 11; r = 0.27115960925626320099 ddof = 12; r = 0.27115960925626320099 ddof = 13; r = 0.27115960925626320099 ddof = 14; r = 0.27115960925626314548 ddof = 15; r = 0.27115960925626314548 ddof = 16; r = 0.27115960925626314548 ddof = 17; r = 0.27115960925626320099 ddof = 18; r = 0.27115960925626320099 ddof = 19; r = 0.27115960925626320099 Cheers 2015-03-10 11:55 GMT-04:00 Alistair Miles : > Hi, > > I'm trying to calculate correlation coefficients and looking at the > np.corrcoef function. It has bias and ddof arguments, however when I try > different values of ddof with test data the results are always the same, > i.e., changing ddof has no effect. From some back-of-the-envelope algebra I > reckon the n/(n-ddof) normalisations should get cancelled out when > calculating correlation coefficients from a covariance matrix, and > therefore the ddof (and bias) arguments to np.corrcoef are redundant. > > I'd be very grateful if someone could verify this is true or tell me if > I've missed something. > > Thanks, > Alistair > -- > Alistair Miles > Head of Epidemiological Informatics > Centre for Genomics and Global Health > The Wellcome Trust Centre for Human Genetics > Roosevelt Drive > Oxford > OX3 7BN > United Kingdom > Web: http://purl.org/net/aliman > Email: alimanfoo at gmail.com > Tel: +44 (0)1865 287721 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Sasha -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Mar 10 12:27:40 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Mar 2015 16:27:40 +0000 (UTC) Subject: [SciPy-User] np.corrcoef ddof is redundant? References: Message-ID: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Alistair Miles wrote: > I'm trying to calculate correlation coefficients and looking at the > np.corrcoef function. It has bias and ddof arguments, however when I try > different values of ddof with test data the results are always the same, > i.e., changing ddof has no effect. From some back-of-the-envelope algebra I > reckon the n/(n-ddof) normalisations should get cancelled out when > calculating correlation coefficients from a covariance matrix, and > therefore the ddof (and bias) arguments to np.corrcoef are redundant. > > I'd be very grateful if someone could verify this is true or tell me if > I've missed something. You are right. It should cancel out or np.corrcoef would be wrong. The sample size does not go into the Pearson product-moment correlation. Sturla > Thanks, > Alistair > > -- > Alistair Miles > Head of Epidemiological Informatics > Centre for Genomics and Global Health <http://cggh.org> > The Wellcome Trust Centre for Human Genetics > Roosevelt Drive > Oxford > OX3 7BN > United Kingdom > Web: http://purl.org/net/aliman > Email: alimanfoo at gmail.com > Tel: +44 (0)1865 287721 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user From sturla.molden at gmail.com Tue Mar 10 12:34:34 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Mar 2015 16:34:34 +0000 (UTC) Subject: [SciPy-User] np.corrcoef ddof is redundant? References: Message-ID: <1461254283447697856.341495sturla.molden-gmail.com@news.gmane.org> Oleksandr Huziy wrote: > It does change for me, though very little.... Probably rounding error. From sturla.molden at gmail.com Tue Mar 10 12:30:07 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Mar 2015 16:30:07 +0000 (UTC) Subject: [SciPy-User] Should one NOT start with Cython first? References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> Message-ID: <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Sturla Molden wrote: > - Even in the cases where Cython helps, it might not be worth the effort: > Python code which is fast enough is still fast enough, even if Cython code > is faster. > > - In scientific computing we usually care more about correctness than > speed. Speed is never important befre you have code which is actually > correct "Fast" code which is incorrect is not better than "slow" code which is correct, on the countrary. We prefer "slow and correct" code to "fast but incorrect" code. As a consequence of this, you will often be better off spending your time writing rigorous test cases than optimizing for speed. And your attitude when writing tests should be to do your best to prove that your code does not work. If you can prove your code wrong it is a far more important investment of your own resources than spening time on micro-optimizations. Another thing which is special for scientific computing is a rather peculiar meaning of "fast" and "slow": It is not the time to market a product, nor the CPU time. Rather it is the time you spend working on the project, from its planning until your paper is published. It might not be that C is "fast" at all in this context. It is better to spend a short time writing code and have the computer work for a week, rather than spend a week writing code and have the computer work for a short time. This is because a lot of the code we write in science is customized for a particular project, and will only be used once. What can be working against you here is your own pride. It is often against our pride to just accept atrociously slow code and use it, rather than try to improve on it. But put your pride away and focus on your real goal. Sturla From ndbecker2 at gmail.com Tue Mar 10 13:14:19 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2015 13:14:19 -0400 Subject: [SciPy-User] soft limiter function References: Message-ID: Athanasios Anastasiou wrote: > Hello Neal > > I was going to go with Matt's suggestion as well but then I started > thinking "Why does he want the derivative to be 1 near the origin?" and so > I concluded that you are probably trying to do something like audio > compression or limiting. The rest of my response is governed by this > assumption. If that's wrong, stop reading :D > > The compressor's curve is the "sum" of two curves. The linear part is 1:1, > while the limiting part, past the threshold point becomes 1:2, 1:4 and in > general it changes slope depending on how hard you want the compressor to > be. > > So, at its very minimum, you can define a function that accepts two > parameters (threshold and compression ratio) and from these it works out > the slopes of two curves. In your case it is even easier, you know you > want 1:1 up to the threshold, so all you have to do is have one parameter > that determines the slope of the limiter. (example: > http://en.wikipedia.org/wiki/Dynamic_range_compression#mediaviewer/File:Audio-level-compresion-diagram-01.svg). > That's the basic idea. > > Having said this, such a "kink" will be audible as high frequency noise. > So you want to make the transition as soft as possible. You can check the > responses of popular diodes used for this, they are all based on some > exp() function and the parameters for it are available in the datasheets. > > The other thing you can be doing is follow the original idea of setting up > two lines of different slopes and then interpolating them via a spline. By > controlling the "strength" of the node of the spline you can be changing > the transition to avoid that "kink". > > Finally, you can map the curve of an existing compressor that you like, in > the steady state, to see the kinds of curves it uses for different > settings and then interpolate through that family of curves through a > single parameter. > > Hope this helps. > > All the best > Athanasios Anastasiou > > P.S. You can still add a low pass filter right after the comp curve to > reduce the audible effect of the "kink" much more easily. > > > > > > > > > On Tue, Mar 10, 2015 at 3:30 PM, Neal Becker wrote: > >> Matt Newville wrote: >> >> > Neal, >> > >> > >> > On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker >> wrote: >> > >> >> I'm looking for a parameterized set of functions, similar to logistic, >> >> where >> >> a parameter determines the 'sharpness' of the transition from the >> >> linear >> >> region to the flat region. I need to keep all the same scaling and >> >> derivative near the origin - so like a family of logistic functions >> >> that would overlay near the origin, but would become increasingly >> >> sharp limiters >> >> as the parameter was varied. In the limit, would approach the ideal >> >> limiter >> >> >> >> x |x<1| >> >> y = { 1 x > 1 >> >> -1 x < -1 >> >> >> > >> > This might be too simplistic, but have you considered the "classic" >> > step-like functions (here, going from 0 to 1, but not necessarily at >> > x=+/-1): >> > >> > arctan: y(a) = 0.5 + arctan(a) / pi >> > error fcn: y(a) = 0.5 * (1 + erf(a)) >> > logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) >> > >> > where a = (x-x0)/sigma ? That gives you a knob (sigma) to control >> the >> > sharpness of the step. >> > >> > --Matt >> >> Thanks, but I also need the derivative near the origin to be 1 - cannot >> change the steepness near the origin >> The rapp amplifier model looks like it will do what I need (I'm modeling amplifiers, nothing to do with audio) http://www.ieee802.org/16/tg1/phy/pres/802161pp-00_15.pdf From matthieu.brucher at gmail.com Tue Mar 10 13:21:28 2015 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 10 Mar 2015 17:21:28 +0000 Subject: [SciPy-User] soft limiter function In-Reply-To: References: Message-ID: You can still modify the functions to have such derivative at the origin (this is often used to model amps in audio processing). Just use a scaling factor. Cheers, 2015-03-10 15:30 GMT+00:00 Neal Becker : > Matt Newville wrote: > >> Neal, >> >> >> On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker wrote: >> >>> I'm looking for a parameterized set of functions, similar to logistic, >>> where >>> a parameter determines the 'sharpness' of the transition from the linear >>> region to the flat region. I need to keep all the same scaling and >>> derivative near the origin - so like a family of logistic functions that >>> would overlay near the origin, but would become increasingly sharp >>> limiters >>> as the parameter was varied. In the limit, would approach the ideal >>> limiter >>> >>> x |x<1| >>> y = { 1 x > 1 >>> -1 x < -1 >>> >> >> This might be too simplistic, but have you considered the "classic" >> step-like functions (here, going from 0 to 1, but not necessarily at >> x=+/-1): >> >> arctan: y(a) = 0.5 + arctan(a) / pi >> error fcn: y(a) = 0.5 * (1 + erf(a)) >> logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) >> >> where a = (x-x0)/sigma ? That gives you a knob (sigma) to control the >> sharpness of the step. >> >> --Matt > > Thanks, but I also need the derivative near the origin to be 1 - cannot > change the steepness near the origin > > -- > Those who fail to understand recursion are doomed to repeat it > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ From athanastasiou at gmail.com Tue Mar 10 14:59:46 2015 From: athanastasiou at gmail.com (Athanasios Anastasiou) Date: Tue, 10 Mar 2015 18:59:46 +0000 Subject: [SciPy-User] soft limiter function In-Reply-To: References: Message-ID: " I'm modeling amplifiers, nothing to do with audio" This is so "wrong" in so many levels!!! :D I am just joking about the "wrong" but the clipping characteristics of instrument amplifiers, their "sound signature", is exactly why one would like to produce such a model. See here for example: http://en.m.wikipedia.org/wiki/Amplifier_modeling Anyway, all the best for your project. Athanasios Anastasiou On 10 Mar 2015 17:21, "Matthieu Brucher" wrote: > You can still modify the functions to have such derivative at the > origin (this is often used to model amps in audio processing). Just > use a scaling factor. > > Cheers, > > 2015-03-10 15:30 GMT+00:00 Neal Becker : > > Matt Newville wrote: > > > >> Neal, > >> > >> > >> On Tue, Mar 10, 2015 at 9:10 AM, Neal Becker > wrote: > >> > >>> I'm looking for a parameterized set of functions, similar to logistic, > >>> where > >>> a parameter determines the 'sharpness' of the transition from the > linear > >>> region to the flat region. I need to keep all the same scaling and > >>> derivative near the origin - so like a family of logistic functions > that > >>> would overlay near the origin, but would become increasingly sharp > >>> limiters > >>> as the parameter was varied. In the limit, would approach the ideal > >>> limiter > >>> > >>> x |x<1| > >>> y = { 1 x > 1 > >>> -1 x < -1 > >>> > >> > >> This might be too simplistic, but have you considered the "classic" > >> step-like functions (here, going from 0 to 1, but not necessarily at > >> x=+/-1): > >> > >> arctan: y(a) = 0.5 + arctan(a) / pi > >> error fcn: y(a) = 0.5 * (1 + erf(a)) > >> logistic: y(a) = 1.0 - 1.0 /(1.0 + exp(a)) > >> > >> where a = (x-x0)/sigma ? That gives you a knob (sigma) to control > the > >> sharpness of the step. > >> > >> --Matt > > > > Thanks, but I also need the derivative near the origin to be 1 - cannot > > change the steepness near the origin > > > > -- > > Those who fail to understand recursion are doomed to repeat it > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > Music band: http://liliejay.com/ > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Mar 10 16:12:25 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 10 Mar 2015 13:12:25 -0700 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Tue, Mar 10, 2015 at 9:27 AM, Sturla Molden wrote: > Alistair Miles wrote: > >> I'm trying to calculate correlation coefficients and looking at the >> np.corrcoef function. It has bias and ddof arguments, however when I try >> different values of ddof with test data the results are always the same, >> i.e., changing ddof has no effect. From some back-of-the-envelope algebra I >> reckon the n/(n-ddof) normalisations should get cancelled out when >> calculating correlation coefficients from a covariance matrix, and >> therefore the ddof (and bias) arguments to np.corrcoef are redundant. >> >> I'd be very grateful if someone could verify this is true or tell me if >> I've missed something. > > You are right. It should cancel out or np.corrcoef would be wrong. The > sample size does not go into the Pearson product-moment correlation. Oh dear - that's embarrassing. https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient I guess we should deprecate the 'bias' and 'ddof' input arguments asap. Cheers, Matthew From memilanuk at gmail.com Tue Mar 10 18:50:50 2015 From: memilanuk at gmail.com (memilanuk) Date: Tue, 10 Mar 2015 15:50:50 -0700 Subject: [SciPy-User] Experimental Design with Python Message-ID: Hello there, Is anyone on the list involved with experimental design, aka 'design of experiments', aka 'DoE'? https://en.wikipedia.org/wiki/Design_of_experiments http://www.itl.nist.gov/div898/handbook/pri/pri.htm I'm having a very hard time finding much if anything about DoE involving Python, which I kind of thought would be a natural fit given the utility of scipy, etc.. R has an entire task view page on CRAN (http://cran.r-project.org/web/views/ExperimentalDesign.html). Is there some particular reason Python would be unsuitable for this sort of endeavor, or something to explain the utter lack of interest? I've found a few packages (pyDOE, pyDesign, experimentator) but nothing like the hype surrounding 'machine learning'. I'm looking at using it to optimize something (for a hobby), and the commercial stats packages such as Minitab, JMP, Stat-ease, etc. are pretty much out-of-reach. At this point I'm leaning towards using R; though I'm more comfortable overall with Python, I'm not really at a point to where I feel comfortable 'rolling my own' for this sort of thing just yet. Thanks, Monte -- Shiny! Let's be bad guys. Reach me @ memilanuk (at) gmail dot com From alan.isaac at gmail.com Tue Mar 10 19:07:35 2015 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 10 Mar 2015 19:07:35 -0400 Subject: [SciPy-User] Experimental Design with Python In-Reply-To: References: Message-ID: <54FF7937.6080408@gmail.com> On 3/10/2015 6:50 PM, memilanuk wrote: > I'm having a very hard time finding much if anything about DoE involving > Python, which I kind of thought would be a natural fit given the utility > of scipy, etc. Look at statsmodels and in particular model specification using patsy. There is also many LHS variants in Python floating around (including in pyDOE). fwiw, Alan Isaac From alimanfoo at googlemail.com Tue Mar 10 19:09:24 2015 From: alimanfoo at googlemail.com (Alistair Miles) Date: Tue, 10 Mar 2015 23:09:24 +0000 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: Thanks for the responses, glad to know I'm not going crazy. Cheers, Alistair. On Tuesday, 10 March 2015, Matthew Brett wrote: > Hi, > > On Tue, Mar 10, 2015 at 9:27 AM, Sturla Molden > wrote: > > Alistair Miles > wrote: > > > >> I'm trying to calculate correlation coefficients and looking at the > >> np.corrcoef function. It has bias and ddof arguments, however when I try > >> different values of ddof with test data the results are always the same, > >> i.e., changing ddof has no effect. From some back-of-the-envelope > algebra I > >> reckon the n/(n-ddof) normalisations should get cancelled out when > >> calculating correlation coefficients from a covariance matrix, and > >> therefore the ddof (and bias) arguments to np.corrcoef are redundant. > >> > >> I'd be very grateful if someone could verify this is true or tell me if > >> I've missed something. > > > > You are right. It should cancel out or np.corrcoef would be wrong. The > > sample size does not go into the Pearson product-moment correlation. > > Oh dear - that's embarrassing. > > > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient > > I guess we should deprecate the 'bias' and 'ddof' input arguments asap. > > Cheers, > > Matthew > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Web: http://purl.org/net/aliman Email: alimanfoo at gmail.com Tel: +44 (0)1865 287721 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Mar 10 22:21:34 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 11 Mar 2015 03:21:34 +0100 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: On 10/03/15 21:12, Matthew Brett wrote: > Oh dear - that's embarrassing. > > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient > > I guess we should deprecate the 'bias' and 'ddof' input arguments asap. It is an unfortunate consequence of implementing np.corrcoef on top of np.cov. np.corrcoef should not be computed with np.cov because it just adds additional rounding error to the result. https://github.com/numpy/numpy/blob/32e23a1d52a05d3a56f693010eaf8d96826db75f/numpy/lib/function_base.py Sturla From matthew.brett at gmail.com Tue Mar 10 22:56:19 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 10 Mar 2015 19:56:19 -0700 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Mar 10, 2015 at 7:21 PM, Sturla Molden wrote: > On 10/03/15 21:12, Matthew Brett wrote: > >> Oh dear - that's embarrassing. >> >> https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient >> >> I guess we should deprecate the 'bias' and 'ddof' input arguments asap. > > It is an unfortunate consequence of implementing np.corrcoef on top of > np.cov. Except we should have realized that bias / ddof cancels and therefore should not have implemented the bias / ddof input arguments (or passed them to cov in the function). > np.corrcoef should not be computed with np.cov because it just adds > additional rounding error to the result. What algorithm do you think we should use to minimize rounding error? Cheers, Matthew From sturla.molden at gmail.com Tue Mar 10 23:58:11 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 11 Mar 2015 04:58:11 +0100 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: On 11/03/15 03:56, Matthew Brett wrote: >> np.corrcoef should not be computed with np.cov because it just adds >> additional rounding error to the result. > > What algorithm do you think we should use to minimize rounding error? I was not actually thinking about that. I just thought we could reuse some of the code from np.cov to avoid the redundant division and multiplications. But since you asked, to minimize rounding error there is a two-pass method which can be used for both cov and corrcoef. Cf. this Matlab code: http://home.online.no/~pjacklam/matlab/software/util/statutil/covmat.m This would be very easy to use in NumPy. Another method which is less known is to use the SVD. It can also be used to compute the corrcoef. Here for real values and rowvar=False: def cov(X, ddof): nx,p = X.shape mean = X.mean(axis=0) CX = X - mean[None,:] u,s,pc = np.linalg.svd(CX/np.sqrt(nx-ddof), full_matrices=False) s2 = s**2 tmp = np.eye(p) * s2[:,None] return np.dot(pc.T,np.dot(tmp,pc)) Sturla From memilanuk at gmail.com Wed Mar 11 10:00:07 2015 From: memilanuk at gmail.com (Monte Milanuk) Date: Wed, 11 Mar 2015 14:00:07 +0000 (UTC) Subject: [SciPy-User] Experimental Design with Python References: <54FF7937.6080408@gmail.com> Message-ID: Alan G Isaac gmail.com> writes: > > Look at statsmodels and in particular model specification using patsy. > There is also many LHS variants in Python floating around (including in pyDOE). > Cool. I hadn't come across patsy before; definitely looks interesting with the R-like syntax. Another one I see mentioned occasionally is openMDAO. Is there some place that would make sense as a central location for info like this, or a tutorial on setting up and analyzing experiments from start to finish using various python tools? From josef.pktd at gmail.com Wed Mar 11 11:55:51 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Mar 2015 11:55:51 -0400 Subject: [SciPy-User] Experimental Design with Python In-Reply-To: References: <54FF7937.6080408@gmail.com> Message-ID: On Wed, Mar 11, 2015 at 10:00 AM, Monte Milanuk wrote: > Alan G Isaac gmail.com> writes: > > > > > Look at statsmodels and in particular model specification > using patsy. > > There is also many LHS variants in Python floating around > (including in pyDOE). > > > > Cool. I hadn't come across patsy before; definitely looks > interesting with the R-like syntax. > > Another one I see mentioned occasionally is openMDAO. > Thanks for the pointer, I haven't seen this before. > > Is there some place that would make sense as a central location > for info like this, or a tutorial on setting up and analyzing > experiments from start to finish using various python tools? > statsmodels would be a good location for DOE related tools and information. There is nothing specific yet in there (except some power and sample size calculations), and I don't have an overview for what belongs to DOE and would be useful. (It's on the long term wishlist.) A notebook or a documentation rst page would make a good start. AFAIK, there is no other general purpose statistics package besides statsmodels, everything else is topic or domain specific. Josef > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Mar 11 13:04:58 2015 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 11 Mar 2015 10:04:58 -0700 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation Message-ID: Hi to the list, I'm seeking the advise of the scientific python community to solve the following fitting problem. Both suggestions on the methodology and on particular software packages are appreciated. I often encounter the need to fit a sample containing a (dominant) exponentially-distributed sub-population. Mostly the non-exponential samples (from an unknown distribution) are distributed close to the origin of the exponential distribution, therefore a simple approach I used so far is selecting all the samples higher than a threshold and fitting the exponential "tail" with MLE. The problem is that the choice of the threshold is somewhat arbitrary and moreover there can be a small set of outlier on the extreme right-side of the distribution that would bias the MLE fit. To improve the accuracy, I'm thinking of using (if necessary implementing) some kind of robust fitting procedure. For example using a scheme in which the outlier are identified by putting a threshold on the residual and then this threshold is optimized using some "goodness of fit" cost function. If this approach reasonable? I am surely not the first to tackle this problem, so I would appreciated some suggestion and specific pointers to help me getting started. Thank you, Antonio -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.gullikson at gmail.com Wed Mar 11 14:08:34 2015 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Wed, 11 Mar 2015 13:08:34 -0500 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: Antonio, The statsmodels package has a robust linear model module that I have used before. You will have to transform your data to be linear first by taking the log of the y-axis. http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/robust_models_0.html Kevin Gullikson On Wed, Mar 11, 2015 at 12:04 PM, Antonino Ingargiola wrote: > Hi to the list, > > I'm seeking the advise of the scientific python community to solve the > following fitting problem. Both suggestions on the methodology and on > particular software packages are appreciated. > > I often encounter the need to fit a sample containing a (dominant) > exponentially-distributed sub-population. Mostly the non-exponential > samples (from an unknown distribution) are distributed close to the origin > of the exponential distribution, therefore a simple approach I used so far > is selecting all the samples higher than a threshold and fitting the > exponential "tail" with MLE. > > The problem is that the choice of the threshold is somewhat arbitrary and > moreover there can be a small set of outlier on the extreme right-side of > the distribution that would bias the MLE fit. > > To improve the accuracy, I'm thinking of using (if necessary implementing) > some kind of robust fitting procedure. For example using a scheme in which > the outlier are identified by putting a threshold on the residual and then > this threshold is optimized using some "goodness of fit" cost function. If > this approach reasonable? > > I am surely not the first to tackle this problem, so I would appreciated > some suggestion and specific pointers to help me getting started. > > Thank you, > Antonio > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From twocolorflipflop at gmail.com Wed Mar 11 19:14:52 2015 From: twocolorflipflop at gmail.com (c) Date: Thu, 12 Mar 2015 01:14:52 +0200 Subject: [SciPy-User] fitting discrete probability distributions to data Message-ID: hi, i have some data: A) a 1d array (dimensions 1x50), made by summing the columns of a 2d array (dimensions ~20k x 50). B) a 1D array that is just a particular row of that 2d array i need to fit a sum of 2 negative binomial distributions to A), and to fit a single negative binomial distrib. to B). i have spent a while now reading the documentation for numpy.stats and the statsmodel package and various stack overflow posts, etc.. but i do not yet understand how to go about fitting a discrete probability distribution to a vector of data. specific subquestions: - do i need to load data in as a pandas df? an ndarray? does it not matter? - i understand endog and exog in the context of the examples given in the docs (where you have one column that you want to use to predict some other column) but not what they should be in the case where i basically am trying to fit a curve to the normalized histogram of my data - if someone can explain how to fit with statsmodels' "Negative Binomial ( http://statsmodels.sourceforge.net/devel/generated/statsmodels.discrete.discrete_model.NegativeBinomial.html#statsmodels.discrete.discrete_model.NegativeBinomial) that would be a good start. but i do also need to know how to fit to a sum of two of these, or possibly a sum of two other discrete distributions - is the patsy formula syntax relevant here? i have never used R and could not find an example of the "R-like" syntax that is similar enough to my use case to parse how it works - honestly i don't know what i'm doing, please help! if these questions reveal grave ignorance, or are not directly relevant enough to scipy for this mailing list, i apologize and thanks for bearing with me. i barely know how to flip a coin, this stuff is new to me. thanks a lot c -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Mar 11 19:36:04 2015 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 11 Mar 2015 16:36:04 -0700 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: Hi Kevin, If I apply the log transform to the sample to linearize the models, what is the correct way to weight the residuals? Without weighting residual close to the tail will be amplified and bias the fit. Antonio On Wed, Mar 11, 2015 at 11:08 AM, Kevin Gullikson wrote: > Antonio, > > The statsmodels package has a robust linear model module that I have used > before. You will have to transform your data to be linear first by taking > the log of the y-axis. > > > http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/robust_models_0.html > > > Kevin Gullikson > > On Wed, Mar 11, 2015 at 12:04 PM, Antonino Ingargiola > wrote: > >> Hi to the list, >> >> I'm seeking the advise of the scientific python community to solve the >> following fitting problem. Both suggestions on the methodology and on >> particular software packages are appreciated. >> >> I often encounter the need to fit a sample containing a (dominant) >> exponentially-distributed sub-population. Mostly the non-exponential >> samples (from an unknown distribution) are distributed close to the origin >> of the exponential distribution, therefore a simple approach I used so far >> is selecting all the samples higher than a threshold and fitting the >> exponential "tail" with MLE. >> >> The problem is that the choice of the threshold is somewhat arbitrary and >> moreover there can be a small set of outlier on the extreme right-side of >> the distribution that would bias the MLE fit. >> >> To improve the accuracy, I'm thinking of using (if necessary >> implementing) some kind of robust fitting procedure. For example using a >> scheme in which the outlier are identified by putting a threshold on the >> residual and then this threshold is optimized using some "goodness of fit" >> cost function. If this approach reasonable? >> >> I am surely not the first to tackle this problem, so I would appreciated >> some suggestion and specific pointers to help me getting started. >> >> Thank you, >> Antonio >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Mar 11 19:42:51 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Mar 2015 19:42:51 -0400 Subject: [SciPy-User] fitting discrete probability distributions to data In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 7:14 PM, c wrote: > hi, > > i have some data: > > A) a 1d array (dimensions 1x50), made by summing the columns of a 2d array > (dimensions ~20k x 50). > > B) a 1D array that is just a particular row of that 2d array > > i need to fit a sum of 2 negative binomial distributions to A), and to fit > a single negative binomial distrib. to B). > > i have spent a while now reading the documentation for numpy.stats and the > statsmodel package and various stack overflow posts, etc.. but i do not yet > understand how to go about fitting a discrete probability distribution to a > vector of data. > Do you have the data in the form of histograms (counts) or the original data ? statsmodels can only estimate based on the original data which is assumed to consist of observations drawn from a Negative Binomial distribution. Fitting histogram and fitting mixtures of distributions is not supported "out of the box", and would require some custom models. If you just want to fit a distribution to a histogram or discrete counts, then using curve_fit or leastsq is one possibility. Josef > > specific subquestions: > > - do i need to load data in as a pandas df? an ndarray? does it not matter? > > - i understand endog and exog in the context of the examples given in the > docs (where you have one column that you want to use to predict some other > column) but not what they should be in the case where i basically am trying > to fit a curve to the normalized histogram of my data > > - if someone can explain how to fit with statsmodels' "Negative Binomial ( > http://statsmodels.sourceforge.net/devel/generated/statsmodels.discrete.discrete_model.NegativeBinomial.html#statsmodels.discrete.discrete_model.NegativeBinomial) > that would be a good start. but i do also need to know how to fit to a sum > of two of these, or possibly a sum of two other discrete distributions > > - is the patsy formula syntax relevant here? i have never used R and could > not find an example of the "R-like" syntax that is similar enough to my use > case to parse how it works > > - honestly i don't know what i'm doing, please help! > > if these questions reveal grave ignorance, or are not directly relevant > enough to scipy for this mailing list, i apologize and thanks for bearing > with me. i barely know how to flip a coin, this stuff is new to me. > > thanks a lot > c > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From twocolorflipflop at gmail.com Wed Mar 11 19:49:09 2015 From: twocolorflipflop at gmail.com (c) Date: Thu, 12 Mar 2015 01:49:09 +0200 Subject: [SciPy-User] fitting discrete probability distributions to data In-Reply-To: References: Message-ID: yup, i have the original data On Thu, Mar 12, 2015 at 1:42 AM, wrote: > > > On Wed, Mar 11, 2015 at 7:14 PM, c wrote: > >> hi, >> >> i have some data: >> >> A) a 1d array (dimensions 1x50), made by summing the columns of a 2d >> array (dimensions ~20k x 50). >> >> B) a 1D array that is just a particular row of that 2d array >> >> i need to fit a sum of 2 negative binomial distributions to A), and to >> fit a single negative binomial distrib. to B). >> >> i have spent a while now reading the documentation for numpy.stats and >> the statsmodel package and various stack overflow posts, etc.. but i do not >> yet understand how to go about fitting a discrete probability distribution >> to a vector of data. >> > > Do you have the data in the form of histograms (counts) or the original > data ? > > statsmodels can only estimate based on the original data which is assumed > to consist of observations drawn from a Negative Binomial distribution. > Fitting histogram and fitting mixtures of distributions is not supported > "out of the box", and would require some custom models. > > If you just want to fit a distribution to a histogram or discrete counts, > then using curve_fit or leastsq is one possibility. > > Josef > > > >> >> specific subquestions: >> >> - do i need to load data in as a pandas df? an ndarray? does it not >> matter? >> >> - i understand endog and exog in the context of the examples given in the >> docs (where you have one column that you want to use to predict some other >> column) but not what they should be in the case where i basically am trying >> to fit a curve to the normalized histogram of my data >> >> - if someone can explain how to fit with statsmodels' "Negative Binomial ( >> http://statsmodels.sourceforge.net/devel/generated/statsmodels.discrete.discrete_model.NegativeBinomial.html#statsmodels.discrete.discrete_model.NegativeBinomial) >> that would be a good start. but i do also need to know how to fit to a sum >> of two of these, or possibly a sum of two other discrete distributions >> >> - is the patsy formula syntax relevant here? i have never used R and >> could not find an example of the "R-like" syntax that is similar enough to >> my use case to parse how it works >> >> - honestly i don't know what i'm doing, please help! >> >> if these questions reveal grave ignorance, or are not directly relevant >> enough to scipy for this mailing list, i apologize and thanks for bearing >> with me. i barely know how to flip a coin, this stuff is new to me. >> >> thanks a lot >> c >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Mar 11 20:07:34 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Mar 2015 20:07:34 -0400 Subject: [SciPy-User] fitting discrete probability distributions to data In-Reply-To: References: Message-ID: (please comment inline or post at the bottom in scipy related mailing lists) On Wed, Mar 11, 2015 at 7:49 PM, c wrote: > yup, i have the original data > To estimate a single Negative Binomial, you can use statsmodels.NegativeBinomial and regress on a constant. endog is your negative binomial data, exog = np.ones(len(data)) and result = sm.NegativeBinomial(data, exog).fit() result.params has the estimated parameters but they are in a mean-dispersion parameterization used for regression, not in the "standard" parameterization of a Negative Binomial distribution. There is somewhere (!?) a helper function to transform the params into the standard form as used for example by scipy.stats.negbin estimating the mixture of two or more NegativeBinomial distributions takes a bit of work. Josef > > On Thu, Mar 12, 2015 at 1:42 AM, wrote: > >> >> >> On Wed, Mar 11, 2015 at 7:14 PM, c wrote: >> >>> hi, >>> >>> i have some data: >>> >>> A) a 1d array (dimensions 1x50), made by summing the columns of a 2d >>> array (dimensions ~20k x 50). >>> >>> B) a 1D array that is just a particular row of that 2d array >>> >>> i need to fit a sum of 2 negative binomial distributions to A), and to >>> fit a single negative binomial distrib. to B). >>> >>> i have spent a while now reading the documentation for numpy.stats and >>> the statsmodel package and various stack overflow posts, etc.. but i do not >>> yet understand how to go about fitting a discrete probability distribution >>> to a vector of data. >>> >> >> Do you have the data in the form of histograms (counts) or the original >> data ? >> >> statsmodels can only estimate based on the original data which is assumed >> to consist of observations drawn from a Negative Binomial distribution. >> Fitting histogram and fitting mixtures of distributions is not supported >> "out of the box", and would require some custom models. >> >> If you just want to fit a distribution to a histogram or discrete counts, >> then using curve_fit or leastsq is one possibility. >> >> Josef >> >> >> >>> >>> specific subquestions: >>> >>> - do i need to load data in as a pandas df? an ndarray? does it not >>> matter? >>> >>> - i understand endog and exog in the context of the examples given in >>> the docs (where you have one column that you want to use to predict some >>> other column) but not what they should be in the case where i basically am >>> trying to fit a curve to the normalized histogram of my data >>> >>> - if someone can explain how to fit with statsmodels' "Negative Binomial >>> ( >>> http://statsmodels.sourceforge.net/devel/generated/statsmodels.discrete.discrete_model.NegativeBinomial.html#statsmodels.discrete.discrete_model.NegativeBinomial) >>> that would be a good start. but i do also need to know how to fit to a sum >>> of two of these, or possibly a sum of two other discrete distributions >>> >>> - is the patsy formula syntax relevant here? i have never used R and >>> could not find an example of the "R-like" syntax that is similar enough to >>> my use case to parse how it works >>> >>> - honestly i don't know what i'm doing, please help! >>> >>> if these questions reveal grave ignorance, or are not directly relevant >>> enough to scipy for this mailing list, i apologize and thanks for bearing >>> with me. i barely know how to flip a coin, this stuff is new to me. >>> >>> thanks a lot >>> c >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Mar 11 20:15:19 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Mar 2015 20:15:19 -0400 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 7:36 PM, Antonino Ingargiola wrote: > Hi Kevin, > > If I apply the log transform to the sample to linearize the models, what > is the correct way to weight the residuals? Without weighting residual > close to the tail will be amplified and bias the fit. > In RLM the robust linear model the weights are automatically chosen to downweight extreme residuals. The weighting scheme depends on the "norm" which defines the shape of the objective and of the weight function. RLM produces an unbiased estimator of the mean or mean function for symmetric distribution and is calibrated for the normal distribution. I don't know how well this is approximated by the log of an exponentially distributed variable, but it won't exactly satisfy the assumptions. There should be a more direct way of estimating the parameter for the exponential distribution in a robust way, but I never tried. (one idea would be to estimate a trimmed mean and use the estimated distribution to correct for the trimming. scipy.stats.distributions have an `expect` method that can be used to calculate the mean of a trimmed distribution, i.e. conditional on lower and upper bounds) What's your sample size? (for very large sample sizes one approach that is sometimes used, is to fit a distribution to the central part of a histogram) Josef > > Antonio > > On Wed, Mar 11, 2015 at 11:08 AM, Kevin Gullikson < > kevin.gullikson at gmail.com> wrote: > >> Antonio, >> >> The statsmodels package has a robust linear model module that I have used >> before. You will have to transform your data to be linear first by taking >> the log of the y-axis. >> >> >> http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/robust_models_0.html >> >> >> Kevin Gullikson >> >> On Wed, Mar 11, 2015 at 12:04 PM, Antonino Ingargiola > > wrote: >> >>> Hi to the list, >>> >>> I'm seeking the advise of the scientific python community to solve the >>> following fitting problem. Both suggestions on the methodology and on >>> particular software packages are appreciated. >>> >>> I often encounter the need to fit a sample containing a (dominant) >>> exponentially-distributed sub-population. Mostly the non-exponential >>> samples (from an unknown distribution) are distributed close to the origin >>> of the exponential distribution, therefore a simple approach I used so far >>> is selecting all the samples higher than a threshold and fitting the >>> exponential "tail" with MLE. >>> >>> The problem is that the choice of the threshold is somewhat arbitrary >>> and moreover there can be a small set of outlier on the extreme right-side >>> of the distribution that would bias the MLE fit. >>> >>> To improve the accuracy, I'm thinking of using (if necessary >>> implementing) some kind of robust fitting procedure. For example using a >>> scheme in which the outlier are identified by putting a threshold on the >>> residual and then this threshold is optimized using some "goodness of fit" >>> cost function. If this approach reasonable? >>> >>> I am surely not the first to tackle this problem, so I would appreciated >>> some suggestion and specific pointers to help me getting started. >>> >>> Thank you, >>> Antonio >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daoust.mj at gmail.com Wed Mar 11 20:15:44 2015 From: daoust.mj at gmail.com (Mark Daoust) Date: Wed, 11 Mar 2015 20:15:44 -0400 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: What about the EM algorithm. You could fit a mixture of an exponential and a [whatever] distribution? since you seem to already believe that that's what it is? Isn't it almost what you already have in mind already, just with soft thresholding? Mark Daoust On Wed, Mar 11, 2015 at 7:36 PM, Antonino Ingargiola wrote: > Hi Kevin, > > If I apply the log transform to the sample to linearize the models, what > is the correct way to weight the residuals? Without weighting residual > close to the tail will be amplified and bias the fit. > > Antonio > > On Wed, Mar 11, 2015 at 11:08 AM, Kevin Gullikson < > kevin.gullikson at gmail.com> wrote: > >> Antonio, >> >> The statsmodels package has a robust linear model module that I have used >> before. You will have to transform your data to be linear first by taking >> the log of the y-axis. >> >> >> http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/robust_models_0.html >> >> >> Kevin Gullikson >> >> On Wed, Mar 11, 2015 at 12:04 PM, Antonino Ingargiola > > wrote: >> >>> Hi to the list, >>> >>> I'm seeking the advise of the scientific python community to solve the >>> following fitting problem. Both suggestions on the methodology and on >>> particular software packages are appreciated. >>> >>> I often encounter the need to fit a sample containing a (dominant) >>> exponentially-distributed sub-population. Mostly the non-exponential >>> samples (from an unknown distribution) are distributed close to the origin >>> of the exponential distribution, therefore a simple approach I used so far >>> is selecting all the samples higher than a threshold and fitting the >>> exponential "tail" with MLE. >>> >>> The problem is that the choice of the threshold is somewhat arbitrary >>> and moreover there can be a small set of outlier on the extreme right-side >>> of the distribution that would bias the MLE fit. >>> >>> To improve the accuracy, I'm thinking of using (if necessary >>> implementing) some kind of robust fitting procedure. For example using a >>> scheme in which the outlier are identified by putting a threshold on the >>> residual and then this threshold is optimized using some "goodness of fit" >>> cost function. If this approach reasonable? >>> >>> I am surely not the first to tackle this problem, so I would appreciated >>> some suggestion and specific pointers to help me getting started. >>> >>> Thank you, >>> Antonio >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed Mar 11 20:26:11 2015 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 12 Mar 2015 11:26:11 +1100 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: I'm not sure if this helps, but if you can transform standard deviations as follows: f = a * np.log(b * A) then sigma_f = np.abs(a * sigma_A / A) On 12 March 2015 at 11:15, Mark Daoust wrote: > What about the EM algorithm. You could fit a mixture of an exponential and > a [whatever] distribution? since you seem to already believe that that's > what it is? > > Isn't it almost what you already have in mind already, just with soft > thresholding? > > > > > > > Mark Daoust > > On Wed, Mar 11, 2015 at 7:36 PM, Antonino Ingargiola > wrote: > >> Hi Kevin, >> >> If I apply the log transform to the sample to linearize the models, what >> is the correct way to weight the residuals? Without weighting residual >> close to the tail will be amplified and bias the fit. >> >> Antonio >> >> On Wed, Mar 11, 2015 at 11:08 AM, Kevin Gullikson < >> kevin.gullikson at gmail.com> wrote: >> >>> Antonio, >>> >>> The statsmodels package has a robust linear model module that I have >>> used before. You will have to transform your data to be linear first by >>> taking the log of the y-axis. >>> >>> >>> http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/robust_models_0.html >>> >>> >>> Kevin Gullikson >>> >>> On Wed, Mar 11, 2015 at 12:04 PM, Antonino Ingargiola < >>> tritemio at gmail.com> wrote: >>> >>>> Hi to the list, >>>> >>>> I'm seeking the advise of the scientific python community to solve the >>>> following fitting problem. Both suggestions on the methodology and on >>>> particular software packages are appreciated. >>>> >>>> I often encounter the need to fit a sample containing a (dominant) >>>> exponentially-distributed sub-population. Mostly the non-exponential >>>> samples (from an unknown distribution) are distributed close to the origin >>>> of the exponential distribution, therefore a simple approach I used so far >>>> is selecting all the samples higher than a threshold and fitting the >>>> exponential "tail" with MLE. >>>> >>>> The problem is that the choice of the threshold is somewhat arbitrary >>>> and moreover there can be a small set of outlier on the extreme right-side >>>> of the distribution that would bias the MLE fit. >>>> >>>> To improve the accuracy, I'm thinking of using (if necessary >>>> implementing) some kind of robust fitting procedure. For example using a >>>> scheme in which the outlier are identified by putting a threshold on the >>>> residual and then this threshold is optimized using some "goodness of fit" >>>> cost function. If this approach reasonable? >>>> >>>> I am surely not the first to tackle this problem, so I would >>>> appreciated some suggestion and specific pointers to help me getting >>>> started. >>>> >>>> Thank you, >>>> Antonio >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Mar 11 23:48:30 2015 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 11 Mar 2015 20:48:30 -0700 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 5:15 PM, wrote: > RLM produces an unbiased estimator of the mean or mean function for > symmetric distribution and is calibrated for the normal distribution. I > don't know how well this is approximated by the log of an exponentially > distributed variable, but it won't exactly satisfy the assumptions. > Yes, RLM will heavily weight the tail that before the transform is almost 0 and after becomes arbitrary large negative values. There should be a more direct way of estimating the parameter for the > exponential distribution in a robust way, but I never tried. > (one idea would be to estimate a trimmed mean and use the estimated > distribution to correct for the trimming. scipy.stats.distributions have an > `expect` method that can be used to calculate the mean of a trimmed > distribution, i.e. conditional on lower and upper bounds) > Thanks, I wasn't aware of the expect method. However I already tried to trim the distribution (on the left-side). But the thresholds are arbitrary and I would like to make the fitting not dependent on them. > What's your sample size? > (for very large sample sizes one approach that is sometimes used, is to > fit a distribution to the central part of a histogram) > The sample size is from a few 100s to a few 1000s, not really huge. The problem is that I don't know if there is a "robust" a criterion to trim the distribution, and then what would be the accuracy of such a fit. Antonio -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Thu Mar 12 00:10:39 2015 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 11 Mar 2015 21:10:39 -0700 Subject: [SciPy-User] Robust fitting of an exponential distribution subpopulation In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 5:15 PM, Mark Daoust wrote: > What about the EM algorithm. You could fit a mixture of an exponential and > a [whatever] distribution? since you seem to already believe that that's > what it is? > Yes EM would be great, except that I don't know how to model the outliers :(. And while in different samples I'm always looking at the exponential tail, the distribution of the outliers can be completely different. Isn't it almost what you already have in mind already, just with soft > thresholding? Yes almost. If only I had a way to identify the "sweet" purely exponential part of the distribution in some robust way :). Honestly, I was thinking of doing a curve fit of the empirical CDF, select the samples with residuals below a threshold and re-perform the fit on the sub-population iteratively in some way. But there are a few important details I'm unsure of. Is using the ECDF a good approach when the sample is contaminated? Should I use a test of exponentiality for the samples selection (like Kolgomorov-Smirnov)? I could try to delve into these details but I don't want to reinvent the wheel or end up with a custom sub-optimal solution. Antonio -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Mar 12 20:33:57 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 12 Mar 2015 17:33:57 -0700 Subject: [SciPy-User] np.corrcoef ddof is redundant? In-Reply-To: References: <1862603727447697417.024738sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Tue, Mar 10, 2015 at 1:12 PM, Matthew Brett wrote: > Hi, > > On Tue, Mar 10, 2015 at 9:27 AM, Sturla Molden wrote: >> Alistair Miles wrote: >> >>> I'm trying to calculate correlation coefficients and looking at the >>> np.corrcoef function. It has bias and ddof arguments, however when I try >>> different values of ddof with test data the results are always the same, >>> i.e., changing ddof has no effect. From some back-of-the-envelope algebra I >>> reckon the n/(n-ddof) normalisations should get cancelled out when >>> calculating correlation coefficients from a covariance matrix, and >>> therefore the ddof (and bias) arguments to np.corrcoef are redundant. >>> >>> I'd be very grateful if someone could verify this is true or tell me if >>> I've missed something. >> >> You are right. It should cancel out or np.corrcoef would be wrong. The >> sample size does not go into the Pearson product-moment correlation. > > Oh dear - that's embarrassing. > > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient > > I guess we should deprecate the 'bias' and 'ddof' input arguments asap. https://github.com/numpy/numpy/pull/5675 Cheers, Matthew From jeffreback at gmail.com Fri Mar 13 11:33:36 2015 From: jeffreback at gmail.com (Jeff Reback) Date: Fri, 13 Mar 2015 11:33:36 -0400 Subject: [SciPy-User] Pandas v0.16.0 release candidate 1 Message-ID: Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.16.0. Please try this RC and report any issues here: Pandas Issues We will be releasing officially in 1 week or so. This is a major release from 0.15.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. - Highlights include: - DataFrame.assign method, see *here* - Series.to_coo/from_coo methods to interact with scipy.sparse, see *here* - Backwards incompatible change to Timedelta to conform the .seconds attribute with datetime.timedelta, see *here* - Changes to the .loc slicing API to conform with the behavior of .ix see *here * - Changes to the default for ordering in the Categorical constructor, see *here * Here are the full whatsnew and documentation links: v0.16.0 Whatsnew Source tarballs, windows builds, and mac wheels are available here: Pandas v0.16.0rc1 Release A big thank you to everyone who contributed to this release! Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmhobson at gmail.com Fri Mar 13 13:45:27 2015 From: pmhobson at gmail.com (Paul Hobson) Date: Fri, 13 Mar 2015 10:45:27 -0700 Subject: [SciPy-User] [pydata] Pandas v0.16.0 release candidate 1 In-Reply-To: References: Message-ID: Thanks for the all the hard work! Really looking forward to using the `assign` method in long chained statements. -Paul On Fri, Mar 13, 2015 at 8:33 AM, Jeff Reback wrote: > Hi, > > I'm pleased to announce the availability of the first release candidate of > Pandas 0.16.0. > Please try this RC and report any issues here: Pandas Issues > > We will be releasing officially in 1 week or so. > > This is a major release from 0.15.2 and includes a small number of API > changes, several new features, enhancements, and performance improvements > along with a large number of bug fixes. We recommend that all users upgrade > to this version. > > - Highlights include: > - DataFrame.assign method, see *here* > > - Series.to_coo/from_coo methods to interact with scipy.sparse, see > *here* > > - Backwards incompatible change to Timedelta to conform the .seconds attribute > with datetime.timedelta, see *here* > > - Changes to the .loc slicing API to conform with the behavior of > .ix see *here > * > - Changes to the default for ordering in the Categorical constructor, > see *here > * > > > Here are the full whatsnew and documentation links: > v0.16.0 Whatsnew > > > Source tarballs, windows builds, and mac wheels are available here: > > Pandas v0.16.0rc1 Release > > A big thank you to everyone who contributed to this release! > > Jeff > > -- > You received this message because you are subscribed to the Google Groups > "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pydata+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Mar 13 14:27:12 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 13 Mar 2015 11:27:12 -0700 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Mar 10, 2015 at 9:30 AM, Sturla Molden wrote: > We prefer "slow and correct" code to "fast but > incorrect" code. As a consequence of this, you will often be better off > spending your time writing rigorous test cases than optimizing for speed. > And you'll really want those tests if/when you do optimize for speed. Rather it is the time you spend working on the > project, from its planning until your paper is published. Exactly -- or even if you aren't an academic, development time is far more valuable than run time. They key here is that Cython is really great, and can be very easy to use -- but it is not at all cost free. The edit-compile-test cycle is much slower, and it's a lot harder to debug than Python -- so you really don't want to use it unless there is a real gain to to doing so. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Fri Mar 13 22:19:42 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Fri, 13 Mar 2015 19:19:42 -0700 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: Thanks so much everyone for the discussion. The recommendations for stuff like Numba and "just stick with Python at first" came in really useful as it caused me to bump into this (while learning about Numba): https://jakevdp.github.io/blog/2015/02/24/optimizing-python-with-numpy-and-numba/ I have promised myself to never start writing "Cython-first" in mind again. I'll stick to Python, then do profiling (including line based, if useful), use Numba when I have to, and use Cython to wrap C code if I must. On Fri, Mar 13, 2015 at 11:27 AM, Chris Barker wrote: > On Tue, Mar 10, 2015 at 9:30 AM, Sturla Molden > wrote: > >> We prefer "slow and correct" code to "fast but >> incorrect" code. As a consequence of this, you will often be better off >> spending your time writing rigorous test cases than optimizing for speed. >> > > And you'll really want those tests if/when you do optimize for speed. > > Rather it is the time you spend working on the >> project, from its planning until your paper is published. > > > Exactly -- or even if you aren't an academic, development time is far more > valuable than run time. > > They key here is that Cython is really great, and can be very easy to use > -- but it is not at all cost free. The edit-compile-test cycle is much > slower, and it's a lot harder to debug than Python -- so you really don't > want to use it unless there is a real gain to to doing so. > > -Chris > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From highscot at gmail.com Sat Mar 14 01:25:55 2015 From: highscot at gmail.com (Scott High) Date: Sat, 14 Mar 2015 00:25:55 -0500 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: Hey All, First: > I have promised myself to never start writing "Cython-first" in mind again. > I agree, good choice. Beyond that, I have a few comments based on my experience using Python/Cython for research in numerical methods. - Most code in scientific programming is not CPU bound. You don't get a > faster harddrive or faster network connection from using Cython. This > accounts for the majority (often 80 to 90 %) of the code we write. > This is often true, but is not necessarily an argument again using Cython. (You only mentioned hard drives and network connections, but hitting any level in the memory hierarchy that is not cache can wreck performance). If you carefully read code that uses numpy array operations (especially slicing) you will notice that data will often be accessed multiple times in the same mathematical operation. The largest speed ups I have gotten with Cython are not from reducing the number of flops, but in using C style loops to minimize the number of passes through memory. For the computer scientists in the room: Cython can often allow you to better utilize cache. For me a good rule has been: usually don't convert to Cython to reduce the number of flops, but consider converting to better manage memory. There are some serious weasel words in that rule, mostly because it is very hard to judge potential speed ups by inspecting code, even for experts. They key here is that Cython is really great, and can be very easy to use > -- but it is not at all cost free. The edit-compile-test cycle is much > slower, and it's a lot harder to debug than Python -- so you really don't > want to use it unless there is a real gain to to doing so. > If you intend to write the majority of your program in Cython the compile time can really hurt you, and I am not crazy about the large amount of unreadable code generated. On the other hand, I have found that if you only intend to convert a small number of functions then the IPython Cython cell magic can make development nearly as quick as it would be with pure Python. As far as debugging goes, it is a pain. No way around that one. - Optimizing NumPy code with Numba is often easier than using Cython. > I have used Numba and was very much impressed. The problem is that installation can be a nightmare. Depending on your users system they may have to install LLVM (I had to build it from source). This is a very heavy duty dependency, do not discount it. - When programming Cython I often find I want to use C or Fortran instead. > I find that when 'optimizing' Cython I often want to use C (not Fortran!) instead. Usually after converting a function to Cython I find that its performance is good enough. When it is not I skip the confusing optimization tips for Cython and go straight to C. This is a straightforward process when you already have things expressed with c types. - Bottlenecks are often obscured. Use a profiler to locate them. > Yes. I am a huge fan of the IPython line profiler magic. - The best way to deal with a bottleneck is very often to use a better > algorithm ot datastructure, not move from Python to C or Cython. > Yes. On Fri, Mar 13, 2015 at 9:19 PM, Brian Merchant wrote: > Thanks so much everyone for the discussion. > > The recommendations for stuff like Numba and "just stick with Python at > first" came in really useful as it caused me to bump into this (while > learning about Numba): > https://jakevdp.github.io/blog/2015/02/24/optimizing-python-with-numpy-and-numba/ > > > I have promised myself to never start writing "Cython-first" in mind > again. I'll stick to Python, then do profiling (including line based, if > useful), use Numba when I have to, and use Cython to wrap C code if I must. > > On Fri, Mar 13, 2015 at 11:27 AM, Chris Barker > wrote: > >> On Tue, Mar 10, 2015 at 9:30 AM, Sturla Molden >> wrote: >> >>> We prefer "slow and correct" code to "fast but >>> incorrect" code. As a consequence of this, you will often be better off >>> spending your time writing rigorous test cases than optimizing for speed. >>> >> >> And you'll really want those tests if/when you do optimize for speed. >> >> Rather it is the time you spend working on the >>> project, from its planning until your paper is published. >> >> >> Exactly -- or even if you aren't an academic, development time is far >> more valuable than run time. >> >> They key here is that Cython is really great, and can be very easy to use >> -- but it is not at all cost free. The edit-compile-test cycle is much >> slower, and it's a lot harder to debug than Python -- so you really don't >> want to use it unless there is a real gain to to doing so. >> >> -Chris >> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- -Scott High -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Sat Mar 14 19:01:34 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Sat, 14 Mar 2015 16:01:34 -0700 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? Message-ID: I need to write about 2000 lines worth of code (based on my last implementation of the project I am working on). I anticipate that I'll be using profilers (including line profilers), numpy, Numba, possibly Cython and almost definitely matplotlib animations. I will need to write my own classes, so my program isn't just going to be one long script. Previously, I used Spyder for development, but I am wondering if it makes sense for one to use an IPython notebook alone. I am considering that option so that I force myself to program "literately" -- lots of headings, and nice comments. Graphs embedded to show the effects of optimizations, and so forth. So far, I know that IPython might have some trouble with integrating in matplotlib animations, but that too is possible with a little research. Otherwise, I should be able to do everything else quite easily, including debugging, according to the IPython documentation. In fact, writing parallel code in IPython might be a very nice idea given how it seems to offer a lot of functionality in that direction. Do people use IPython (notebooks) as a complete IDE often? If not, why not? Kind regards, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sat Mar 14 19:03:11 2015 From: andyfaff at gmail.com (Andrew Nelson) Date: Sun, 15 Mar 2015 10:03:11 +1100 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: I'm converted to pycharmm now, great ide for python. On 15/03/2015 10:01 AM, "Brian Merchant" wrote: > I need to write about 2000 lines worth of code (based on my last > implementation of the project I am working on). I anticipate that I'll be > using profilers (including line profilers), numpy, Numba, possibly Cython > and almost definitely matplotlib animations. I will need to write my own > classes, so my program isn't just going to be one long script. > > > Previously, I used Spyder for development, but I am wondering if it makes > sense for one to use an IPython notebook alone. I am considering that > option so that I force myself to program "literately" -- lots of headings, > and nice comments. Graphs embedded to show the effects of optimizations, > and so forth. > > So far, I know that IPython might have some trouble with integrating in > matplotlib animations, but that too is possible with a little research. > Otherwise, I should be able to do everything else quite easily, including > debugging, according to the IPython documentation. In fact, writing > parallel code in IPython might be a very nice idea given how it seems to > offer a lot of functionality in that direction. > > > Do people use IPython (notebooks) as a complete IDE often? If not, why not? > > Kind regards, > Brian > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Mar 14 19:08:07 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Mar 2015 16:08:07 -0700 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: Hi, On Sat, Mar 14, 2015 at 4:01 PM, Brian Merchant wrote: > > So far, I know that IPython might have some trouble with integrating in > matplotlib animations, but that too is possible with a little research. > Otherwise, I should be able to do everything else quite easily, including > debugging, according to the IPython documentation. In fact, writing parallel > code in IPython might be a very nice idea given how it seems to offer a lot > of functionality in that direction. I use the notebook a lot, but I don't use it as an IDE. My usual workflow is to write code with my text editor, test at the terminal and explore code at the IPython terminal. When I want to demo or teach what I have done, then I usually make a notebook. I find that writing substantial code in the notebook quickly makes me lose focus on what I am trying to do. It's partly that the notebook is really not good enough as a code editor for serious coding work. It is also the state in the notebook that makes it very easy to write code that assumes variables that have since been deleted or renamed. Also - if I want to write a reasonably long document of any type, I really need my familiar text editor so that I am not distracted by the process of editing, as I often am in the notebook. So - I think the notebook is really ideal for demos, short and informal explorations, and as a teaching tool, but I would personally find it very distracting as an IDE. Cheers, Matthew From josef.pktd at gmail.com Sat Mar 14 19:17:56 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 14 Mar 2015 19:17:56 -0400 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: On Sat, Mar 14, 2015 at 7:08 PM, Matthew Brett wrote: > Hi, > > On Sat, Mar 14, 2015 at 4:01 PM, Brian Merchant > wrote: > > > > So far, I know that IPython might have some trouble with integrating in > > matplotlib animations, but that too is possible with a little research. > > Otherwise, I should be able to do everything else quite easily, including > > debugging, according to the IPython documentation. In fact, writing > parallel > > code in IPython might be a very nice idea given how it seems to offer a > lot > > of functionality in that direction. > > I use the notebook a lot, but I don't use it as an IDE. > > My usual workflow is to write code with my text editor, test at the > terminal and explore code at the IPython terminal. > > When I want to demo or teach what I have done, then I usually make a > notebook. > > I find that writing substantial code in the notebook quickly makes me > lose focus on what I am trying to do. It's partly that the notebook > is really not good enough as a code editor for serious coding work. > It is also the state in the notebook that makes it very easy to write > code that assumes variables that have since been deleted or renamed. > > Also - if I want to write a reasonably long document of any type, I > really need my familiar text editor so that I am not distracted by the > process of editing, as I often am in the notebook. > > So - I think the notebook is really ideal for demos, short and > informal explorations, and as a teaching tool, but I would personally > find it very distracting as an IDE. > That's also pretty much my experience and usage. I got used to notebooks to try out and work on finished examples, and to try out code in PRs. However, whenever I get into debugging code or developing new code, then I quickly get annoyed by missing features and switch back to spyder. I currently have IDLE, spyder, eclipse/pydev and several notebooks open, plus a shell window for nosetests, and of course one for git. All for different purposes and use cases, but all related to statsmodels code development and review. ("Dis"iDE) Jpsef > > Cheers, > > Matthew > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yw5aj at virginia.edu Sat Mar 14 19:23:50 2015 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Sat, 14 Mar 2015 19:23:50 -0400 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: Me too. I tried to use IPython Notebooks as my IDE for a while, but probably because that it is host in a browser, the shortcut key feature is not as good, as specifically, I often mistakenly invoked the browser shortcut. I also got experience of accidentally deleting code without saving them. So, I use notebooks when I collaborate with my colleagues and send them code snippets (github + nbviewer); but for decent development, I always use spyder. Shawn On Sat, Mar 14, 2015 at 7:17 PM, wrote: > > > On Sat, Mar 14, 2015 at 7:08 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Mar 14, 2015 at 4:01 PM, Brian Merchant >> wrote: >> > >> > So far, I know that IPython might have some trouble with integrating in >> > matplotlib animations, but that too is possible with a little research. >> > Otherwise, I should be able to do everything else quite easily, >> > including >> > debugging, according to the IPython documentation. In fact, writing >> > parallel >> > code in IPython might be a very nice idea given how it seems to offer a >> > lot >> > of functionality in that direction. >> >> I use the notebook a lot, but I don't use it as an IDE. >> >> My usual workflow is to write code with my text editor, test at the >> terminal and explore code at the IPython terminal. >> >> When I want to demo or teach what I have done, then I usually make a >> notebook. >> >> I find that writing substantial code in the notebook quickly makes me >> lose focus on what I am trying to do. It's partly that the notebook >> is really not good enough as a code editor for serious coding work. >> It is also the state in the notebook that makes it very easy to write >> code that assumes variables that have since been deleted or renamed. >> >> Also - if I want to write a reasonably long document of any type, I >> really need my familiar text editor so that I am not distracted by the >> process of editing, as I often am in the notebook. >> >> So - I think the notebook is really ideal for demos, short and >> informal explorations, and as a teaching tool, but I would personally >> find it very distracting as an IDE. > > > That's also pretty much my experience and usage. > > I got used to notebooks to try out and work on finished examples, and to try > out code in PRs. > > However, whenever I get into debugging code or developing new code, then I > quickly get annoyed by missing features and switch back to spyder. > > I currently have IDLE, spyder, eclipse/pydev and several notebooks open, > plus a shell window for nosetests, and of course one for git. > All for different purposes and use cases, but all related to statsmodels > code development and review. > ("Dis"iDE) > > Jpsef > >> >> >> Cheers, >> >> Matthew >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From matthew.brett at gmail.com Sat Mar 14 20:24:05 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Mar 2015 17:24:05 -0700 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Fri, Mar 13, 2015 at 10:25 PM, Scott High wrote: > Hey All, > > First: >> >> I have promised myself to never start writing "Cython-first" in mind >> again. > > I agree, good choice. > > Beyond that, I have a few comments based on my experience using > Python/Cython for research in numerical methods. > >> - Most code in scientific programming is not CPU bound. You don't get a >> faster harddrive or faster network connection from using Cython. This >> accounts for the majority (often 80 to 90 %) of the code we write. > > This is often true, but is not necessarily an argument again using Cython. > (You only mentioned hard drives and network connections, but hitting any > level in the memory hierarchy that is not cache can wreck performance). If > you carefully read code that uses numpy array operations (especially > slicing) you will notice that data will often be accessed multiple times in > the same mathematical operation. The largest speed ups I have gotten with > Cython are not from reducing the number of flops, but in using C style loops > to minimize the number of passes through memory. For the computer scientists > in the room: Cython can often allow you to better utilize cache. For me a > good rule has been: usually don't convert to Cython to reduce the number of > flops, but consider converting to better manage memory. There are some > serious weasel words in that rule, mostly because it is very hard to judge > potential speed ups by inspecting code, even for experts. > >> They key here is that Cython is really great, and can be very easy to use >> -- but it is not at all cost free. The edit-compile-test cycle is much >> slower, and it's a lot harder to debug than Python -- so you really don't >> want to use it unless there is a real gain to to doing so. > > If you intend to write the majority of your program in Cython the compile > time can really hurt you, and I am not crazy about the large amount of > unreadable code generated. On the other hand, I have found that if you only > intend to convert a small number of functions then the IPython Cython cell > magic can make development nearly as quick as it would be with pure Python. > As far as debugging goes, it is a pain. No way around that one. > >> - Optimizing NumPy code with Numba is often easier than using Cython. > > I have used Numba and was very much impressed. The problem is that > installation can be a nightmare. Depending on your users system they may > have to install LLVM (I had to build it from source). This is a very heavy > duty dependency, do not discount it. Yes, I would think hard before committing to numba. Maybe because of dependencies, or some other reason, I believe it is fair to say that Cython is in much wider use than numba. For example, I don't know if any use of numba in the packages I use (or write) but Cython is very common. Cheers, Matthew From athanastasiou at gmail.com Sun Mar 15 05:08:48 2015 From: athanastasiou at gmail.com (Athanasios Anastasiou) Date: Sun, 15 Mar 2015 09:08:48 +0000 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: Hello all The previous posts pretty much describe my experience too but here are a few more observations. *) iPython notebooks are an ideal alternative to spreadsheet nomograms or "workbooks" in the style of Mathematica. An engineering design process for example can be captured once and re-used many times. (Or a draft for a paper or a tutorial or as a quick way to create web content). *) A few modules will not work properly through iPython. This means that it will either not be possible to use it at all (I.e. multiple processes ) or that it will be too slow or that if the code fails at a minor point, the whole notebook will have to be restarted (I.e. if it involves db connection(s) ). *) Creating a user interface with typical widgets is not exactly straightforward (I.e https://jakevdp.github.io/blog/2013/06/01/ipython-notebook-javascript-python-communication/ ) so, you would have to revert to using other means for that (possibly an ide with qt-designer or gtk-Glade or something). *) A typical IDE will have facilities for file management, version control, refactoring, rich search and replace and others. You can still emulate some of these with different software up and running of course and through scripting. It's a matter of convenience and its trade offs. IPython notebooks definitely have a place in the toolkit and are constantly evolving, but probably in parallel with IDEs rather than as alternatives. All the best AA On 14 Mar 2015 23:23, "Yuxiang Wang" wrote: > Me too. > > I tried to use IPython Notebooks as my IDE for a while, but probably > because that it is host in a browser, the shortcut key feature is not > as good, as specifically, I often mistakenly invoked the browser > shortcut. I also got experience of accidentally deleting code without > saving them. > > So, I use notebooks when I collaborate with my colleagues and send > them code snippets (github + nbviewer); but for decent development, I > always use spyder. > > Shawn > > On Sat, Mar 14, 2015 at 7:17 PM, wrote: > > > > > > On Sat, Mar 14, 2015 at 7:08 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Sat, Mar 14, 2015 at 4:01 PM, Brian Merchant > >> wrote: > >> > > >> > So far, I know that IPython might have some trouble with integrating > in > >> > matplotlib animations, but that too is possible with a little > research. > >> > Otherwise, I should be able to do everything else quite easily, > >> > including > >> > debugging, according to the IPython documentation. In fact, writing > >> > parallel > >> > code in IPython might be a very nice idea given how it seems to offer > a > >> > lot > >> > of functionality in that direction. > >> > >> I use the notebook a lot, but I don't use it as an IDE. > >> > >> My usual workflow is to write code with my text editor, test at the > >> terminal and explore code at the IPython terminal. > >> > >> When I want to demo or teach what I have done, then I usually make a > >> notebook. > >> > >> I find that writing substantial code in the notebook quickly makes me > >> lose focus on what I am trying to do. It's partly that the notebook > >> is really not good enough as a code editor for serious coding work. > >> It is also the state in the notebook that makes it very easy to write > >> code that assumes variables that have since been deleted or renamed. > >> > >> Also - if I want to write a reasonably long document of any type, I > >> really need my familiar text editor so that I am not distracted by the > >> process of editing, as I often am in the notebook. > >> > >> So - I think the notebook is really ideal for demos, short and > >> informal explorations, and as a teaching tool, but I would personally > >> find it very distracting as an IDE. > > > > > > That's also pretty much my experience and usage. > > > > I got used to notebooks to try out and work on finished examples, and to > try > > out code in PRs. > > > > However, whenever I get into debugging code or developing new code, then > I > > quickly get annoyed by missing features and switch back to spyder. > > > > I currently have IDLE, spyder, eclipse/pydev and several notebooks open, > > plus a shell window for nosetests, and of course one for git. > > All for different purposes and use cases, but all related to statsmodels > > code development and review. > > ("Dis"iDE) > > > > Jpsef > > > >> > >> > >> Cheers, > >> > >> Matthew > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > -- > Yuxiang "Shawn" Wang > Gerling Research Lab > University of Virginia > yw5aj at virginia.edu > +1 (434) 284-0836 > https://sites.google.com/a/virginia.edu/yw5aj/ > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gnurser at gmail.com Sun Mar 15 13:11:01 2015 From: gnurser at gmail.com (George Nurser) Date: Sun, 15 Mar 2015 17:11:01 +0000 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: I've found numba very good, and the mailing list helpful. It's new, and under pretty active development, so I guess it's not surprising that it's presently less used than cython. But there's e.g. Stephan Hoyer's numbagg https://github.com/shoyer/numbagg To build it yourself may be a pain, but, it is very easy to install it with conda install numba if you have the anaconda distribution, and, on the mac, with macports, sudo port install py34-numba (or py27-numba) works like a charm. Both macports and conda give up to date versions. cheers, George Nurser. On 15 March 2015 at 00:24, Matthew Brett wrote: > Hi, > > On Fri, Mar 13, 2015 at 10:25 PM, Scott High wrote: > > Hey All, > > > > First: > >> > >> I have promised myself to never start writing "Cython-first" in mind > >> again. > > > > I agree, good choice. > > > > Beyond that, I have a few comments based on my experience using > > Python/Cython for research in numerical methods. > > > >> - Most code in scientific programming is not CPU bound. You don't get a > >> faster harddrive or faster network connection from using Cython. This > >> accounts for the majority (often 80 to 90 %) of the code we write. > > > > This is often true, but is not necessarily an argument again using > Cython. > > (You only mentioned hard drives and network connections, but hitting any > > level in the memory hierarchy that is not cache can wreck performance). > If > > you carefully read code that uses numpy array operations (especially > > slicing) you will notice that data will often be accessed multiple times > in > > the same mathematical operation. The largest speed ups I have gotten with > > Cython are not from reducing the number of flops, but in using C style > loops > > to minimize the number of passes through memory. For the computer > scientists > > in the room: Cython can often allow you to better utilize cache. For me a > > good rule has been: usually don't convert to Cython to reduce the number > of > > flops, but consider converting to better manage memory. There are some > > serious weasel words in that rule, mostly because it is very hard to > judge > > potential speed ups by inspecting code, even for experts. > > > >> They key here is that Cython is really great, and can be very easy to > use > >> -- but it is not at all cost free. The edit-compile-test cycle is much > >> slower, and it's a lot harder to debug than Python -- so you really > don't > >> want to use it unless there is a real gain to to doing so. > > > > If you intend to write the majority of your program in Cython the compile > > time can really hurt you, and I am not crazy about the large amount of > > unreadable code generated. On the other hand, I have found that if you > only > > intend to convert a small number of functions then the IPython Cython > cell > > magic can make development nearly as quick as it would be with pure > Python. > > As far as debugging goes, it is a pain. No way around that one. > > > >> - Optimizing NumPy code with Numba is often easier than using Cython. > > > > I have used Numba and was very much impressed. The problem is that > > installation can be a nightmare. Depending on your users system they may > > have to install LLVM (I had to build it from source). This is a very > heavy > > duty dependency, do not discount it. > > Yes, I would think hard before committing to numba. Maybe because of > dependencies, or some other reason, I believe it is fair to say that > Cython is in much wider use than numba. For example, I don't know if > any use of numba in the packages I use (or write) but Cython is very > common. > > Cheers, > > Matthew > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Sun Mar 15 13:36:08 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Sun, 15 Mar 2015 10:36:08 -0700 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: On 15 March 2015 at 02:08, Athanasios Anastasiou wrote: > *) A few modules will not work properly through iPython. This means that > it will either not be possible to use it at all (I.e. multiple processes ) > or that it will be too slow or that if the code fails at a minor point, the > whole notebook will have to be restarted (I.e. if it involves db > connection(s) ). Please let us (IPython) know about these kinds of cases. It may be that there are things for us to fix in IPython, or we may be able to work with package authors to avoid things like database connections that can only be reset by restarting the kernel. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Tue Mar 17 09:28:41 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 17 Mar 2015 14:28:41 +0100 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: On 15/03/15 18:11, George Nurser wrote: > To build it yourself may be a pain, but, it is very easy to install it with > conda install numba Most scientists will use Anaconda or Enthought Canopy. With Anaconda we do as you said (conda install numba), with Canopy we just click on it in the graphical package manager. The rest can install Numba from PyPI: $ pip install numba It is that hard to install... Maybe it was a PITA to build at some point in history, but that is all in the past. Sturla From u.fechner at tudelft.nl Tue Mar 17 09:56:05 2015 From: u.fechner at tudelft.nl (Uwe Fechner) Date: Tue, 17 Mar 2015 14:56:05 +0100 Subject: [SciPy-User] Should one NOT start with Cython first? In-Reply-To: References: <1810637652447694139.808192sturla.molden-gmail.com@news.gmane.org> <384511167447697756.715298sturla.molden-gmail.com@news.gmane.org> Message-ID: <55083275.4060904@tudelft.nl> Well, even though I like Numba, installing it is a nightmare. I wrote a lot of code for Numba 0.11, which is not compatible to newer versions: In newer version object oriented code (jit compiled classes) are not longer supported. pip install numba does not work easily, not even for the current version of numba: You need to have a specific version of llvm and llvmlite installed first, which is not easily available for Ubuntu 12.04, which I am using. We had to write our own installer just to be able to continue to install Numba 0.11, which is needed for our current code base (see: https://bitbucket.org/ufechner/freekitesim ). Am 17.03.2015 um 14:28 schrieb Sturla Molden: > On 15/03/15 18:11, George Nurser wrote: > >> To build it yourself may be a pain, but, it is very easy to install it with >> conda install numba > > Most scientists will use Anaconda or Enthought Canopy. With Anaconda we > do as you said (conda install numba), with Canopy we just click on it in > the graphical package manager. > > The rest can install Numba from PyPI: > > $ pip install numba > > It is that hard to install... > > Maybe it was a PITA to build at some point in history, but that is all > in the past. > > > > Sturla > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- --------------------------------------------- Uwe Fechner, M.Sc. Delft University of Technology Faculty of Aerospace Engineering/ Wind Energy Kluyverweg 1, 2629 HS Delft, The Netherlands Phone: +31-15-27-88902 From jbednar at inf.ed.ac.uk Tue Mar 17 14:37:44 2015 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Tue, 17 Mar 2015 18:37:44 +0000 Subject: [SciPy-User] ANN: HoloViews 1.0 released In-Reply-To: <21767.24743.332660.885819@hebb.inf.ed.ac.uk> References: <21764.49404.322423.198518@hebb.inf.ed.ac.uk> <5504D19F.3070206@ed.ac.uk> <21765.26302.991473.772921@hebb.inf.ed.ac.uk> <21767.14083.59177.198073@hebb.inf.ed.ac.uk> <1426544487.2728367.241229649.23B94496@webmail.messagingengine.com> <21767.24743.332660.885819@hebb.inf.ed.ac.uk> Message-ID: <21768.29816.377872.958413@hebb.inf.ed.ac.uk> We are pleased to announce the first public release of HoloViews, a Python package for scientific and engineering data visualization: http://ioam.github.io/holoviews HoloViews provides composable, sliceable, declarative data structures for building even complex visualizations easily. It's designed to exploit the rich ecosystem of scientific Python tools already available, using Numpy for data storage, matplotlib and mpld3 as plotting backends, and integrating fully with IPython Notebook to make your data instantly visible. If you look at the website for just about any other visualization package, you'll see a long list of pretty pictures, each one of which has a page or two of code putting it together. There are pretty pictures in HoloViews too, but there is *no* hidden code -- *all* of the steps needed to build a given figure are shown right before the HoloViews plot, with just a few lines needed for nearly all of our examples, even complex multi-figure subplots and animations. This concise but flexible specification makes it practical to explore and analyze your data interactively, while leaving a full record for later reproducibility in the notebook. It may sound like magic, but it's not -- HoloViews simply lets you annotate your data with appropriate metadata, and then the data can display itself! HoloViews provides a set of general, compositional, multidimensional data structures suitable for both discrete and continuous real-world data, and pairs them with separate customizable plotting classes to visualize them without extensive coding. An large collection of continuously tested IPython Notebook tutorials accompanies HoloViews, showing you precisely the small number of steps required to generate any of the plots. Some of the most important features: - Freely available under a BSD license - Python 2 and 3 compatible - Minimal external dependencies -- easy to integrate into your workflow - Builds figures by slicing, sampling, and composing your data - Builds web-embeddable animations without any extra coding - Easily customizable without obscuring the underlying data objects - Includes interfaces to pandas and Seaborn - Winner of the 2015 UK Open Source Award For the rest, check out ioam.github.io/holoviews! Jean-Luc Stevens Philipp Rudiger James A. Bednar The University of Edinburgh School of Informatics -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From pawel.kw at gmail.com Wed Mar 18 11:06:19 2015 From: pawel.kw at gmail.com (=?UTF-8?Q?Pawe=C5=82_Kwa=C5=9Bniewski?=) Date: Wed, 18 Mar 2015 16:06:19 +0100 Subject: [SciPy-User] Asymmetric peak fitting Message-ID: Hi All, I'm currently trying to fit some experimental data in the form of asymmetric peaks. In principle it's difficult to find a distribution describing this kind of data. My goal is to get the full width half maximum of the peak and the peak position. I tried to look for some ready solutions within scipy but could not find any. I did find a nice paper though: http://arxiv.org/abs/0711.4449 Since it's not so trivial (at least not for me) to implement this, I'd like to ask if anyone in the community has already done this (or something similar) and would like to share the code. Cheers, Pawel Kwasniewski -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgodshall at enthought.com Wed Mar 18 12:58:36 2015 From: cgodshall at enthought.com (cgodshall at enthought.com) Date: Wed, 18 Mar 2015 11:58:36 -0500 Subject: [SciPy-User] SciPy 2015 Call for Propsals Open - tutorial & talk submissions due April 1st Message-ID: <00ea01d0619c$c81583b0$58408b10$@enthought.com> **SciPy 2015 Conference (Scientific Computing with Python) Call for Proposals: Submit Your Tutorial and Talk Ideas by April 1, 2015 at http://scipy2015.scipy.org.** SciPy 2015, the fourteenth annual Scientific Computing with Python conference, will be held July 6-12, 2015 in Austin, Texas. SciPy is a community dedicated to the advancement of scientific computing through open source Python software for mathematics, science, and engineering. The annual SciPy Conference brings together over 500 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The full program will consist of two days of tutorials by followed by three days of presentations, and concludes with two days of developer sprints. More info available on the conference website at http://scipy2015.scipy.org; you can also sign up on the website for mailing list updates or follow @scipyconf on Twitter. We hope you'll join us - early bird registration is open until May 15, 2015 at http://scipy2015.scipy.org We encourage you to submit tutorial or talk proposals in the categories below; please also share with others who you'd like to see participate! Submit via the conference website @ http://scipy2015.scipy.org. *SCIPY TUTORIAL SESSION PROPOSALS - DEADLINE EXTENDED TO WED APRIL 1, 2015* The SciPy experience kicks off with two days of tutorials. These sessions provide extremely affordable access to expert training, and consistently receive fantastic feedback from participants. We're looking for submissions on topics from introductory to advanced - we'll have attendees across the gamut looking to learn. Whether you are a major contributor to a scientific Python library or an expert-level user, this is a great opportunity to share your knowledge and stipends are available. Submit Your Tutorial Proposal on the SciPy 2015 website: http://scipy2015.scipy.org *SCIPY TALK AND POSTER SUBMISSIONS - DUE April 1, 2015* SciPy 2015 will include 3 major topic tracks and 7 mini-symposia tracks. Submit Your Talk Proposal on the SciPy 2015 website: http://scipy2015.scipy.org Major topic tracks include: - Scientific Computing in Python (General track) - Python in Data Science - Quantitative and Computational Social Sciences Mini-symposia will include the applications of Python in: - Astronomy and astrophysics - Computational life and medical sciences - Engineering - Geographic information systems (GIS) - Geophysics - Oceanography and meteorology - Visualization, vision and imaging If you have any questions or comments, feel free to contact us at: scipy-organizers at scipy.org. From ndbecker2 at gmail.com Wed Mar 18 13:36:14 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 18 Mar 2015 13:36:14 -0400 Subject: [SciPy-User] optimize - initial step size too small (for noisy objective) Message-ID: I tried using scipy.optimize.minimize, with a single variable, with a bound of [(-2,2)]. The initial step was only 1e-8. The objective function could be (a little) noisy - this seemed to cause minimize to never attempt a step in that direction again. Is there some way to cause it to take a bigger step, e.g., 1e-3? I'm trying brent now, probably a better choice. -- Those who fail to understand recursion are doomed to repeat it From jkhilmer at chemistry.montana.edu Wed Mar 18 17:36:35 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Wed, 18 Mar 2015 15:36:35 -0600 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: On Wed, Mar 18, 2015 at 9:06 AM, Pawe? Kwa?niewski wrote: > Hi All, > > I'm currently trying to fit some experimental data in the form of > asymmetric peaks. In principle it's difficult to find a distribution > describing this kind of data. My goal is to get the full width half maximum > of the peak and the peak position. I tried to look for some ready solutions > within scipy but could not find any. I did find a nice paper though: > > http://arxiv.org/abs/0711.4449 > > Since it's not so trivial (at least not for me) to implement this, I'd > like to ask if anyone in the community has already done this (or something > similar) and would like to share the code. > > There are a few publications detailing modified gaussians for peak tailing in chromatography: http://en.wikipedia.org/wiki/Exponentially_modified_Gaussian_distribution. Be careful about the equations on the Wikipedia page, they are numerically unstable: reference [http://dx.doi.org/10.1002%2Fcem.1343] for details. I have written python versions of those equations which are accurate but not fast. Asymmetric peaks are also common in many forms of spectroscopy, especially mass spectrometry. I have also written Scipy code to fit peaks for that application. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Wed Mar 18 21:16:26 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Wed, 18 Mar 2015 20:16:26 -0500 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Pawel, On Wed, Mar 18, 2015 at 10:06 AM, Pawe? Kwa?niewski wrote: > Hi All, > > I'm currently trying to fit some experimental data in the form of > asymmetric peaks. In principle it's difficult to find a distribution > describing this kind of data. My goal is to get the full width half maximum > of the peak and the peak position. I tried to look for some ready solutions > within scipy but could not find any. I did find a nice paper though: > > http://arxiv.org/abs/0711.4449 > > Since it's not so trivial (at least not for me) to implement this, I'd > like to ask if anyone in the community has already done this (or something > similar) and would like to share the code. > > Cheers, > > Pawel Kwasniewski > You might find the lmfit module (http://lmfit.github.io/lmfit-py/) useful. This is a module for least-squares minimization and curve-fitting, built on top of scipy.optimize. It includes several built-in Models, a few representing asymmetric peaks such as an exponentially damped Gaussian ( http://lmfit.github.io/lmfit-py/builtin_models.html#exponentialgaussianmodel). A quick look at the paper you referenced suggests this function might be sufficient, but if this or one of the other built-in Models isn't exactly what you're looking forward, it's very easy to make a new Model class from a Python function that calculates and returns the model function. Lmfit Models can also be added or multiplied together to make composite models (say, Gaussian + Step + Quadratic), and allow you to place bounds and/or constraints on any of the Parameters in the fit. Though there are many features you can use to set up models and their parameters, using these models (either built-in or one that you define yourself) for curve-fitting data is pretty straightforward. See http://lmfit.github.io/lmfit-py/builtin_models.html#example-1-fit-peaked-data-to-gaussian-lorentzian-and-voigt-profiles for a simple example. That example uses symmetric peaks, but the use of any Model for curve-fitting is basically the same. Hope that helps, --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Thu Mar 19 04:09:12 2015 From: pawel.kw at gmail.com (=?UTF-8?Q?Pawe=C5=82_Kwa=C5=9Bniewski?=) Date: Thu, 19 Mar 2015 09:09:12 +0100 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Jonathan, Matt, Thank you for your answers. I must say that lmfit is already a good friend of mine - I use it for quite some time for fitting. Somehow I didn't think of looking there for asymmetric peak models. Thanks for pointing this out! I suppose this will solve my problem. Cheers, Pawel 2015-03-19 2:16 GMT+01:00 Matt Newville : > Pawel, > > On Wed, Mar 18, 2015 at 10:06 AM, Pawe? Kwa?niewski > wrote: > >> Hi All, >> >> I'm currently trying to fit some experimental data in the form of >> asymmetric peaks. In principle it's difficult to find a distribution >> describing this kind of data. My goal is to get the full width half maximum >> of the peak and the peak position. I tried to look for some ready solutions >> within scipy but could not find any. I did find a nice paper though: >> >> http://arxiv.org/abs/0711.4449 >> >> Since it's not so trivial (at least not for me) to implement this, I'd >> like to ask if anyone in the community has already done this (or something >> similar) and would like to share the code. >> >> Cheers, >> >> Pawel Kwasniewski >> > > You might find the lmfit module (http://lmfit.github.io/lmfit-py/) > useful. This is a module for least-squares minimization and curve-fitting, > built on top of scipy.optimize. It includes several built-in Models, a few > representing asymmetric peaks such as an exponentially damped Gaussian ( > http://lmfit.github.io/lmfit-py/builtin_models.html#exponentialgaussianmodel). > A quick look at the paper you referenced suggests this function might be > sufficient, but if this or one of the other built-in Models isn't exactly > what you're looking forward, it's very easy to make a new Model class from > a Python function that calculates and returns the model function. Lmfit > Models can also be added or multiplied together to make composite models > (say, Gaussian + Step + Quadratic), and allow you to place bounds and/or > constraints on any of the Parameters in the fit. > > Though there are many features you can use to set up models and their > parameters, using these models (either built-in or one that you define > yourself) for curve-fitting data is pretty straightforward. See > http://lmfit.github.io/lmfit-py/builtin_models.html#example-1-fit-peaked-data-to-gaussian-lorentzian-and-voigt-profiles > for a simple example. That example uses symmetric peaks, but the use of > any Model for curve-fitting is basically the same. > > Hope that helps, > > --Matt Newville > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Thu Mar 19 05:57:01 2015 From: pawel.kw at gmail.com (=?UTF-8?Q?Pawe=C5=82_Kwa=C5=9Bniewski?=) Date: Thu, 19 Mar 2015 10:57:01 +0100 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Matt, I have one more question: the weights for fitting a model should be 1./error_bar or the error_bar? The name "weights" suggests the former, but I'd like to be sure. Cheers, Pawe? 2015-03-19 9:09 GMT+01:00 Pawe? Kwa?niewski : > Jonathan, Matt, > > Thank you for your answers. I must say that lmfit is already a good friend > of mine - I use it for quite some time for fitting. Somehow I didn't think > of looking there for asymmetric peak models. Thanks for pointing this out! > I suppose this will solve my problem. > > Cheers, > > Pawel > > 2015-03-19 2:16 GMT+01:00 Matt Newville : > >> Pawel, >> >> On Wed, Mar 18, 2015 at 10:06 AM, Pawe? Kwa?niewski >> wrote: >> >>> Hi All, >>> >>> I'm currently trying to fit some experimental data in the form of >>> asymmetric peaks. In principle it's difficult to find a distribution >>> describing this kind of data. My goal is to get the full width half maximum >>> of the peak and the peak position. I tried to look for some ready solutions >>> within scipy but could not find any. I did find a nice paper though: >>> >>> http://arxiv.org/abs/0711.4449 >>> >>> Since it's not so trivial (at least not for me) to implement this, I'd >>> like to ask if anyone in the community has already done this (or something >>> similar) and would like to share the code. >>> >>> Cheers, >>> >>> Pawel Kwasniewski >>> >> >> You might find the lmfit module (http://lmfit.github.io/lmfit-py/) >> useful. This is a module for least-squares minimization and curve-fitting, >> built on top of scipy.optimize. It includes several built-in Models, a few >> representing asymmetric peaks such as an exponentially damped Gaussian ( >> http://lmfit.github.io/lmfit-py/builtin_models.html#exponentialgaussianmodel). >> A quick look at the paper you referenced suggests this function might be >> sufficient, but if this or one of the other built-in Models isn't exactly >> what you're looking forward, it's very easy to make a new Model class from >> a Python function that calculates and returns the model function. Lmfit >> Models can also be added or multiplied together to make composite models >> (say, Gaussian + Step + Quadratic), and allow you to place bounds and/or >> constraints on any of the Parameters in the fit. >> >> Though there are many features you can use to set up models and their >> parameters, using these models (either built-in or one that you define >> yourself) for curve-fitting data is pretty straightforward. See >> http://lmfit.github.io/lmfit-py/builtin_models.html#example-1-fit-peaked-data-to-gaussian-lorentzian-and-voigt-profiles >> for a simple example. That example uses symmetric peaks, but the use of >> any Model for curve-fitting is basically the same. >> >> Hope that helps, >> >> --Matt Newville >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Thu Mar 19 07:04:17 2015 From: pawel.kw at gmail.com (=?UTF-8?Q?Pawe=C5=82_Kwa=C5=9Bniewski?=) Date: Thu, 19 Mar 2015 12:04:17 +0100 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Hi, Ok, now I have a problem. In the attached script I try to fit some example experimental data I have. The peak is actually a numerical derivative of a step function - this is why the values are so large. I tried different models, all except the ExponentialGaussianModel can handle the fit pretty well using the automatically guessed parameters. I don't really understand why. Is there something I'm missing here? Cheers, Pawel 2015-03-19 10:57 GMT+01:00 Pawe? Kwa?niewski : > Matt, > > I have one more question: the weights for fitting a model should be > 1./error_bar or the error_bar? The name "weights" suggests the former, but > I'd like to be sure. > > Cheers, > > Pawe? > > 2015-03-19 9:09 GMT+01:00 Pawe? Kwa?niewski : > >> Jonathan, Matt, >> >> Thank you for your answers. I must say that lmfit is already a good >> friend of mine - I use it for quite some time for fitting. Somehow I didn't >> think of looking there for asymmetric peak models. Thanks for pointing this >> out! I suppose this will solve my problem. >> >> Cheers, >> >> Pawel >> >> 2015-03-19 2:16 GMT+01:00 Matt Newville : >> >>> Pawel, >>> >>> On Wed, Mar 18, 2015 at 10:06 AM, Pawe? Kwa?niewski >>> wrote: >>> >>>> Hi All, >>>> >>>> I'm currently trying to fit some experimental data in the form of >>>> asymmetric peaks. In principle it's difficult to find a distribution >>>> describing this kind of data. My goal is to get the full width half maximum >>>> of the peak and the peak position. I tried to look for some ready solutions >>>> within scipy but could not find any. I did find a nice paper though: >>>> >>>> http://arxiv.org/abs/0711.4449 >>>> >>>> Since it's not so trivial (at least not for me) to implement this, I'd >>>> like to ask if anyone in the community has already done this (or something >>>> similar) and would like to share the code. >>>> >>>> Cheers, >>>> >>>> Pawel Kwasniewski >>>> >>> >>> You might find the lmfit module (http://lmfit.github.io/lmfit-py/) >>> useful. This is a module for least-squares minimization and curve-fitting, >>> built on top of scipy.optimize. It includes several built-in Models, a few >>> representing asymmetric peaks such as an exponentially damped Gaussian ( >>> http://lmfit.github.io/lmfit-py/builtin_models.html#exponentialgaussianmodel). >>> A quick look at the paper you referenced suggests this function might be >>> sufficient, but if this or one of the other built-in Models isn't exactly >>> what you're looking forward, it's very easy to make a new Model class from >>> a Python function that calculates and returns the model function. Lmfit >>> Models can also be added or multiplied together to make composite models >>> (say, Gaussian + Step + Quadratic), and allow you to place bounds and/or >>> constraints on any of the Parameters in the fit. >>> >>> Though there are many features you can use to set up models and their >>> parameters, using these models (either built-in or one that you define >>> yourself) for curve-fitting data is pretty straightforward. See >>> http://lmfit.github.io/lmfit-py/builtin_models.html#example-1-fit-peaked-data-to-gaussian-lorentzian-and-voigt-profiles >>> for a simple example. That example uses symmetric peaks, but the use of >>> any Model for curve-fitting is basically the same. >>> >>> Hope that helps, >>> >>> --Matt Newville >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- import lmfit import pylab as pl data = pl.loadtxt('example_data.txt') x = data[:,0] y = data[:,1] #model = lmfit.models.ExponentialGaussianModel() model = lmfit.models.GaussianModel() #model = lmfit.models.SkewedGaussianModel() #model = lmfit.models.VoigtModel() pars = model.guess(y,x=x) initguess = model.eval(params=pars,x=x) out = model.fit(y,pars,x=x) print(out.fit_report(min_correl=0.25)) fitres = model.eval(params=out.params,x=x) fig = pl.figure() ax = pl.subplot(111) ax.plot(x,y,'o') ax.plot(x,fitres,'-',lw=1) ax.plot(x,initguess,'--k',lw=1) pl.show() -------------- next part -------------- -2.000000000000000000e+00 -1.030000023022569076e+05 -1.990000000223520082e+00 -3.850000086055233987e+04 -1.980000000447029951e+00 2.440000054538901168e+04 -1.970000000670550033e+00 -3.200000071526428201e+03 -1.960000000894069894e+00 -9.700000216814485611e+03 -1.950000001117589976e+00 -3.010000067279546420e+04 -1.940000001341100067e+00 4.500000100584039501e+03 -1.930000001564619927e+00 2.190000048950899145e+04 -1.920000001788140009e+00 -5.130000114665804722e+04 -1.910000002011660092e+00 1.200000026822410473e+04 -1.900000002235169960e+00 4.980000111313003435e+04 -1.890000002458690043e+00 -1.590000035539693818e+04 -1.880000002682209903e+00 -1.900000042468816537e+04 -1.870000002905729986e+00 -4.200000093878436928e+03 -1.860000003129240076e+00 -2.090000046715698409e+04 -1.850000003352759936e+00 -1.840000041127696022e+04 -1.840000003576280019e+00 1.870000041798256279e+04 -1.830000003799800101e+00 3.200000071526428201e+03 -1.820000004023309970e+00 -2.460000054985941460e+04 -1.810000004246830052e+00 2.060000046045138151e+04 -1.800000004470349912e+00 -2.900000064820825355e+04 -1.790000004693869995e+00 -3.850000086055233987e+04 -1.780000004917380085e+00 5.070000113324684207e+04 -1.770000005140899946e+00 5.080000113548204536e+04 -1.760000005364420028e+00 1.500000033528013046e+04 -1.750000005587940111e+00 -3.430000076667389658e+04 -1.740000005811449979e+00 -2.500000055880021682e+04 -1.730000006034970061e+00 8.100000181051270374e+03 -1.720000006258489922e+00 8.400000187756873856e+03 -1.710000006482010004e+00 1.350000030175211759e+04 -1.700000006705520095e+00 -2.700000060350423610e+03 -1.690000006929039955e+00 -3.800000084937633346e+03 -1.680000007152560038e+00 4.000000089408035251e+02 -1.670000007376069906e+00 -3.020000067503066384e+04 -1.660000007599589988e+00 -5.000000111760043637e+02 -1.650000007823110071e+00 2.330000052080180467e+04 -1.640000008046629931e+00 -1.900000042468816673e+03 -1.630000008270140022e+00 3.430000076667389658e+04 -1.620000008493660104e+00 7.900000176580869265e+03 -1.610000008717179965e+00 -3.170000070855867671e+04 -1.600000008940700047e+00 -9.100000203403280466e+03 -1.590000009164209915e+00 1.920000042915856829e+04 -1.580000009387729998e+00 1.010000022575528783e+04 -1.570000009611250080e+00 -4.150000092760836560e+04 -1.560000009834769941e+00 1.060000023693129333e+04 -1.550000010058280031e+00 5.070000113324684207e+04 -1.540000010281799891e+00 -1.370000030622252052e+04 -1.530000010505319974e+00 -3.260000072867548442e+04 -1.520000010728840056e+00 2.880000064373785062e+04 -1.510000010952349925e+00 3.610000080690751929e+04 -1.500000011175870007e+00 -1.090000024363689590e+04 -1.490000011399390090e+00 -2.700000060350423519e+04 -1.480000011622909950e+00 -1.820000040680656093e+04 -1.470000011846420040e+00 -1.400000031292812309e+03 -1.460000012069939901e+00 -1.450000032410412678e+04 -1.450000012293459983e+00 -1.190000026598890327e+04 -1.440000012516980066e+00 -3.400000075996829764e+03 -1.430000012740489934e+00 -2.390000053421300981e+04 -1.420000012964010017e+00 1.620000036210254075e+04 -1.410000013187530099e+00 2.700000060350423519e+04 -1.400000013411049959e+00 -1.550000034645613596e+04 -1.390000013634560050e+00 2.350000052527220396e+04 -1.380000013858079910e+00 1.700000037998414882e+03 -1.370000014081599993e+00 -4.020000089855075203e+04 -1.360000014305110083e+00 3.140000070185307413e+04 -1.350000014528629944e+00 1.610000035986734110e+04 -1.340000014752150026e+00 -3.970000088737475016e+04 -1.330000014975670108e+00 -2.990000066832506127e+04 -1.320000015199179977e+00 5.500000122936047774e+03 -1.310000015422700059e+00 3.030000067726586349e+04 -1.300000015646219920e+00 1.310000029281131538e+04 -1.290000015869740002e+00 -1.770000039563055543e+04 -1.280000016093250093e+00 2.400000053644821037e+03 -1.270000016316769953e+00 7.000000156464061547e+02 -1.260000016540290035e+00 1.100000024587209509e+03 -1.250000016763809896e+00 1.000000022352008813e+02 -1.240000016987319986e+00 -3.170000070855867671e+04 -1.230000017210840069e+00 -3.100000069291227192e+03 -1.220000017434359929e+00 2.130000047609778630e+04 -1.210000017657880012e+00 2.300000051409620210e+04 -1.200000017881390102e+00 -1.370000030622252052e+04 -1.190000018104909962e+00 -2.300000051409620028e+03 -1.180000018328430045e+00 -2.170000048503858852e+04 -1.170000018551949905e+00 -2.000000044704017625e+02 -1.160000018775459996e+00 1.860000041574736315e+04 -1.150000018998980078e+00 -1.370000030622252052e+04 -1.140000019222499938e+00 1.620000036210254075e+04 -1.130000019446020021e+00 7.500000167640065229e+03 -1.120000019669529889e+00 -3.000000067056026182e+03 -1.110000019893049972e+00 -9.800000219049685256e+03 -1.100000020116570054e+00 -1.570000035092653707e+04 -1.090000020340089915e+00 1.150000025704810105e+04 -1.080000020563600005e+00 5.200000116230445201e+03 -1.070000020787120087e+00 8.000000178816070502e+02 -1.060000021010639948e+00 4.100000091643235464e+03 -1.050000021234150038e+00 -4.000000089408034910e+03 -1.040000021457669899e+00 2.810000062809144583e+04 -1.030000021681189981e+00 -4.000000089408034910e+03 -1.020000021904710064e+00 -1.420000031739852420e+04 -1.010000022128219932e+00 -1.090000024363689590e+04 -1.000000022351740014e+00 -1.860000041574736315e+04 -9.900000225752589866e-01 1.100000024587209509e+03 -9.800000227987769597e-01 2.480000055432981753e+04 -9.700000230222940445e-01 2.300000051409620210e+04 -9.600000232458110183e-01 -4.750000106172041706e+04 -9.500000234693289913e-01 -3.600000080467231328e+03 -9.400000236928459652e-01 1.630000036433774221e+04 -9.300000239163640492e-01 -5.200000116230445201e+03 -9.200000241398810230e-01 3.050000068173626642e+04 -9.100000243633989960e-01 5.900000131876851810e+03 -9.000000245869159698e-01 -8.900000198932877538e+03 -8.900000248104330547e-01 -1.120000025034249848e+04 -8.800000250339510277e-01 -2.730000061020983776e+04 -8.700000252574680015e-01 1.200000026822410473e+04 -8.600000254809859745e-01 4.210000094101957075e+04 -8.500000257045029484e-01 4.400000098348838037e+03 -8.400000259280200332e-01 -1.330000029728171648e+04 -8.300000261515380062e-01 -3.830000085608193331e+04 -8.200000263750549800e-01 -1.000000022352008818e+04 -8.100000265985729531e-01 2.420000054091861239e+04 -8.000000268220900379e-01 -3.700000082702432337e+03 -7.900000270456080109e-01 -2.880000064373785062e+04 -7.800000272691249847e-01 -2.440000054538901168e+04 -7.700000274926419586e-01 4.480000100136999390e+04 -7.600000277161600426e-01 1.140000025481289958e+04 -7.500000279396770164e-01 -3.520000078679070430e+04 -7.400000281631949894e-01 2.230000049844979367e+04 -7.300000283867119633e-01 2.990000066832506127e+04 -7.200000286102290481e-01 -4.500000100584039319e+04 -7.100000288337470211e-01 -3.880000086725794245e+04 -7.000000290572639949e-01 4.640000103713320277e+04 -6.900000292807819680e-01 2.180000048727379180e+04 -6.800000295042990528e-01 -1.070000023916649297e+04 -6.700000297278170258e-01 -2.850000063703224805e+04 -6.600000299513339996e-01 2.590000057891702818e+04 -6.500000301748509735e-01 3.200000071526427928e+04 -6.400000303983689465e-01 -3.460000077337949915e+04 -6.300000306218860313e-01 -4.300000096113637483e+03 -6.200000308454040043e-01 -2.270000050739059952e+04 -6.100000310689209782e-01 8.500000189992073501e+03 -6.000000312924389512e-01 6.200000138582454383e+03 -5.900000315159560360e-01 -8.200000183286470929e+03 -5.800000317394730098e-01 2.150000048056818923e+04 -5.700000319629909828e-01 7.400000165404864674e+03 -5.600000321865079567e-01 1.420000031739852420e+04 -5.500000324100260407e-01 -3.250000072644028478e+04 -5.400000326335430145e-01 -1.080000024140169444e+04 -5.300000328570599883e-01 1.970000044033457380e+04 -5.200000330805779614e-01 4.200000093878436928e+03 -5.100000333040950462e-01 -7.100000158699262101e+03 -5.000000335276130192e-01 -2.730000061020983776e+04 -4.900000337511299930e-01 1.000000022352008813e+02 -4.800000339746480216e-01 4.910000109748362593e+04 -4.700000341981649954e-01 8.200000183286470929e+03 -4.600000344216820247e-01 -2.100000046939218373e+04 -4.500000346451999977e-01 -1.860000041574736315e+04 -4.400000348687170271e-01 9.100000203403280466e+03 -4.300000350922350001e-01 3.870000086502273916e+04 -4.200000353157519739e-01 -3.600000080467231328e+03 -4.100000355392690032e-01 3.100000069291227192e+03 -4.000000357627869763e-01 1.640000036657294186e+04 -3.900000359863040056e-01 -1.800000040233615800e+04 -3.800000362098219786e-01 -2.850000063703224805e+04 -3.700000364333390079e-01 -8.200000183286470929e+03 -3.600000366568569810e-01 4.700000105054441519e+03 -3.500000368803740103e-01 1.740000038892495286e+04 -3.400000371038909841e-01 8.000000178816069820e+03 -3.300000373274090126e-01 -8.000000178816070502e+02 -3.200000375509259865e-01 1.800000040233615664e+03 -3.100000377744440150e-01 -1.670000037327854443e+04 -3.000000379979609888e-01 -7.800000174345667801e+03 -2.900000382214780181e-01 -5.600000125171249238e+03 -2.800000384449959911e-01 3.250000072644028478e+04 -2.700000386685130205e-01 1.740000038892495286e+04 -2.600000388920309935e-01 -3.820000085384673730e+04 -2.500000391155480228e-01 1.780000039786575508e+04 -2.400000393390659958e-01 2.450000054762421496e+04 -2.300000395625829974e-01 5.700000127406449792e+03 -2.200000397860999990e-01 -1.080000024140169444e+04 -2.100000400096179998e-01 -1.860000041574736315e+04 -2.000000402331350013e-01 1.400000031292812309e+04 -1.900000404566530021e-01 2.900000064820825355e+04 -1.800000406801700037e-01 5.900000131876851810e+03 -1.700000409036870053e-01 -1.210000027045930619e+04 -1.600000411272050060e-01 -5.500000122936047774e+03 -1.500000413507220076e-01 -1.610000035986734110e+04 -1.400000415742400084e-01 2.530000056550582303e+04 -1.300000417977570100e-01 4.420000098795878876e+04 -1.200000420212749969e-01 -1.410000031516332274e+04 -1.100000422447919984e-01 -1.700000037998414882e+03 -1.000000424683090000e-01 3.450000077114430314e+04 -9.000004269182680650e-02 4.850000108407242078e+04 -8.000004291534419665e-02 3.910000087396354502e+04 -7.000004313886169782e-02 4.080000091196195717e+04 -6.000004336237910185e-02 1.392000031113996229e+05 -5.000004358589649894e-02 2.265000050627299934e+05 -4.000004380941390297e-02 2.708000060529239709e+05 -3.000004403293130006e-02 3.696000082613024279e+05 -2.000004425644870062e-02 4.938000110374219366e+05 -1.000004447996620006e-02 5.917000132256835932e+05 -4.470348358154300184e-08 5.609000125372416805e+05 9.999955072999000549e-03 4.315000096448917757e+05 1.999995484948159999e-02 3.461000077360302093e+05 2.999995462596419943e-02 3.046000068084218656e+05 3.999995440244669825e-02 2.439000054516549280e+05 4.999995417892930116e-02 1.664000037193742464e+05 5.999995395541189713e-02 1.268000028342347068e+05 6.999995373189449310e-02 1.371000030644604121e+05 7.999995350837710295e-02 1.217000027202394704e+05 8.999995328485969892e-02 1.015000022687288874e+05 9.999995306134219775e-02 9.250000206756081025e+04 1.099999528378250019e-01 7.800000174345668347e+04 1.199999526143070011e-01 6.680000149311417772e+04 1.299999523907900134e-01 6.350000141935255670e+04 1.399999521672730118e-01 5.590000124947728909e+04 1.499999519437550111e-01 3.580000080020190944e+04 1.599999517202380095e-01 3.710000082925952302e+04 1.699999514967200087e-01 3.850000086055233987e+04 1.799999512732030071e-01 2.210000049397939438e+04 1.899999510496850064e-01 1.980000044256977344e+04 1.999999508261680048e-01 2.490000055656501718e+04 2.099999506026510032e-01 -5.700000127406449792e+03 2.199999503791330024e-01 -1.000000022352008727e+03 2.299999501556160009e-01 2.330000052080180467e+04 2.399999499320980001e-01 7.900000176580869265e+03 2.499999497085809985e-01 -3.700000082702432337e+03 2.599999494850639969e-01 2.700000060350423610e+03 2.699999492615460239e-01 2.080000046492178080e+04 2.799999490380289946e-01 1.410000031516332274e+04 2.899999488145110216e-01 -5.400000120700847219e+03 2.999999485909939922e-01 -1.450000032410412678e+04 3.099999483674760192e-01 4.400000098348838037e+03 3.199999481439589899e-01 1.550000034645613596e+04 3.299999479204420161e-01 -6.300000140817654938e+03 3.399999476969239876e-01 4.200000093878436928e+03 3.499999474734070137e-01 1.090000024363689590e+04 3.599999472498889852e-01 2.300000051409620028e+03 3.699999470263720114e-01 7.900000176580869265e+03 3.799999468028549821e-01 4.100000091643235464e+03 3.899999465793370090e-01 -1.190000026598890327e+04 3.999999463558199797e-01 6.500000145288056956e+03 4.099999461323020067e-01 1.260000028163530988e+04 4.199999459087849774e-01 -6.100000136347252919e+03 4.299999456852670043e-01 -6.000000134112052592e+02 4.399999454617499750e-01 1.020000022799048929e+04 4.499999452382330012e-01 1.200000026822410518e+03 4.599999450147149727e-01 -1.090000024363689590e+04 4.699999447911979988e-01 6.800000151993659529e+03 4.799999445676800258e-01 3.800000084937633346e+03 4.899999443441629965e-01 -1.820000040680656093e+04 4.999999441206460227e-01 7.300000163169664120e+03 5.099999438971279941e-01 2.390000053421300981e+04 5.199999436736110203e-01 5.800000129641650346e+03 5.299999434500930473e-01 -7.700000172110467247e+03 5.399999432265759625e-01 5.600000125171249238e+03 5.499999430030579894e-01 5.400000120700847219e+03 5.599999427795410156e-01 -2.070000046268658116e+04 5.699999425560240418e-01 1.020000022799048929e+04 5.799999423325059578e-01 8.500000189992073501e+03 5.899999421089889839e-01 -2.000000044704017455e+03 5.999999418854710109e-01 9.400000210108883039e+03 6.099999416619540371e-01 -1.130000025257769812e+04 6.199999414384369523e-01 -5.000000111760043637e+02 6.299999412149189792e-01 5.600000125171249238e+03 6.399999409914020054e-01 -6.800000151993659529e+03 6.499999407678840324e-01 -7.000000156464061547e+02 6.599999405443669476e-01 8.800000196697676074e+03 6.699999403208489746e-01 1.510000033751533192e+04 6.799999400973320007e-01 -7.000000156464061547e+03 6.899999398738150269e-01 -1.920000042915856829e+04 6.999999396502970539e-01 1.610000035986734110e+04 7.099999394267799691e-01 2.040000045598097859e+04 7.199999392032619960e-01 -1.380000030845772017e+04 7.299999389797450222e-01 -4.500000100584039501e+03 7.399999387562270492e-01 1.500000033528013046e+04 7.499999385327099644e-01 -6.900000154228860083e+03 7.599999383091929905e-01 -4.500000100584039501e+03 7.699999380856750175e-01 -2.600000058115222600e+03 7.799999378621580437e-01 -4.200000093878436928e+03 7.899999376386399597e-01 4.000000089408035251e+02 7.999999374151229858e-01 4.500000100584039501e+03 8.099999371916060120e-01 1.310000029281131538e+04 8.199999369680880390e-01 6.100000136347252919e+03 8.299999367445709542e-01 -7.300000163169664120e+03 8.399999365210529811e-01 -2.400000053644821037e+03 8.499999362975360073e-01 -1.000000022352008818e+04 8.599999360740180343e-01 -8.600000192227274965e+03 8.699999358505009495e-01 1.060000023693129333e+04 8.799999356269839756e-01 -1.400000031292812309e+03 8.899999354034660026e-01 4.600000102819240055e+03 8.999999351799490288e-01 3.200000071526428201e+03 9.099999349564309448e-01 -3.200000071526428201e+03 9.199999347329139709e-01 9.000000201168078320e+02 9.299999345093969971e-01 1.140000025481289958e+04 9.399999342858790241e-01 1.200000026822410518e+03 9.499999340623620503e-01 -8.000000178816069820e+03 9.599999338388439662e-01 3.300000073761628755e+03 9.699999336153269924e-01 -2.000000044704017625e+02 9.799999333918090194e-01 -3.500000078232030774e+03 9.899999331682920456e-01 -1.400000031292812309e+03 9.999999329447749608e-01 4.000000089408034910e+03 From jkhilmer at chemistry.montana.edu Thu Mar 19 12:02:32 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Thu, 19 Mar 2015 10:02:32 -0600 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: > Ok, now I have a problem. In the attached script I try to fit some example > experimental data I have. The peak is actually a numerical derivative of a > step function - this is why the values are so large. I tried different > models, all except the ExponentialGaussianModel can handle the fit pretty > well using the automatically guessed parameters. I don't really understand > why. Is there something I'm missing here? > It might not be your script. I can't reproduce the figure on Wikipedia ( http://en.wikipedia.org/wiki/File:EMG_Distribution_PDF.png) using the lmfit model. Specifically, cases with sigma != 1.0 don't seem right. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodecastillo at yahoo.es Thu Mar 19 14:22:54 2015 From: pablodecastillo at yahoo.es (pablo p del castillo) Date: Thu, 19 Mar 2015 18:22:54 +0000 (UTC) Subject: [SciPy-User] lbfgs and cobyla Message-ID: <1021818747.1053602.1426789374506.JavaMail.yahoo@mail.yahoo.com> Hello, I am checking how use the optimization functions in scipy. So i am looking for minimun area for a box with vol = 108, so i define for cobyla;def area(x):? ? return x[0]*x[1]+2.0*x[0]*x[2]+2.0*x[1]*x[2]def constr(x):? ? return x[0]*x[1]*x[2]-108.0x = [10,10,10] sol=fmin_cobyla(area,x,constr,rhoend=1e-6,maxfun=10000)And the solution is perfect, 6,6,3 if now i build the problem with lbfgs, i use lagrange multiplier, changing the objective to: def area(x):? ? return x[0]*x[1]+2.0*x[0]*x[2]+2.0*x[1]*x[2]+x[3]*constr(x)x = [10,10,10,10]?mybounds = [(0.00001,100), (0.00001,100), (0.00001,100), (0,1000)]fmin_l_bfgs_b(area, x,approx_grad=True,bounds=mybounds) getting bad results, what is my mistake?? Advanced thanks Pablo -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Thu Mar 19 14:55:39 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Thu, 19 Mar 2015 13:55:39 -0500 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Hi Pawel, On Thu, Mar 19, 2015 at 6:04 AM, Pawe? Kwa?niewski wrote: > Hi, > > Ok, now I have a problem. In the attached script I try to fit some example > experimental data I have. The peak is actually a numerical derivative of a > step function - this is why the values are so large. I tried different > models, all except the ExponentialGaussianModel can handle the fit pretty > well using the automatically guessed parameters. I don't really understand > why. Is there something I'm missing here? > Sorry for the trouble, but I'm not sure what is happening. It works OK for me with your data. With the attached script (trivial modifications of your script), I get printed results of [[Model]] Model(expgaussian) [[Fit Statistics]] # function evals = 124 # data points = 301 # variables = 4 chi-square = 139414753927.191 reduced chi-square = 469409945.883 [[Variables]] amplitude: 51178.7742 +/- 1.13e+03 (2.21%) (init= 48629) sigma: 0.02217557 +/- 0.001300 (5.86%) (init= 0.035) center: -0.03345298 +/- 0.001281 (3.83%) (init=-0.005000045) gamma: 21.0962075 +/- 1.320993 (6.26%) (init= 1) [[Correlations]] (unreported correlations are < 0.250) C(center, gamma) = 0.771 C(sigma, gamma) = 0.616 C(amplitude, gamma) = -0.530 C(sigma, center) = 0.475 C(amplitude, center) = -0.408 and the attached figure of data and best-fit. [image: Inline image 1] If you're getting very different results, please be more specific. Cheers, --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: example_expgauss.png Type: image/png Size: 40509 bytes Desc: not available URL: -------------- next part -------------- import lmfit import pylab as pl data = pl.loadtxt('example_data.txt') x = data[:,0] y = data[:,1] model = lmfit.models.ExponentialGaussianModel() pars = model.guess(y,x=x) out = model.fit(y,pars,x=x) print(out.fit_report(min_correl=0.25)) fig = pl.figure() ax = pl.subplot(111) ax.plot(x,y,'o') ax.plot(x,fitres,'-',lw=1) ax.plot(x,out.best_fit,'-',lw=1) pl.savefig('example_expgauss.png') pl.show() From newville at cars.uchicago.edu Thu Mar 19 15:01:57 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Thu, 19 Mar 2015 14:01:57 -0500 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Hi Jonathon, On Thu, Mar 19, 2015 at 11:02 AM, jkhilmer at chemistry.montana.edu < jkhilmer at chemistry.montana.edu> wrote: > > Ok, now I have a problem. In the attached script I try to fit some example >> experimental data I have. The peak is actually a numerical derivative of a >> step function - this is why the values are so large. I tried different >> models, all except the ExponentialGaussianModel can handle the fit pretty >> well using the automatically guessed parameters. I don't really understand >> why. Is there something I'm missing here? >> > > It might not be your script. I can't reproduce the figure on Wikipedia ( > http://en.wikipedia.org/wiki/File:EMG_Distribution_PDF.png) using the > lmfit model. Specifically, cases with sigma != 1.0 don't seem right. > > Jonathan > Hmm, not sure what you're seeing. From the link you give, this script #!/usr/bin/env python from numpy import linspace import matplotlib.pyplot as plt from lmfit.lineshapes import expgaussian x = linspace(-4, 6, 101) red = expgaussian(x, amplitude=1, center= 0.0, sigma=1.0, gamma=1.0) green = expgaussian(x, amplitude=1, center=-2.0, sigma=1.0, gamma=1.0) blue = expgaussian(x, amplitude=1, center= 0.0, sigma=3.0, gamma=1.0) yellow = expgaussian(x, amplitude=1, center=-3.0, sigma=1.0, gamma=0.25) plt.plot(x, red, 'r-') plt.plot(x, green, 'g-') plt.plot(x, blue, 'b-') plt.plot(x, yellow, 'y-') plt.savefig('test_expgauss.png') plt.show() ######################################### gives a plot that looks pretty similar to the one at the wikipedia page (except for the nice labels): [image: Inline image 2] Do you see something different? If so, what do you see? Cheers, --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_expgauss.png Type: image/png Size: 53745 bytes Desc: not available URL: From jkhilmer at chemistry.montana.edu Thu Mar 19 15:26:49 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Thu, 19 Mar 2015 13:26:49 -0600 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: >Do you see something different? If so, what do you see? Mine is definitely different. Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32 scipy 0.14.0 numpy 1.8.1 lmfit 0.8.3 Using your test script: [image: Inline image 1] Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_expgauss.png Type: image/png Size: 47924 bytes Desc: not available URL: From newville at cars.uchicago.edu Thu Mar 19 20:41:16 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Thu, 19 Mar 2015 19:41:16 -0500 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Jonathon, Pawel, On Thu, Mar 19, 2015 at 2:26 PM, jkhilmer at chemistry.montana.edu < jkhilmer at chemistry.montana.edu> wrote: > >Do you see something different? If so, what do you see? > > Mine is definitely different. > Ah, sorry the very poor memory. There was a fix to the expgaussian() function a month ago, after the 0.8.3 release (and thanks to Tim Spillane for the fix!). That explains the problem you're seeing, and might explain why Pawel was having trouble too. The fix is fairly simple (change `arg2` to have `/(s2*sigma)` instead of `/s`, see https://github.com/lmfit/lmfit-py/blob/master/lmfit/lineshapes.py#L116), and should be orthogonal to any other changes in the current branch. --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkhilmer at chemistry.montana.edu Thu Mar 19 21:30:53 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Thu, 19 Mar 2015 19:30:53 -0600 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: > > > Ah, sorry the very poor memory. There was a fix to the expgaussian() > function a month ago, after the 0.8.3 release (and thanks to Tim Spillane > for the fix!). That explains the problem you're seeing, and might > explain why Pawel was having trouble too. The fix is fairly simple > (change `arg2` to have `/(s2*sigma)` instead of `/s`, see > https://github.com/lmfit/lmfit-py/blob/master/lmfit/lineshapes.py#L116), > and should be orthogonal to any other changes in the current branch. > > Thanks for checking into this. Do you have any thoughts on the analysis by Kalambet and Tikhonov in the paper I linked previously? Specifically, in the code: gss = gamma*sigma*sigma arg1 = gamma*(center +gss/2.0 - x) arg2 = (center + gss - x)/(s2*sigma) return amplitude*(gamma/2) * exp(arg1) * erfc(arg2) The exp(arg1) factor is much too large and erfc(arg2) is much too small, when x<1/gamma ? Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Thu Mar 19 23:15:38 2015 From: newville at cars.uchicago.edu (Matt Newville) Date: Thu, 19 Mar 2015 22:15:38 -0500 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Hi Jonathon, On Thu, Mar 19, 2015 at 8:30 PM, jkhilmer at chemistry.montana.edu < jkhilmer at chemistry.montana.edu> wrote: > >> Ah, sorry the very poor memory. There was a fix to the expgaussian() >> function a month ago, after the 0.8.3 release (and thanks to Tim Spillane >> for the fix!). That explains the problem you're seeing, and might >> explain why Pawel was having trouble too. The fix is fairly simple >> (change `arg2` to have `/(s2*sigma)` instead of `/s`, see >> https://github.com/lmfit/lmfit-py/blob/master/lmfit/lineshapes.py#L116), >> and should be orthogonal to any other changes in the current branch. >> >> > Thanks for checking into this. Do you have any thoughts on the analysis > by Kalambet and Tikhonov in the paper I linked previously? Specifically, > in the code: > > gss = gamma*sigma*sigma > arg1 = gamma*(center +gss/2.0 - x) > arg2 = (center + gss - x)/(s2*sigma) > return amplitude*(gamma/2) * exp(arg1) * erfc(arg2) > > The exp(arg1) factor is much too large and erfc(arg2) is much too small, > when x<1/gamma ? > > > Jonathan > I haven't looked at the paper in great detail, and wasn't aware of (and/or never experienced myself) the potential instability -- but I haven't used this function much myself. If I understand correctly, the recommendation is that a version of the exponentially modified Gaussian that used Eq. 6 from the Kalambet, et al article (and scipy.special.erfcx()) would be better behaved. Is that correct, and the whole story? A Pull Request or a patch for candidate replacement function based on this would be greatly appreciated. --Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.kw at gmail.com Fri Mar 20 04:43:52 2015 From: pawel.kw at gmail.com (=?UTF-8?Q?Pawe=C5=82_Kwa=C5=9Bniewski?=) Date: Fri, 20 Mar 2015 09:43:52 +0100 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: Hi Matt, Jonathan, 2015-03-20 4:15 GMT+01:00 Matt Newville : > Hi Jonathon, > > On Thu, Mar 19, 2015 at 8:30 PM, jkhilmer at chemistry.montana.edu < > jkhilmer at chemistry.montana.edu> wrote: > >> >>> Ah, sorry the very poor memory. There was a fix to the expgaussian() >>> function a month ago, after the 0.8.3 release (and thanks to Tim Spillane >>> for the fix!). That explains the problem you're seeing, and might >>> explain why Pawel was having trouble too. The fix is fairly simple >>> (change `arg2` to have `/(s2*sigma)` instead of `/s`, see >>> https://github.com/lmfit/lmfit-py/blob/master/lmfit/lineshapes.py#L116), >>> and should be orthogonal to any other changes in the current branch. >>> >>> That was the problem! Thanks! I'm sorry, I should also attach the fit result figure to my post. But now it works like a charm. Thanks! Pawel > >> Thanks for checking into this. Do you have any thoughts on the analysis >> by Kalambet and Tikhonov in the paper I linked previously? Specifically, >> in the code: >> >> gss = gamma*sigma*sigma >> arg1 = gamma*(center +gss/2.0 - x) >> arg2 = (center + gss - x)/(s2*sigma) >> return amplitude*(gamma/2) * exp(arg1) * erfc(arg2) >> >> The exp(arg1) factor is much too large and erfc(arg2) is much too small, >> when x<1/gamma ? >> >> >> Jonathan >> > > I haven't looked at the paper in great detail, and wasn't aware of (and/or > never experienced myself) the potential instability -- but I haven't used > this function much myself. If I understand correctly, the recommendation > is that a version of the exponentially modified Gaussian that used Eq. 6 > from the Kalambet, et al article (and scipy.special.erfcx()) would be > better behaved. Is that correct, and the whole story? > > A Pull Request or a patch for candidate replacement function based on > this would be greatly appreciated. > > --Matt > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Fri Mar 20 06:03:16 2015 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Fri, 20 Mar 2015 10:03:16 +0000 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: FWIW, this distribution has been implemented in scipy a couple of weeks ago as `exponnorm` --- it's only available in the dev version at the moment. So `exponnorm.fit` may work --- if it doesn't, a patch would be welcome :-) On Fri, Mar 20, 2015 at 8:43 AM, Pawe? Kwa?niewski wrote: > Hi Matt, Jonathan, > > > > > > 2015-03-20 4:15 GMT+01:00 Matt Newville : >> >> Hi Jonathon, >> >> On Thu, Mar 19, 2015 at 8:30 PM, jkhilmer at chemistry.montana.edu >> wrote: >>>> >>>> >>>> Ah, sorry the very poor memory. There was a fix to the expgaussian() >>>> function a month ago, after the 0.8.3 release (and thanks to Tim Spillane >>>> for the fix!). That explains the problem you're seeing, and might explain >>>> why Pawel was having trouble too. The fix is fairly simple (change `arg2` >>>> to have `/(s2*sigma)` instead of `/s`, see >>>> https://github.com/lmfit/lmfit-py/blob/master/lmfit/lineshapes.py#L116), and >>>> should be orthogonal to any other changes in the current branch. >>>> > > That was the problem! Thanks! I'm sorry, I should also attach the fit result > figure to my post. But now it works like a charm. > > Thanks! > > Pawel > >>> >>> >>> Thanks for checking into this. Do you have any thoughts on the analysis >>> by Kalambet and Tikhonov in the paper I linked previously? Specifically, in >>> the code: >>> >>> gss = gamma*sigma*sigma >>> arg1 = gamma*(center +gss/2.0 - x) >>> arg2 = (center + gss - x)/(s2*sigma) >>> return amplitude*(gamma/2) * exp(arg1) * erfc(arg2) >>> >>> The exp(arg1) factor is much too large and erfc(arg2) is much too small, >>> when x<1/gamma ? >>> >>> >>> Jonathan >> >> >> I haven't looked at the paper in great detail, and wasn't aware of (and/or >> never experienced myself) the potential instability -- but I haven't used >> this function much myself. If I understand correctly, the recommendation >> is that a version of the exponentially modified Gaussian that used Eq. 6 >> from the Kalambet, et al article (and scipy.special.erfcx()) would be better >> behaved. Is that correct, and the whole story? >> >> A Pull Request or a patch for candidate replacement function based on >> this would be greatly appreciated. >> >> --Matt >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From alimanfoo at googlemail.com Fri Mar 20 08:19:04 2015 From: alimanfoo at googlemail.com (Alistair Miles) Date: Fri, 20 Mar 2015 12:19:04 +0000 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: FWIW I use PyCharm for any heavy coding (refactoring support is particularly useful) but IPython notebooks for informal testing and exploration, in particular the %autoreload magic is very helpful as you can have any changes you make to functions/classes you are working on in PyCharm be immediately effective in the notebook without having to restart or reload modules manually. Hth, Alistair On Saturday, March 14, 2015, Brian Merchant wrote: > I need to write about 2000 lines worth of code (based on my last > implementation of the project I am working on). I anticipate that I'll be > using profilers (including line profilers), numpy, Numba, possibly Cython > and almost definitely matplotlib animations. I will need to write my own > classes, so my program isn't just going to be one long script. > > > Previously, I used Spyder for development, but I am wondering if it makes > sense for one to use an IPython notebook alone. I am considering that > option so that I force myself to program "literately" -- lots of headings, > and nice comments. Graphs embedded to show the effects of optimizations, > and so forth. > > So far, I know that IPython might have some trouble with integrating in > matplotlib animations, but that too is possible with a little research. > Otherwise, I should be able to do everything else quite easily, including > debugging, according to the IPython documentation. In fact, writing > parallel code in IPython might be a very nice idea given how it seems to > offer a lot of functionality in that direction. > > > Do people use IPython (notebooks) as a complete IDE often? If not, why not? > > Kind regards, > Brian > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Web: http://purl.org/net/aliman Email: alimanfoo at gmail.com Tel: +44 (0)1865 287721 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkhilmer at chemistry.montana.edu Fri Mar 20 11:49:10 2015 From: jkhilmer at chemistry.montana.edu (jkhilmer at chemistry.montana.edu) Date: Fri, 20 Mar 2015 09:49:10 -0600 Subject: [SciPy-User] Asymmetric peak fitting In-Reply-To: References: Message-ID: > I haven't looked at the paper in great detail, and wasn't aware of (and/or > never experienced myself) the potential instability -- but I haven't used > this function much myself. If I understand correctly, the recommendation > is that a version of the exponentially modified Gaussian that used Eq. 6 > from the Kalambet, et al article (and scipy.special.erfcx()) would be > better behaved. Is that correct, and the whole story? > > A Pull Request or a patch for candidate replacement function based on > this would be greatly appreciated. > > The attached test script should show the issue. The form of the equation using erfcx is likewise problematic, but at the other limit (small gamma). They recommend switching between the two equations at x = 1/gamma. [image: Inline image 1] I did this: y2 = emg_kalambet2(x, u, s, a, gamma) y6 = emg_kalambet6(x, u, s, a, gamma) y = scipy.zeros(shape=x.shape) y[scipy.where(x>1./gamma)] = y2 y[scipy.where(x<=1./gamma)] = y6 It's very likely that's not the optimal approach. Maybe this whole issue doesn't matter; the details of floating point numbers are really outside my area of expertise. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kalambet_test.png Type: image/png Size: 45872 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kalambet_test.py Type: text/x-python Size: 1875 bytes Desc: not available URL: From ghisvail at gmail.com Fri Mar 20 12:55:15 2015 From: ghisvail at gmail.com (Ghislain Vaillant) Date: Fri, 20 Mar 2015 16:55:15 +0000 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? Message-ID: On Saturday, March 14, 2015, Brian Merchant wrote: > I need to write about 2000 lines worth of code (based on my last > implementation of the project I am working on). I anticipate that I'll be > using profilers (including line profilers), numpy, Numba, possibly Cython > and almost definitely matplotlib animations. I will need to write my own > classes, so my program isn't just going to be one long script. > > > Previously, I used Spyder for development, but I am wondering if it makes > sense for one to use an IPython notebook alone. I am considering that > option so that I force myself to program "literately" -- lots of headings, > and nice comments. Graphs embedded to show the effects of optimizations, > and so forth. > > So far, I know that IPython might have some trouble with integrating in > matplotlib animations, but that too is possible with a little research. > Otherwise, I should be able to do everything else quite easily, including > debugging, according to the IPython documentation. In fact, writing > parallel code in IPython might be a very nice idea given how it seems to > offer a lot of functionality in that direction. > > > Do people use IPython (notebooks) as a complete IDE often? If not, why not? > > Kind regards, > Brian > Hi Brian, Do you find anything wrong with Spyder ? I use both IEP [1] and Spyder extensively and find them pretty good for the use case you describe. However if you're looking for more project-style features like refactoring tools, project views, VCS integration, etc..., a full-featured IDE may be more appropriate, like PyCharm as kindly suggested by Alistair. Cheers, Ghis [1] http://www.iep-project.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Sat Mar 21 11:47:22 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Sat, 21 Mar 2015 08:47:22 -0700 Subject: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE? In-Reply-To: References: Message-ID: Hi all, Thank you for the wonderful discussion. I especially enjoyed learning about IEP (which I had not heard of), and PyCharm (which I had simply disregarded in the past)! Ghis: I don't find much wrong with Spyder, apart from occasional stability issues which are quickly fixed with a restart of the program, and thus are hardly a hassle. In the end, I have decided to use PyCharm (which I can get an academic license for!) *primarily* because it has nice built-in VCS integration -- just a convenience really, than an absolutely killer function, but one that is valuable to me. I wanted to write code in a literate, and organized fashion. I thought IPython notebooks might be a good idea, but as mentioned by others, there are some issues with it. However, I think using a VCS to track and comment on changes made to code is just as good! PyCharm has excellent integration with git, including nice visual views of branches, merges etc. *PyCharm also has integrated IPython notebook compatibility now*, so if I occasionally want to write a couple of tests in a more literate fashion, I can do so. Then of course, there are all the other IDE features that PyCharm has covered in a more stable/polished fashion than Spyder. Spyder doesn't have refactoring functionality (as far as I know?), so PyCharm is quite nice there particularly. Brian On Fri, Mar 20, 2015 at 9:55 AM, Ghislain Vaillant wrote: > On Saturday, March 14, 2015, Brian Merchant wrote: > > > I need to write about 2000 lines worth of code (based on my last > > implementation of the project I am working on). I anticipate that I'll be > > using profilers (including line profilers), numpy, Numba, possibly Cython > > and almost definitely matplotlib animations. I will need to write my own > > classes, so my program isn't just going to be one long script. > > > > > > Previously, I used Spyder for development, but I am wondering if it makes > > sense for one to use an IPython notebook alone. I am considering that > > option so that I force myself to program "literately" -- lots of > headings, > > and nice comments. Graphs embedded to show the effects of optimizations, > > and so forth. > > > > So far, I know that IPython might have some trouble with integrating in > > matplotlib animations, but that too is possible with a little research. > > Otherwise, I should be able to do everything else quite easily, including > > debugging, according to the IPython documentation. In fact, writing > > parallel code in IPython might be a very nice idea given how it seems to > > offer a lot of functionality in that direction. > > > > > > Do people use IPython (notebooks) as a complete IDE often? If not, why > not? > > > > Kind regards, > > Brian > > > > > Hi Brian, > > Do you find anything wrong with Spyder ? I use both IEP [1] and Spyder > extensively > and find them pretty good for the use case you describe. However if you're > looking for > more project-style features like refactoring tools, project views, VCS > integration, etc..., > a full-featured IDE may be more appropriate, like PyCharm as kindly > suggested by Alistair. > > > Cheers, > Ghis > > > > > [1] http://www.iep-project.org/ > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yw5aj at virginia.edu Sat Mar 21 12:32:34 2015 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Sat, 21 Mar 2015 12:32:34 -0400 Subject: [SciPy-User] optimize - initial step size too small (for noisy objective) In-Reply-To: References: Message-ID: Hi Neal, Have you tried the options keyword? For example, scipy.optimize.minimize(func, np.ones(3), args=(arg1, arg2, -1), method='SLSQP', bounds=bounds, options={'eps': 1e-3}) For more details on what options are available, see: http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.show_options.html#scipy.optimize.show_options Shawn On Wed, Mar 18, 2015 at 1:36 PM, Neal Becker wrote: > I tried using scipy.optimize.minimize, with a single variable, with a bound > of [(-2,2)]. The initial step was only 1e-8. The objective function could > be (a little) noisy - this seemed to cause minimize to never attempt a step > in that direction again. > > Is there some way to cause it to take a bigger step, e.g., 1e-3? > > I'm trying brent now, probably a better choice. > > -- > Those who fail to understand recursion are doomed to repeat it > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From camillechambon at yahoo.fr Sat Mar 21 15:00:51 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Sat, 21 Mar 2015 20:00:51 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently Message-ID: <550DBFE3.50101@yahoo.fr> Hello, I have a function: def my_function(x1, x2): y1 = compute_y1(x1) y2 = compute_y2(x2) x1_new = compute_new_x1(x1, y1, y2) x2_new = compute_new_x2(x2, y1, y2) return x1_new, x2_new, y1, y2 I would like to find the convergent values of y1 and y2, that is when (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. By now, I initialize x1 and x2 and I call main_function numerous times: epsilon = 0.01 x1, x2 = 0.1, 0.1 x1_new, x2_new = 100.0, 100.0 while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= epsilon: x1, x2 = x1_new, x2_new x1_new, x2_new, y1, y2 = my_function(x1, x2) print 'y1 = ', y1 print 'y2 = ', y2 It works. But I wonder if it's the most efficient method. I read about scipy.optimize.minimize, but I can not see how it could be applied to my problem. Thanks in advance for your help. Cheers, Camille From rnelsonchem at gmail.com Sat Mar 21 17:28:29 2015 From: rnelsonchem at gmail.com (Ryan Nelson) Date: Sat, 21 Mar 2015 17:28:29 -0400 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <550DBFE3.50101@yahoo.fr> References: <550DBFE3.50101@yahoo.fr> Message-ID: For scipy.optimize.minimize, the function needs to take all of the values as a tuple, and it should return a single y value. For a silly example: #### def func(x): x1 = x[0] x2 = x[1] y = x1**2 + x2**2 return y fit = scipy.optimize.minimize(func, [10, 3]) #### Right now, your function spits out two y values, which that function can't handle. The minimizer will do the work of "compute_new_x1", etc. I hope this helps you with your problem. If not, it might be useful to have a simple, but complete, example. Ryan On Sat, Mar 21, 2015 at 3:00 PM, Camille Chambon wrote: > Hello, > > I have a function: > > def my_function(x1, x2): > y1 = compute_y1(x1) > y2 = compute_y2(x2) > x1_new = compute_new_x1(x1, y1, y2) > x2_new = compute_new_x2(x2, y1, y2) > return x1_new, x2_new, y1, y2 > > I would like to find the convergent values of y1 and y2, that is when > (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. > > By now, I initialize x1 and x2 and I call main_function numerous times: > > epsilon = 0.01 > x1, x2 = 0.1, 0.1 > x1_new, x2_new = 100.0, 100.0 > while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= epsilon: > x1, x2 = x1_new, x2_new > x1_new, x2_new, y1, y2 = my_function(x1, x2) > print 'y1 = ', y1 > print 'y2 = ', y2 > > It works. But I wonder if it's the most efficient method. > > I read about scipy.optimize.minimize, but I can not see how it could be > applied to my problem. > > Thanks in advance for your help. > > Cheers, > > Camille > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yw5aj at virginia.edu Sat Mar 21 17:33:54 2015 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Sat, 21 Mar 2015 17:33:54 -0400 Subject: [SciPy-User] Numerical inverse Laplace Transform? Message-ID: Dear all, Trying my luck here - is anyone aware of any numerical laplace transform that is callable by python? Thanks! Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From yw5aj at virginia.edu Sat Mar 21 17:34:33 2015 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Sat, 21 Mar 2015 17:34:33 -0400 Subject: [SciPy-User] Numerical inverse Laplace Transform? In-Reply-To: References: Message-ID: Sorry - I meant to say numerical inverse laplace transform in the main text, not the laplace transform. Shawn On Sat, Mar 21, 2015 at 5:33 PM, Yuxiang Wang wrote: > Dear all, > > Trying my luck here - is anyone aware of any numerical laplace > transform that is callable by python? > > Thanks! > > Shawn > > -- > Yuxiang "Shawn" Wang > Gerling Research Lab > University of Virginia > yw5aj at virginia.edu > +1 (434) 284-0836 > https://sites.google.com/a/virginia.edu/yw5aj/ -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From vanforeest at gmail.com Sun Mar 22 06:15:50 2015 From: vanforeest at gmail.com (nicky van foreest) Date: Sun, 22 Mar 2015 11:15:50 +0100 Subject: [SciPy-User] Numerical inverse Laplace Transform? In-Reply-To: References: Message-ID: Perhaps this is of use to you: http://nicky.vanforeest.com/queueing/euler/euler.html On 21 March 2015 at 22:34, Yuxiang Wang wrote: > Sorry - I meant to say numerical inverse laplace transform in the main > text, not the laplace transform. > > Shawn > > On Sat, Mar 21, 2015 at 5:33 PM, Yuxiang Wang wrote: > > Dear all, > > > > Trying my luck here - is anyone aware of any numerical laplace > > transform that is callable by python? > > > > Thanks! > > > > Shawn > > > > -- > > Yuxiang "Shawn" Wang > > Gerling Research Lab > > University of Virginia > > yw5aj at virginia.edu > > +1 (434) 284-0836 > > https://sites.google.com/a/virginia.edu/yw5aj/ > > > > -- > Yuxiang "Shawn" Wang > Gerling Research Lab > University of Virginia > yw5aj at virginia.edu > +1 (434) 284-0836 > https://sites.google.com/a/virginia.edu/yw5aj/ > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Sun Mar 22 14:54:36 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Sun, 22 Mar 2015 11:54:36 -0700 Subject: [SciPy-User] Setting up the model for a large nonlinear-in-the-parameters ODE system Message-ID: Hi all, I have a use case where I need to model a large ODE system. Here's how I designed the implementation of my model in the past: - create a class called MyODESystem (except, with a more relevant name, generic name just for the purposes of conceptual discussion) - MyODESystem would take *a lot* of parameters -- these would define things like baseline values for various parameters that need a function to calculate their actual parameters - MyODESystem would have *many* methods that basically had this form: def calculate_parameter(self, some_relevant_variable, another_relevant_variable, ...): return (self.parameter_baseline_value + some_relevant_variable)^2 - another_relevant_variable + .... The purpose of these methods was to calculate some nonlinear-in-the-parameter variable. In the example, I just show how some baseline value given to the function initially along with other variable values would be combined in some way to calculate the parameter value -- the form of the math is not important as its just for example purposes. - MyODESystem would have a numpy array data structure where each row was the value of the different ODE variables at some time, and there were as many rows as there were sampling time point - finally, there would be a function, calculate_rhs (which would be supplied to scipy.integrate.odeint) which would call all these different methods, given initial conditions, and return the RHS of the ODE system Overall, the class was quite messy and kludge-y, partially because I had started to prematurely optimize (I know, I know, the root of all evil :( ) and gone "Cython from the very start". Every time I wanted to change the RHS, I would go in and manually change equations, deleting parameters and variables that were no longer necessary, manually changing methods, or if I needed new variables, I would lazily add them "some_var = 2" style, where some_var can't be accessed nicely without going to the source code and changing what it is equal to... Honestly, I am not sure if there is a better way to set up an ODE system manager while we keep the discussion at the conceptual level alone. The only thing to change maybe would be to force myself to not be lazy when I need to change the RHS/methods/variables, and so forth, and just force myself to do it in a way that is always transparent (no hidden constants, and so forth)> Still, I was wondering if there were any tips you guys might have for setting up and managing classes for large ODE systems? One way would be to PyDSTool approach ( http://www.ni.gsu.edu/~rclewley/PyDSTool/Tutorial/Tutorial_linear.html) whether I simply use PyDSTool or re-invent a wheel similar to it. Any other interesting ways? Kind regards, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From athanastasiou at gmail.com Sun Mar 22 19:08:28 2015 From: athanastasiou at gmail.com (Athanasios Anastasiou) Date: Sun, 22 Mar 2015 23:08:28 +0000 Subject: [SciPy-User] Setting up the model for a large nonlinear-in-the-parameters ODE system In-Reply-To: References: Message-ID: Hello Brian My use case was about coupled oscillators. But, because it was impossible to know how the oscillators were going to be needed to be coupled, I set up a "feedback matrix", kind of like an adjacency matrix which determined connections and their weights. Each oscillator though was modelled on its own and run in its own environment. If you were to look at a specific matrix you might have found an alternative expression that would be more efficient but you would lose the flexibility of connecting the models in any way you like. So, what I would say is, try to break your model down to smaller individual models that can be interconnected, rather than having one big super optimised but rigid one. At least while you are developing it. The interconnection of the models doesn't have to be via a matrix, it could just be hardwired. Hope this helps All the best AA On 22 Mar 2015 18:54, "Brian Merchant" wrote: > Hi all, > > I have a use case where I need to model a large ODE system. Here's how I > designed the implementation of my model in the past: > > > - create a class called MyODESystem (except, with a more relevant > name, generic name just for the purposes of conceptual discussion) > - MyODESystem would take *a lot* of parameters -- these would define > things like baseline values for various parameters that need a function to > calculate their actual parameters > - MyODESystem would have *many* methods that basically had this form: > > def calculate_parameter(self, some_relevant_variable, > another_relevant_variable, ...): > return (self.parameter_baseline_value + some_relevant_variable)^2 > - another_relevant_variable + .... > > The purpose of these methods was to calculate some > nonlinear-in-the-parameter variable. In the example, I just show how some > baseline value given to the function initially along with other variable > values would be combined in some way to calculate the parameter value -- > the form of the math is not important as its just for example purposes. > - MyODESystem would have a numpy array data structure where each row > was the value of the different ODE variables at some time, and there were > as many rows as there were sampling time point > - finally, there would be a function, calculate_rhs (which would be > supplied to scipy.integrate.odeint) which would call all these different > methods, given initial conditions, and return the RHS of the ODE system > > Overall, the class was quite messy and kludge-y, partially because I had > started to prematurely optimize (I know, I know, the root of all evil :( ) > and gone "Cython from the very start". Every time I wanted to change the > RHS, I would go in and manually change equations, deleting parameters and > variables that were no longer necessary, manually changing methods, or if I > needed new variables, I would lazily add them "some_var = 2" style, where > some_var can't be accessed nicely without going to the source code and > changing what it is equal to... > > Honestly, I am not sure if there is a better way to set up an ODE system > manager while we keep the discussion at the conceptual level alone. The > only thing to change maybe would be to force myself to not be lazy when I > need to change the RHS/methods/variables, and so forth, and just force > myself to do it in a way that is always transparent (no hidden constants, > and so forth)> > > Still, I was wondering if there were any tips you guys might have for > setting up and managing classes for large ODE systems? One way would be to > PyDSTool approach ( > http://www.ni.gsu.edu/~rclewley/PyDSTool/Tutorial/Tutorial_linear.html) > whether I simply use PyDSTool or re-invent a wheel similar to it. > > Any other interesting ways? > > Kind regards, > Brian > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From camillechambon at yahoo.fr Mon Mar 23 05:52:48 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Mon, 23 Mar 2015 10:52:48 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: References: <550DBFE3.50101@yahoo.fr> Message-ID: <550FE270.6000209@yahoo.fr> Thanks. Le 21/03/2015 22:28, Ryan Nelson a ?crit : > For scipy.optimize.minimize, the function needs to take all of the > values as a tuple, and it should return a single y value. For a silly > example: > #### > def func(x): > x1 = x[0] > x2 = x[1] > y = x1**2 + x2**2 > return y > > fit = scipy.optimize.minimize(func, [10, 3]) > #### > Right now, your function spits out two y values, which that function > can't handle. The minimizer will do the work of "compute_new_x1", etc. OK. I understand that the minimizer will do the work of "compute_new_x1" and "compute_new_x2". But I do need to return several y values (in fact more than 2). So how could I achieve that? > > I hope this helps you with your problem. If not, it might be useful to > have a simple, but complete, example. The (simplified) complete example is: ##### code ########## from __future__ import division def compute_var1(x1, x2): return 2 * x1 + x2 def compute_var2(x1, x2): return 2 * x1 - x2 def compute_new_x1(x1): return x1 + 10 def compute_new_x2(x2): return x2 + 10 x1_new, x2_new = 100.0, 100.0 x1, x2 = 0.1, 0.1 step = 0 while abs((x1_new - x1)/x1) >= 0.01 or abs((x2_new - x2)/x2) >= 0.01: x1, x2 = x1_new, x2_new var1 = compute_var1(x1, x2) var2 = compute_var2(x1, x2) x1_new = compute_new_x1(x1) x2_new = compute_new_x2(x2) step += 1 print 'var1=', var1, 'var2=', var2, 'in', step, 'steps' #################### Which prints: var1= 3030.0 var2= 1010.0 in 92 steps How could I make this more efficient ? Is there a scipy (or other) function for this purpose? Thanks again. Cheers, Camille > > Ryan > > > On Sat, Mar 21, 2015 at 3:00 PM, Camille Chambon > > wrote: > > Hello, > > I have a function: > > def my_function(x1, x2): > y1 = compute_y1(x1) > y2 = compute_y2(x2) > x1_new = compute_new_x1(x1, y1, y2) > x2_new = compute_new_x2(x2, y1, y2) > return x1_new, x2_new, y1, y2 > > I would like to find the convergent values of y1 and y2, that is when > (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. > > By now, I initialize x1 and x2 and I call main_function numerous > times: > > epsilon = 0.01 > x1, x2 = 0.1, 0.1 > x1_new, x2_new = 100.0, 100.0 > while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= > epsilon: > x1, x2 = x1_new, x2_new > x1_new, x2_new, y1, y2 = my_function(x1, x2) > print 'y1 = ', y1 > print 'y2 = ', y2 > > It works. But I wonder if it's the most efficient method. > > I read about scipy.optimize.minimize, but I can not see how it > could be > applied to my problem. > > Thanks in advance for your help. > > Cheers, > > Camille > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreback at gmail.com Mon Mar 23 06:11:01 2015 From: jeffreback at gmail.com (Jeff Reback) Date: Mon, 23 Mar 2015 06:11:01 -0400 Subject: [SciPy-User] ANN: pandas 0.16.0 released Message-ID: Hello, We are proud to announce v0.16.0 of pandas, a major release from 0.15.2. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was 4 months of work by 60 authors encompassing 204 issues. We recommend that all users upgrade to this version. *Highlights:* - - *DataFrame.assign* method, see here - *Series.to_coo/from_coo* methods to interact with *scipy.sparse*, see here - Backwards incompatible change to *Timedelta* to conform the *.seconds* attribute with *datetime.timedelta*, see here - Changes to the *.loc* slicing API to conform with the behavior of *.ix* see here - Changes to the default for ordering in the *Categorical* constructor, see here - Enhancement to the *.str* accessor to make string operations easier, see here - The *pandas.tools.rplot*, *pandas.sandbox.qtpandas* and *pandas.rpy* modules are deprecated. - We refer users to external packages like seaborn , pandas-qt and rpy2 for similar or equivalent functionality, see here for more detail See a full description of the Whatsnew for v0.16.0 *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with ?relational? or ?labeled? data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows wheels, macosx wheels are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.9 macosx wheels are courtesy of Matthew Brett and are built on Numpy 1.7.1 Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From opossumnano at gmail.com Mon Mar 23 11:44:59 2015 From: opossumnano at gmail.com (Tiziano Zito) Date: Mon, 23 Mar 2015 16:44:59 +0100 Subject: [SciPy-User] Reminder - Summer School "Advanced Scientific Programming in Python" in Munich, Germany Message-ID: <20150323154458.GB17798@eniac> Reminder: Deadline for application is 23:59 UTC, March 31, 2015. Advanced Scientific Programming in Python ========================================= a Summer School by the G-Node, the Bernstein Center for Computational Neuroscience Munich and the Graduate School of Systemic Neurosciences Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques, incorporating theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python is assumed. Participants without any prior experience with Python should work through the proposed introductory materials before the course. Date and Location ================= August 31?September 5, 2015. Munich, Germany. Preliminary Program =================== Day 0 (Mon Aug 31) ? Best Programming Practices ? Best Practices for Scientific Computing ? Version control with git and how to contribute to Open Source with github ? Object-oriented programming & design patterns Day 1 (Tue Sept 1) ? Software Carpentry ? Test-driven development, unit testing & quality assurance ? Debugging, profiling and benchmarking techniques ? Advanced Python: generators, decorators, and context managers Day 2 (Wed Sept 2) ? Scientific Tools for Python ? Advanced NumPy ? The Quest for Speed (intro): Interfacing to C with Cython ? Contributing to Open Source Software/Programming in teams Day 3 (Thu Sept 3) ? The Quest for Speed ? Writing parallel applications in Python ? Python 3: why should I care ? Programming project Day 4 (Fri Sept 4) ? Efficient Memory Management ? When parallelization does not help: the starving CPUs problem ? Programming project Day 5 (Sat Sept 5) ? Practical Software Development ? Programming project ? The Pelita Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects. Applications ============ You can apply on-line at https://python.g-node.org Applications must be submitted before 23:59 UTC, March 31, 2015. Notifications of acceptance will be sent by May 1, 2015. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate is usually around 20%. Prerequisites: You are supposed to know the basics of Python to participate in the lectures Preliminary Faculty =================== ? Pietro Berkes, Enthought Inc., UK ? Marianne Corvellec, Plotly Technologies Inc., Montr?al, Canada ? Kathryn D. Huff, Department of Nuclear Engineering, University of California - Berkeley, USA ? Zbigniew J?drzejewski-Szmek, Krasnow Institute, George Mason University, USA ? Eilif Muller, Blue Brain Project, ?cole Polytechnique F?d?rale de Lausanne, Switzerland ? Juan Nunez-Iglesias, Victorian Life Sciences Computation Initiative, University of Melbourne, Australia ? Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany ? Bartosz Tele?czuk, European Institute for Theoretical Neuroscience, CNRS, Paris, France ? Nelle Varoquaux, Centre for Computational Biology Mines ParisTech, Institut Curie, U900 INSERM, Paris, France ? Tiziano Zito, Forschungszentrum J?lich GmbH, Germany Organized by Tiziano Zito (head) and Zbigniew J?drzejewski-Szmek for the German Neuroinformatics Node of the INCF Germany, Christopher Roppelt for the German Center for Vertigo and Balance Disorders (DSGZ) and the Graduate School of Systemic Neurosciences (GSN) of the Ludwig-Maximilians-Universit?t Munich Germany, Christoph Hartmann for the Frankfurt Institute for Advanced Studies (FIAS) and International Max Planck Research School (IMPRS) for Neural Circuits, Frankfurt Germany, and Jakob Jordan for the Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6), J?lich Research Centre and JARA. Additional funding provided by the Bernstein Center for Computational Neuroscience (BCCN) Munich. Website: https://python.g-node.org Contact: python-info at g-node.org From camillechambon at yahoo.fr Mon Mar 23 13:07:03 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Mon, 23 Mar 2015 18:07:03 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <550FE815.8000208@gmail.com> References: <550FE815.8000208@gmail.com> Message-ID: <55104837.5090403@yahoo.fr> Hello Benjamin, Thanks for your answer. Using scipy.optimize.fixed_point increases computation time a lot. It is not acceptable to me. With scipy.optimize.minimize, the objective function must return only one scalar. But my function returns two scalars. So I can't see how I could apply scipy.optimize.minimize to my problem. Thanks for your help anyway. Cheers, Camille Le 23/03/2015 11:16, Benjamin Trendelkamp-Schroer a ?crit : > Hi Camille, > > my message did not get forwarded to the list so I am forwarding it to > your adress, I hope this will help you. > > On 21.03.2015 20:00, Camille Chambon wrote: >> Hello, >> >> I have a function: >> >> def my_function(x1, x2): >> y1 = compute_y1(x1) >> y2 = compute_y2(x2) >> x1_new = compute_new_x1(x1, y1, y2) >> x2_new = compute_new_x2(x2, y1, y2) >> return x1_new, x2_new, y1, y2 >> >> I would like to find the convergent values of y1 and y2, that is when >> (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. >> >> By now, I initialize x1 and x2 and I call main_function numerous times: >> >> epsilon = 0.01 >> x1, x2 = 0.1, 0.1 >> x1_new, x2_new = 100.0, 100.0 >> while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= > epsilon: >> x1, x2 = x1_new, x2_new >> x1_new, x2_new, y1, y2 = my_function(x1, x2) >> print 'y1 = ', y1 >> print 'y2 = ', y2 >> >> It works. But I wonder if it's the most efficient method. >> >> I read about scipy.optimize.minimize, but I can not see how it could be >> applied to my problem. >> >> Thanks in advance for your help. >> >> Cheers, >> >> Camille >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > Hello Camille, > > if I understand correctly the function 'my_function' maps old values > (x1, x2) to new values (x1_new, x2_new), > > (x1_new, x2_new) = f(x1, x2) > > and you want to find those values (x1*, x2*) which fulfill > > (x1*, x2*) = f(x1*, x2*) > > A point (x1*, x2*) with this property is called a fixed point for the > mapping f and there is a scipy function that might help you to find this > fixed point > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fixed_point.html > > Unfortunately these types of sef-consistent iterations can exhibit > rather slow-convergence. > > If there is a way to formulate your problem as a minimization problem > with an objective function g(x1, x2) for which you can compute the > gradient G(x1, x2) you will be able to use methods for the minimization > of a multi-variate function, e.g. gradient descent or Newton's method, > which often converge much faster than self-consistent iterations. You > can learn more about the available scipy functions at > > http://docs.scipy.org/doc/scipy/reference/optimize.html > > Some of them are also derivative free methods. So if you can not obtain > the gradient of g you can still use them to find the minimizer of g. > > As Ryan suggested your function f needs to be able to process the tuple > x=(x1, x2). I would recommend to store x1 and x2 in a numpy array x and > have your function operate on the elements on that ndarray internally. > > A example function similar to yours, assuming that x1 and x2 are vectors > consisting of 10 elements each, so that x has a total of 20 elements is > > def f(x): > x1 = x[0:10] > x2 = x[10:] > y1 = ... > y2 = ... > > x1_new = ... > x2_new = ... > > x_new = np.zeros(20) > x_new[0:10] = x1_new > x_new[10:] = x2_new > > return x_new > > Now input x and output x_new are both numpy arrays with 20 elements and > it shoud work with scipy.optimize.fixed_point. > > I hope this will help you, > > Benjamin > > > > From rnelsonchem at gmail.com Mon Mar 23 13:32:37 2015 From: rnelsonchem at gmail.com (Ryan Nelson) Date: Mon, 23 Mar 2015 13:32:37 -0400 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <55104837.5090403@yahoo.fr> References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> Message-ID: There are multidimensional root solvers as well. http://docs.scipy.org/doc/scipy/reference/optimize.nonlin.html#module-scipy.optimize.nonlin Ryan On Mon, Mar 23, 2015 at 1:07 PM, Camille Chambon wrote: > Hello Benjamin, > > Thanks for your answer. > > Using scipy.optimize.fixed_point increases computation time a lot. It is > not acceptable to me. > > With scipy.optimize.minimize, the objective function must return only > one scalar. But my function returns two scalars. So I can't see how I > could apply scipy.optimize.minimize to my problem. > > Thanks for your help anyway. > > Cheers, > > Camille > > Le 23/03/2015 11:16, Benjamin Trendelkamp-Schroer a ?crit : > > Hi Camille, > > > > my message did not get forwarded to the list so I am forwarding it to > > your adress, I hope this will help you. > > > > On 21.03.2015 20:00, Camille Chambon wrote: > >> Hello, > >> > >> I have a function: > >> > >> def my_function(x1, x2): > >> y1 = compute_y1(x1) > >> y2 = compute_y2(x2) > >> x1_new = compute_new_x1(x1, y1, y2) > >> x2_new = compute_new_x2(x2, y1, y2) > >> return x1_new, x2_new, y1, y2 > >> > >> I would like to find the convergent values of y1 and y2, that is when > >> (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. > >> > >> By now, I initialize x1 and x2 and I call main_function numerous times: > >> > >> epsilon = 0.01 > >> x1, x2 = 0.1, 0.1 > >> x1_new, x2_new = 100.0, 100.0 > >> while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= > > epsilon: > >> x1, x2 = x1_new, x2_new > >> x1_new, x2_new, y1, y2 = my_function(x1, x2) > >> print 'y1 = ', y1 > >> print 'y2 = ', y2 > >> > >> It works. But I wonder if it's the most efficient method. > >> > >> I read about scipy.optimize.minimize, but I can not see how it could be > >> applied to my problem. > >> > >> Thanks in advance for your help. > >> > >> Cheers, > >> > >> Camille > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > Hello Camille, > > > > if I understand correctly the function 'my_function' maps old values > > (x1, x2) to new values (x1_new, x2_new), > > > > (x1_new, x2_new) = f(x1, x2) > > > > and you want to find those values (x1*, x2*) which fulfill > > > > (x1*, x2*) = f(x1*, x2*) > > > > A point (x1*, x2*) with this property is called a fixed point for the > > mapping f and there is a scipy function that might help you to find this > > fixed point > > > > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fixed_point.html > > > > Unfortunately these types of sef-consistent iterations can exhibit > > rather slow-convergence. > > > > If there is a way to formulate your problem as a minimization problem > > with an objective function g(x1, x2) for which you can compute the > > gradient G(x1, x2) you will be able to use methods for the minimization > > of a multi-variate function, e.g. gradient descent or Newton's method, > > which often converge much faster than self-consistent iterations. You > > can learn more about the available scipy functions at > > > > http://docs.scipy.org/doc/scipy/reference/optimize.html > > > > Some of them are also derivative free methods. So if you can not obtain > > the gradient of g you can still use them to find the minimizer of g. > > > > As Ryan suggested your function f needs to be able to process the tuple > > x=(x1, x2). I would recommend to store x1 and x2 in a numpy array x and > > have your function operate on the elements on that ndarray internally. > > > > A example function similar to yours, assuming that x1 and x2 are vectors > > consisting of 10 elements each, so that x has a total of 20 elements is > > > > def f(x): > > x1 = x[0:10] > > x2 = x[10:] > > y1 = ... > > y2 = ... > > > > x1_new = ... > > x2_new = ... > > > > x_new = np.zeros(20) > > x_new[0:10] = x1_new > > x_new[10:] = x2_new > > > > return x_new > > > > Now input x and output x_new are both numpy arrays with 20 elements and > > it shoud work with scipy.optimize.fixed_point. > > > > I hope this will help you, > > > > Benjamin > > > > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.clewley at gmail.com Mon Mar 23 17:09:59 2015 From: rob.clewley at gmail.com (Rob Clewley) Date: Mon, 23 Mar 2015 17:09:59 -0400 Subject: [SciPy-User] Setting up the model for a large nonlinear-in-the-parameters ODE system In-Reply-To: References: Message-ID: Hi Brian, > I have a use case where I need to model a large ODE system. > The purpose of these methods was to calculate some > nonlinear-in-the-parameter variable. In the example, I just show how some > baseline value given to the function initially along with other variable > values would be combined in some way to calculate the parameter value -- the > form of the math is not important as its just for example purposes. >From what I can understand of your message, you basically have derived (and structured) parameters that you want to preset in advance. Typically, users do have to write some setup code of their own for such cases. Alternatively, you'd set up auxiliary functions in your model that determine the current value of those parameters for use in the RHS. In PyDSTool, at least, unfortunately such functions have to be called every time step, so this is inefficient if these values are meant to be static during each integration. There have been some proposals to include derived parameters inherently in PyDSTool, but really it's very easy to manage them automatically with some simple python wrapper code. > Overall, the class was quite messy and kludge-y, partially because I had > started to prematurely optimize (I know, I know, the root of all evil :( ) > and gone "Cython from the very start". Every time I wanted to change the > RHS, I would go in and manually change equations, deleting parameters and > variables that were no longer necessary, manually changing methods, or if I > needed new variables, I would lazily add them "some_var = 2" style, where > some_var can't be accessed nicely without going to the source code and > changing what it is equal to... Yes, that approach sounds painful and unnecessary. One good way to manage the dependencies between equations and parameters is to make a modular description of your model. Although you could write your own functions (or classes) to do this involving string "elements" (equation terms), PyDSTool offers a set of template ModelSpec classes that can be built up using strings or Symbolic objects, and are composable in the expected ways. Unless you start getting into heavyweight 3rd party libraries such as Modelica, I'm not sure there's any other convenient tools to create dynamical systems in a structured, manageable way. Using the ModelSpec classes means that you can make configurable modules out of your model, which you can interchange or add/delete as you wish. The parameter declarations associated with each module belong to the ModelSpec object, so there's no book-keeping for you to do. In fact, before the model is finally created, there are automatic internal checks for consistency and resolution of all names used in your system. See this tutorial, for instance: http://www.ni.gsu.edu/~rclewley/PyDSTool/Tutorial/Tutorial_compneuro.html or ModelSpec_test.py in the examples/ folder. There are pages specifically about using ModelSpec and the associated tools on the website. Maybe you could help develop a simpler tutorial for a system that's not related to computational neuroscience, based on your use case? An example like the one mentioned by the previous poster on this thread, involving a user creating a coupling matrix to determine what elements are created in the final model, would be a particularly valuable and interesting demonstration. Feel free to discuss details with me off list. -Rob From eric.moore2 at nih.gov Tue Mar 24 08:34:05 2015 From: eric.moore2 at nih.gov (Moore, Eric (NIH/NIDDK) [F]) Date: Tue, 24 Mar 2015 12:34:05 +0000 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <55104837.5090403@yahoo.fr> References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> Message-ID: <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> > -----Original Message----- > From: Camille Chambon [mailto:camillechambon at yahoo.fr] > Sent: Monday, March 23, 2015 1:07 PM > To: Benjamin Trendelkamp-Schroer > Cc: scipy-user at scipy.org > Subject: Re: [SciPy-User] Find the convergent solutions of a > multivariate equation efficiently > > Hello Benjamin, > > Thanks for your answer. > > Using scipy.optimize.fixed_point increases computation time a lot. It > is not acceptable to me. The fair comparison between the code you posted and optimize.fixed_point requires that you set xtol=0.01. Is the fixed_point code still slower in that case? It does basically what your code does, but with some bells and whistles. It shouldn?t be meaningfully slower. There is also a pull request open to add a more sophisticated solver for fixed point problems (https://github.com/scipy/scipy/pull/4460). > > With scipy.optimize.minimize, the objective function must return only > one scalar. But my function returns two scalars. So I can't see how I > could apply scipy.optimize.minimize to my problem. Optimize.minimize is not the tool for this job. -Eric > > Thanks for your help anyway. > > Cheers, > > Camille > > Le 23/03/2015 11:16, Benjamin Trendelkamp-Schroer a ?crit : > > Hi Camille, > > > > my message did not get forwarded to the list so I am forwarding it to > > your adress, I hope this will help you. > > > > On 21.03.2015 20:00, Camille Chambon wrote: > >> Hello, > >> > >> I have a function: > >> > >> def my_function(x1, x2): > >> y1 = compute_y1(x1) > >> y2 = compute_y2(x2) > >> x1_new = compute_new_x1(x1, y1, y2) > >> x2_new = compute_new_x2(x2, y1, y2) > >> return x1_new, x2_new, y1, y2 > >> > >> I would like to find the convergent values of y1 and y2, that is > when > >> (x1_new - x1) / x1 < epsilon and (x2_new - x2) / x2 < epsilon. > >> > >> By now, I initialize x1 and x2 and I call main_function numerous > times: > >> > >> epsilon = 0.01 > >> x1, x2 = 0.1, 0.1 > >> x1_new, x2_new = 100.0, 100.0 > >> while abs((x1_new - x1)/x1) >= epsilon or abs((x2_new - x2)/x2) >= > > epsilon: > >> x1, x2 = x1_new, x2_new > >> x1_new, x2_new, y1, y2 = my_function(x1, x2) print 'y1 = ', y1 > >> print 'y2 = ', y2 > >> > >> It works. But I wonder if it's the most efficient method. > >> > >> I read about scipy.optimize.minimize, but I can not see how it could > >> be applied to my problem. > >> > >> Thanks in advance for your help. > >> > >> Cheers, > >> > >> Camille > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > > Hello Camille, > > > > if I understand correctly the function 'my_function' maps old values > > (x1, x2) to new values (x1_new, x2_new), > > > > (x1_new, x2_new) = f(x1, x2) > > > > and you want to find those values (x1*, x2*) which fulfill > > > > (x1*, x2*) = f(x1*, x2*) > > > > A point (x1*, x2*) with this property is called a fixed point for the > > mapping f and there is a scipy function that might help you to find > > this fixed point > > > > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fix > > ed_point.html > > > > Unfortunately these types of sef-consistent iterations can exhibit > > rather slow-convergence. > > > > If there is a way to formulate your problem as a minimization problem > > with an objective function g(x1, x2) for which you can compute the > > gradient G(x1, x2) you will be able to use methods for the > > minimization of a multi-variate function, e.g. gradient descent or > > Newton's method, which often converge much faster than self- > consistent > > iterations. You can learn more about the available scipy functions at > > > > http://docs.scipy.org/doc/scipy/reference/optimize.html > > > > Some of them are also derivative free methods. So if you can not > > obtain the gradient of g you can still use them to find the minimizer > of g. > > > > As Ryan suggested your function f needs to be able to process the > > tuple x=(x1, x2). I would recommend to store x1 and x2 in a numpy > > array x and have your function operate on the elements on that > ndarray internally. > > > > A example function similar to yours, assuming that x1 and x2 are > > vectors consisting of 10 elements each, so that x has a total of 20 > > elements is > > > > def f(x): > > x1 = x[0:10] > > x2 = x[10:] > > y1 = ... > > y2 = ... > > > > x1_new = ... > > x2_new = ... > > > > x_new = np.zeros(20) > > x_new[0:10] = x1_new > > x_new[10:] = x2_new > > > > return x_new > > > > Now input x and output x_new are both numpy arrays with 20 elements > > and it shoud work with scipy.optimize.fixed_point. > > > > I hope this will help you, > > > > Benjamin > > > > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From camillechambon at yahoo.fr Tue Mar 24 09:47:10 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Tue, 24 Mar 2015 14:47:10 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> Message-ID: <55116ADE.6000201@yahoo.fr> Le 24/03/2015 13:34, Moore, Eric (NIH/NIDDK) [F] a ?crit : > The fair comparison between the code you posted and optimize.fixed_point requires that you set xtol=0.01. Is the fixed_point code still slower in that case? It does basically what your code does, but with some bells and whistles. It shouldn?t be meaningfully slower. > > There is also a pull request open to add a more sophisticated solver for fixed point problems (https://github.com/scipy/scipy/pull/4460). Here is the full example: #################### from __future__ import division def compute_var1(x1, x2): return 2 * x1 + x2 def compute_var2(x1, x2): return 2 * x1 - x2 def compute_new_x1(x1): return x1 + 10 def compute_new_x2(x2): return x2 + 10 def while_iterations(): x1_new, x2_new = 100.0, 100.0 x1, x2 = 0.1, 0.1 step = 0 while abs((x1_new - x1)/x1) >= 0.01 or abs((x2_new - x2)/x2) >= 0.01: x1, x2 = x1_new, x2_new var1 = compute_var1(x1, x2) var2 = compute_var2(x1, x2) x1_new = compute_new_x1(x1) x2_new = compute_new_x2(x2) step += 1 print 'var1=', var1, 'var2=', var2 def with_optimize_fixed_point(): import numpy as np from scipy import optimize def func(x, vars_): x1 = x[0] x2 = x[1] vars_[0] = compute_var1(x1, x2) vars_[1] = compute_var2(x1, x2) x1_new = compute_new_x1(x1) x2_new = compute_new_x2(x2) return np.array([x1_new, x2_new]) vars_ = [0, 0] x1_fixed, x2_fixed = optimize.fixed_point(func, [0.1, 0.1], args=(vars_,)) print 'var1=', vars_[0], 'var2=', vars_[1] def with_optimize_root(): def fun(x, vars_): x1 = x[0] x2 = x[1] vars_[0] = compute_var1(x1, x2) vars_[1] = compute_var2(x1, x2) x1_new = compute_new_x1(x1) x2_new = compute_new_x2(x2) delta_x1 = abs((x1_new - x1)/x1) delta_x2 = abs((x2_new - x2)/x2) return [delta_x1, delta_x2] from scipy import optimize vars_ = [0, 0] sol = optimize.root(fun, [0.1, 0.1], vars_) print 'var1=', vars_[0], 'var2=', vars_[1] ########################## while_iterations() prints: var1= 3030.0 var2= 1010.0 with_optimize_fixed_point() prints: /usr/lib64/python2.7/site-packages/scipy/optimize/minpack.py:608: RuntimeWarning: divide by zero encountered in true_divide p = where(d == 0, p2, p0 - (p1 - p0)*(p1 - p0) / d) var1= -1.40371936438e+17 var2= -4.67906454792e+16 with_optimize_root() prints: /usr/lib64/python2.7/site-packages/scipy/optimize/minpack.py:227: RuntimeWarning: The iteration is not making good progress, as measured by the improvement from the last ten iterations. warnings.warn(msg, RuntimeWarning) var1= 7.76159959998e+16 var2= 2.61013612882e+16 I didn't manage to obtain similar results neither with scipy.optimize.fixed_point nor with scipy.optimize.root yet. Do I use scipy.optimize.fixed_point and scipy.optimize.root in a bad way? Camille From pav at iki.fi Tue Mar 24 14:21:07 2015 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 24 Mar 2015 20:21:07 +0200 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <55116ADE.6000201@yahoo.fr> References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> <55116ADE.6000201@yahoo.fr> Message-ID: 24.03.2015, 15:47, Camille Chambon kirjoitti: [clip] > Do I use scipy.optimize.fixed_point and scipy.optimize.root in a bad way? Your iteration is: x1[n+1] = x1[n] + 10 x2[n+1] = x2[n] + 10 It has no fixed point, both x1 and x2 diverge to infinity. The results returned by your while_iterations() do not have meaning, they depend on the tolerance 0.01 you used. If you use smaller tolerance, the results diverge. From toddrjen at gmail.com Tue Mar 24 16:39:19 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 24 Mar 2015 21:39:19 +0100 Subject: [SciPy-User] Playing numpy array over speakers Message-ID: Is anyone aware of a well-maintained, simple, cross-platform python package that can play a numpy array as sound over speakers? I am aware of https://wiki.python.org/moin/Audio/. However, in all the cases there, as far as I can find they either do not support numpy arrays, are not cross-platform, cannot playback sound at all, or are unmaintained. There is also PySoundCard, which would do what I need but also appears to be unmaintained (no release in over a year, and no commits in 5 months, no release with serious bugfixes mentioned in commits). -------------- next part -------------- An HTML attachment was scrubbed... URL: From kikocorreoso at gmail.com Tue Mar 24 16:51:25 2015 From: kikocorreoso at gmail.com (Kiko) Date: Tue, 24 Mar 2015 21:51:25 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: 2015-03-24 21:39 GMT+01:00 Todd : > Is anyone aware of a well-maintained, simple, cross-platform python > package that can play a numpy array as sound over speakers? > > I am aware of https://wiki.python.org/moin/Audio/. However, in all the > cases there, as far as I can find they either do not support numpy arrays, > are not cross-platform, cannot playback sound at all, or are unmaintained. > There is also PySoundCard, which would do what I need but also appears to > be unmaintained (no release in over a year, and no commits in 5 months, no > release with serious bugfixes mentioned in commits). > Could this be useful for your use case? http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Interactive%20Widgets/Beat%20Frequencies.ipynb > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Tue Mar 24 17:15:42 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 24 Mar 2015 22:15:42 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: On Mar 24, 2015 9:51 PM, "Kiko" wrote: > > > > 2015-03-24 21:39 GMT+01:00 Todd : >> >> Is anyone aware of a well-maintained, simple, cross-platform python package that can play a numpy array as sound over speakers? >> >> I am aware of https://wiki.python.org/moin/Audio/. However, in all the cases there, as far as I can find they either do not support numpy arrays, are not cross-platform, cannot playback sound at all, or are unmaintained. There is also PySoundCard, which would do what I need but also appears to be unmaintained (no release in over a year, and no commits in 5 months, no release with serious bugfixes mentioned in commits). > > > Could this be useful for your use case? http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Interactive%20Widgets/Beat%20Frequencies.ipynb > Sorry, I should have been more specific. I am really looking for a way to play audio as part of a script. Your link would certainly work for playing back sound in an interactive manner in a notebook. However, it is not really useful as part general-purpose script or function, or when control of the playback is needed. Thank you, though, that is certainly very useful in many situations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hturesson at gmail.com Tue Mar 24 17:27:30 2015 From: hturesson at gmail.com (Hjalmar Turesson) Date: Tue, 24 Mar 2015 18:27:30 -0300 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: I have a little function that plays back numpy arrays using PyAudio. I think PyAudio is pretty platform independent (but it takes a bit of installing). My code is not perfect, but it is simple and works ok. Want it? Hjalmar On Tue, Mar 24, 2015 at 6:15 PM, Todd wrote: > > On Mar 24, 2015 9:51 PM, "Kiko" wrote: > > > > > > > > 2015-03-24 21:39 GMT+01:00 Todd : > >> > >> Is anyone aware of a well-maintained, simple, cross-platform python > package that can play a numpy array as sound over speakers? > >> > >> I am aware of https://wiki.python.org/moin/Audio/. However, in all > the cases there, as far as I can find they either do not support numpy > arrays, are not cross-platform, cannot playback sound at all, or are > unmaintained. There is also PySoundCard, which would do what I need but > also appears to be unmaintained (no release in over a year, and no commits > in 5 months, no release with serious bugfixes mentioned in commits). > > > > > > Could this be useful for your use case? > http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Interactive%20Widgets/Beat%20Frequencies.ipynb > > > > Sorry, I should have been more specific. I am really looking for a way to > play audio as part of a script. Your link would certainly work for playing > back sound in an interactive manner in a notebook. However, it is not > really useful as part general-purpose script or function, or when control > of the playback is needed. Thank you, though, that is certainly very useful > in many situations. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From opossumnano at gmail.com Tue Mar 24 17:27:50 2015 From: opossumnano at gmail.com (Tiziano Zito) Date: Tue, 24 Mar 2015 22:27:50 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: <20150324212749.GA6852@multivac> On Tue 24 Mar, 21:39, Todd wrote: > Is anyone aware of a well-maintained, simple, cross-platform python package > that can play a numpy array as sound over speakers? Not exactly what you are looking for, not cross-platform, but may be of inspiration: https://github.com/stefanv/bytebeats Tiziano From jdjensen at eng.ucsd.edu Tue Mar 24 19:45:40 2015 From: jdjensen at eng.ucsd.edu (James Jensen) Date: Tue, 24 Mar 2015 16:45:40 -0700 Subject: [SciPy-User] v0 initial vector for scipy.sparse.linalg.eigs() Message-ID: I have to quickly compute the principal eigenpair of a dense matrix. I've been using scipy.sparse.linalg.eigs() with k=1; it works for dense matrices and is faster than the alternatives in scipy.linalg. In my application, I do multiple successive eigendecompositions; each matrix to be decomposed differs only slightly from the previous. So in a simple power iteration I can use the previous principal eigenvector (with a slight modification) rather than a random initial vector and usually achieve convergence faster than from a random start. The eigs() function allows warm starts; it takes a parameter, v0, for the initial vector. I imagine that the v0 option could make eigs() faster than my warm-started power iteration. However, passing the previous principal eigenvector for v0 does not speed up eigs(); in fact, it makes it somewhat slower, usually between 10% and 50%. Should I be passing a different vector as v0? Is eigs() not expecting v0 to be a guess of the eigenvector(s)? Any other ideas? Thanks for the help, James Jensen PhD student, Bioinformatics and Systems Biology Trey Ideker lab University of California, San Diego -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Tue Mar 24 19:48:42 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Tue, 24 Mar 2015 16:48:42 -0700 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: <20150324212749.GA6852@multivac> References: <20150324212749.GA6852@multivac> Message-ID: music21 (http://web.mit.edu/music21/) is in active development, and meant for music analysis and procedural generation...I don't know if it meets the numpy requirement though. From the simple glance that I gave its docs, I don't think it does...but I simply add it to the pile of "possibles", just in case I am wrong. I too am very interested in your question, so I am glad you asked it. I hope someone is able to come up with a good answer! On Tue, Mar 24, 2015 at 2:27 PM, Tiziano Zito wrote: > On Tue 24 Mar, 21:39, Todd wrote: > > Is anyone aware of a well-maintained, simple, cross-platform python > package > > that can play a numpy array as sound over speakers? > Not exactly what you are looking for, not cross-platform, but may be of > inspiration: > > https://github.com/stefanv/bytebeats > > Tiziano > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaldorusso at gmail.com Tue Mar 24 20:27:21 2015 From: arnaldorusso at gmail.com (Arnaldo Russo) Date: Tue, 24 Mar 2015 21:27:21 -0300 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: <20150324212749.GA6852@multivac> Message-ID: We have a project called datasounds [0] Which can transform an array of data to sounds [1]: [0] www.datasounds.org [1] http://datasounds.readthedocs.org/en/latest/ Cheers, Arnaldo. --- *Arnaldo D'Amaral Pereira Granja Russo* Lab. de Estudos dos Oceanos e Clima Instituto de Oceanografia - FURG 2015-03-24 20:48 GMT-03:00 Brian Merchant : > music21 (http://web.mit.edu/music21/) is in active development, and meant > for music analysis and procedural generation...I don't know if it meets the > numpy requirement though. From the simple glance that I gave its docs, I > don't think it does...but I simply add it to the pile of "possibles", just > in case I am wrong. > > I too am very interested in your question, so I am glad you asked it. I > hope someone is able to come up with a good answer! > > > On Tue, Mar 24, 2015 at 2:27 PM, Tiziano Zito > wrote: > >> On Tue 24 Mar, 21:39, Todd wrote: >> > Is anyone aware of a well-maintained, simple, cross-platform python >> package >> > that can play a numpy array as sound over speakers? >> Not exactly what you are looking for, not cross-platform, but may be of >> inspiration: >> >> https://github.com/stefanv/bytebeats >> >> Tiziano >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From silva at lma.cnrs-mrs.fr Wed Mar 25 05:15:06 2015 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Wed, 25 Mar 2015 10:15:06 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: <1427274906.20332.40.camel@lma.cnrs-mrs.fr> Le mardi 24 mars 2015 ? 21:39 +0100, Todd a ?crit : > Is anyone aware of a well-maintained, simple, cross-platform python package > that can play a numpy array as sound over speakers? > > I am aware of https://wiki.python.org/moin/Audio/. However, in all the > cases there, as far as I can find they either do not support numpy arrays, > are not cross-platform, cannot playback sound at all, or are unmaintained. > There is also PySoundCard, which would do what I need but also appears to > be unmaintained (no release in over a year, and no commits in 5 months, no > release with serious bugfixes mentioned in commits). Scikits.audiolab has playback capabilities (at least on linux and mac): http://cournape.github.io/audiolab/overview.html and it does handle numpy arrays -- Fabrice From toddrjen at gmail.com Wed Mar 25 06:00:21 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 25 Mar 2015 11:00:21 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: <1427274906.20332.40.camel@lma.cnrs-mrs.fr> References: <1427274906.20332.40.camel@lma.cnrs-mrs.fr> Message-ID: On Wed, Mar 25, 2015 at 10:15 AM, Fabrice Silva wrote: > Le mardi 24 mars 2015 ? 21:39 +0100, Todd a ?crit : > > Is anyone aware of a well-maintained, simple, cross-platform python > package > > that can play a numpy array as sound over speakers? > > > > I am aware of https://wiki.python.org/moin/Audio/. However, in all the > > cases there, as far as I can find they either do not support numpy > arrays, > > are not cross-platform, cannot playback sound at all, or are > unmaintained. > > There is also PySoundCard, which would do what I need but also appears to > > be unmaintained (no release in over a year, and no commits in 5 months, > no > > release with serious bugfixes mentioned in commits). > > Scikits.audiolab has playback capabilities (at least on linux and mac): > http://cournape.github.io/audiolab/overview.html > and it does handle numpy arrays > > I saw that, but something that only supports Linux and Mac isn't really that cross-platform. I use Linux, but a lot of the computer we use run windows. Thanks, though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Mar 25 06:05:26 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 25 Mar 2015 11:05:26 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: <20150324212749.GA6852@multivac> Message-ID: On Wed, Mar 25, 2015 at 1:27 AM, Arnaldo Russo wrote: > We have a project called datasounds [0] > > Which can transform an array of data to sounds [1]: > > [0] www.datasounds.org > [1] http://datasounds.readthedocs.org/en/latest/ > > Cheers, > Arnaldo. > > > Does it support sound playback? I couldn't find anything in the documentation or source code about sound playback. > > 2015-03-24 20:48 GMT-03:00 Brian Merchant : > > music21 (http://web.mit.edu/music21/) is in active development, and meant >> for music analysis and procedural generation...I don't know if it meets the >> numpy requirement though. From the simple glance that I gave its docs, I >> don't think it does...but I simply add it to the pile of "possibles", just >> in case I am wrong. >> >> I too am very interested in your question, so I am glad you asked it. I >> hope someone is able to come up with a good answer! >> >> >> On Tue, Mar 24, 2015 at 2:27 PM, Tiziano Zito >> wrote: >> >>> On Tue 24 Mar, 21:39, Todd wrote: >>> > Is anyone aware of a well-maintained, simple, cross-platform python >>> package >>> > that can play a numpy array as sound over speakers? >>> Not exactly what you are looking for, not cross-platform, but may be of >>> inspiration: >>> >>> https://github.com/stefanv/bytebeats >>> >>> Tiziano >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Mar 25 06:06:41 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 25 Mar 2015 11:06:41 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: <20150324212749.GA6852@multivac> Message-ID: On Wed, Mar 25, 2015 at 12:48 AM, Brian Merchant wrote: > music21 (http://web.mit.edu/music21/) is in active development, and meant > for music analysis and procedural generation...I don't know if it meets the > numpy requirement though. From the simple glance that I gave its docs, I > don't think it does...but I simply add it to the pile of "possibles", just > in case I am wrong. > > I too am very interested in your question, so I am glad you asked it. I > hope someone is able to come up with a good answer! > > > It doesn't appear to support numpy, and seems to be designed for producing music, rather than playing raw waveforms. > On Tue, Mar 24, 2015 at 2:27 PM, Tiziano Zito > wrote: > >> On Tue 24 Mar, 21:39, Todd wrote: >> > Is anyone aware of a well-maintained, simple, cross-platform python >> package >> > that can play a numpy array as sound over speakers? >> Not exactly what you are looking for, not cross-platform, but may be of >> inspiration: >> >> https://github.com/stefanv/bytebeats >> >> Tiziano >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Mar 25 06:10:35 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 25 Mar 2015 11:10:35 +0100 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: Thanks, that would be useful. It appears PyAudio is the closest to what I need. So seeing how to translate numpy data into PyAudio data would be useful. On Tue, Mar 24, 2015 at 10:27 PM, Hjalmar Turesson wrote: > I have a little function that plays back numpy arrays using PyAudio. I > think PyAudio is pretty platform independent (but it takes a bit of > installing). My code is not perfect, but it is simple and works ok. Want it? > > Hjalmar > > On Tue, Mar 24, 2015 at 6:15 PM, Todd wrote: > >> >> On Mar 24, 2015 9:51 PM, "Kiko" wrote: >> > >> > >> > >> > 2015-03-24 21:39 GMT+01:00 Todd : >> >> >> >> Is anyone aware of a well-maintained, simple, cross-platform python >> package that can play a numpy array as sound over speakers? >> >> >> >> I am aware of https://wiki.python.org/moin/Audio/. However, in all >> the cases there, as far as I can find they either do not support numpy >> arrays, are not cross-platform, cannot playback sound at all, or are >> unmaintained. There is also PySoundCard, which would do what I need but >> also appears to be unmaintained (no release in over a year, and no commits >> in 5 months, no release with serious bugfixes mentioned in commits). >> > >> > >> > Could this be useful for your use case? >> http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Interactive%20Widgets/Beat%20Frequencies.ipynb >> > >> >> Sorry, I should have been more specific. I am really looking for a way >> to play audio as part of a script. Your link would certainly work for >> playing back sound in an interactive manner in a notebook. However, it is >> not really useful as part general-purpose script or function, or when >> control of the playback is needed. Thank you, though, that is certainly >> very useful in many situations. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johannes.kulick at ipvs.uni-stuttgart.de Wed Mar 25 06:12:24 2015 From: johannes.kulick at ipvs.uni-stuttgart.de (Johannes Kulick) Date: Wed, 25 Mar 2015 11:12:24 +0100 Subject: [SciPy-User] MCTS Package (Monte Carlo Tree Search) Message-ID: <20150325101224.29178.16846@quirm.robotics.tu-berlin.de> Hi all, over the last couple of days I wrote a lightweight monte carlo tree search package for some experiments I want to do for my thesis. It is far from complete etc. but I figured, that there is virtually no MCTS library around in any language so far (did I miss anything?). So it might be useful for some. You can find it at my github repository: https://github.com/hildensia/mcts I consider making it a "scikit.mcts" or the like, but it seems so little so far that I'm afraid it might not worth doing it. Best, Johannes From camillechambon at yahoo.fr Wed Mar 25 06:23:34 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Wed, 25 Mar 2015 11:23:34 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> <55116ADE.6000201@yahoo.fr> Message-ID: <55128CA6.8050307@yahoo.fr> Yes. Sorry. I wrote this simplified system quickly without thinking of its meaningfulness. The original system is convergent, but too much complex to be shared and can not be simplified. To summary, my system looks like: * y1 = g1(x1, x2) * y2 = g2(y1) * next_x1 = f1(y1, y2) * next_x2 = f2(y2, x2) * (next_x1 - x1) / x1 < epsilon * (next_x2 - x2) / x2 < epsilon What I want to compute is y1 and y2. I can do it like that: x1, x2 = x1_0, x2_0 while True: y1 = g1(x1, x2) y2 = g2(y1) next_x1 = f1(y1, y2) next_x2 = f2(y2, x2) if abs((x1_new - x1)/x1) < epsilon and abs((x2_new - x2)/x2) < epsilon: break print 'y1', y1, 'y2', y2 I also manage to use scipy.optimize.root: def fun(x, y1_y2): x1 = x[0] x2 = x[1] y1 = g1(x1, x2) y2 = g2(y1) next_x1 = f1(y1, y2) next_x2 = f2(y2, x2) y1_y2[:] = y1, y2 return [(next_x1 - x1) / x1, (next_x2 - x2) / x2] from scipy import optimize import numpy as np y1_y2 = np.zeros(2) sol = optimize.root(fun, [x1_0, x2_0], (y1_y2,)) print 'y1', y1_y2[0], 'y2', y1_y2[1] Both work fine. But using timeit(...), the solution with scipy.optimize.root is twenty times slower than the one with the while loop. Is it normal? Maybe scipy.optimize.root is much slower because I don't give it a Jacobian ? But I can't define the Jacobian matrix because my system is too complex. Maybe scipy.optimize.root is just not appropriate to solve my system? Thanks again for your help. Cheers, Camille Le 24/03/2015 19:21, Pauli Virtanen a ?crit : > 24.03.2015, 15:47, Camille Chambon kirjoitti: > [clip] >> Do I use scipy.optimize.fixed_point and scipy.optimize.root in a bad way? > Your iteration is: > > x1[n+1] = x1[n] + 10 > x2[n+1] = x2[n] + 10 > > It has no fixed point, both x1 and x2 diverge to infinity. > > The results returned by your while_iterations() do not have meaning, > they depend on the tolerance 0.01 you used. If you use smaller > tolerance, the results diverge. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Mar 25 13:38:33 2015 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 25 Mar 2015 19:38:33 +0200 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: <55128CA6.8050307@yahoo.fr> References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> <55116ADE.6000201@yahoo.fr> <55128CA6.8050307@yahoo.fr> Message-ID: 25.03.2015, 12:23, Camille Chambon kirjoitti: [clip] > Both work fine. But using timeit(...), the solution with > scipy.optimize.root is twenty times slower than the one with the while > loop. Is it normal? The default accuracy in optimize.root is 10^{-8}. Maybe your epsilon was much bigger? From jdjensen at eng.ucsd.edu Wed Mar 25 19:43:17 2015 From: jdjensen at eng.ucsd.edu (James Jensen) Date: Wed, 25 Mar 2015 16:43:17 -0700 Subject: [SciPy-User] discrepancy in chi square cumulative distribution and gamma functions? Message-ID: >From what I understand, the cdf of a chi-square distribution with k degrees of freedom and a test statistic x is lower_incomplete_gamma(k/2, x/2) / gamma(k/2) So you would expect the following two lines of code to yield the same result: cdf1 = scipy.stats.chi2.cdf(x, k) cdf2 = scipy.special.gammainc(0.5 * k, 0.5 * x) / scipy.special.gamma(0.5 * k) Here's the ratio cdf1 / cdf2 for k=1, 2, ...20: 1 1.77245385091 2 1.0 3 0.886226925453 4 1.0 5 1.32934038818 6 2.0 7 3.32335097045 8 6.0 9 11.6317283966 10 24.0 11 52.3427777846 12 120.0 13 287.885277815 14 720.0 15 1871.2543058 16 5040.0 17 14034.4072935 18 40320.0 19 119292.461995 20 362880.0 The discrepancy appears to be a function of k but not of x, i.e. it's constant for different x with the same k. I suspect I'm missing something obvious, but I can't see it. Any thoughts? James -------------- next part -------------- An HTML attachment was scrubbed... URL: From chadfulton at gmail.com Wed Mar 25 20:08:35 2015 From: chadfulton at gmail.com (Chad Fulton) Date: Wed, 25 Mar 2015 17:08:35 -0700 Subject: [SciPy-User] discrepancy in chi square cumulative distribution and gamma functions? In-Reply-To: References: Message-ID: On Wed, Mar 25, 2015 at 4:43 PM, James Jensen wrote: > From what I understand, the cdf of a chi-square distribution with k degrees > of freedom and a test statistic x is > > lower_incomplete_gamma(k/2, x/2) / gamma(k/2) > > So you would expect the following two lines of code to yield the same > result: > > cdf1 = scipy.stats.chi2.cdf(x, k) > cdf2 = scipy.special.gammainc(0.5 * k, 0.5 * x) / scipy.special.gamma(0.5 * > k) > It appears to be simply a naming/expectation issue. What is being calculated with scipy.special.gammainc is not what e.g. Wikipedia calls the lower incomplete gamma function but instead is what it calls the regularized gamma function, see [1]. MATLAB's naming is consistent with Scipy's (not with Wikipedia's), see [2]. In fact, both scipy.stats.chi2.cdf and scipy.special.gammainc call the exact same underlying C function, it's just that scipy.stats.chi2.cdf divides the arguments by 2 first. You'll notice that the discrepancy-ratios you give are just equal to scipy.special.gamma(0.5 * k), k = 1, 2, ..., 20. [1] http://en.wikipedia.org/wiki/Incomplete_gamma_function#Regularized_Gamma_functions_and_Poisson_random_variables [2] http://www.mathworks.com/help/matlab/ref/gammainc.html Cheers, Chad From camillechambon at yahoo.fr Thu Mar 26 04:21:13 2015 From: camillechambon at yahoo.fr (Camille Chambon) Date: Thu, 26 Mar 2015 09:21:13 +0100 Subject: [SciPy-User] Find the convergent solutions of a multivariate equation efficiently In-Reply-To: References: <550FE815.8000208@gmail.com> <55104837.5090403@yahoo.fr> <649847CE7F259144A0FD99AC64E7326DECF82C@msgb09.nih.gov> <55116ADE.6000201@yahoo.fr> <55128CA6.8050307@yahoo.fr> Message-ID: <5513C179.8060209@yahoo.fr> You are right. My epsilon is 0.01, which is much bigger than the default accuracy in optimize.root. But setting tol=0.01: scipy.optimize.root(fun, x0, tol=0.01) scipy.optimize.root is still 17 times slower than the while loop. Camille Le 25/03/2015 18:38, Pauli Virtanen a ?crit : > 25.03.2015, 12:23, Camille Chambon kirjoitti: > [clip] >> Both work fine. But using timeit(...), the solution with >> scipy.optimize.root is twenty times slower than the one with the while >> loop. Is it normal? > The default accuracy in optimize.root is 10^{-8}. Maybe your epsilon was > much bigger? > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From jdjensen at eng.ucsd.edu Thu Mar 26 12:51:23 2015 From: jdjensen at eng.ucsd.edu (James Jensen) Date: Thu, 26 Mar 2015 09:51:23 -0700 Subject: [SciPy-User] discrepancy in chi square cumulative distribution and gamma functions? Message-ID: Thanks, Chad. My reason for looking into this was that I am working with cdf values that are very close to zero or one, such that chi2.cdf rounds them to 0.0 or 1.0. I hope that by working in log space I could distinguish between them. So to get the log of the cdf, I was considering using something along the lines of ln(lower_incomplete_gamma(k/2, x/2)) - ln(gamma(k/2)) and scipy.special has a gammaln function. I then realized that scipy.stats.chi2 has a logcdf function. But it doesn't appear to have the additional precision I'm looking for. Is it possible that it's implemented first computing the cdf, and then taking the logarithm of it? If so, the precision would be lost prior to taking the logarithm. On Wed, Mar 25, 2015 at 4:43 PM, James Jensen eng.ucsd.edu> wrote: > From what I understand, the cdf of a chi-square distribution with k degrees > of freedom and a test statistic x is > > lower_incomplete_gamma(k/2, x/2) / gamma(k/2) > > So you would expect the following two lines of code to yield the same > result: > > cdf1 = scipy.stats.chi2.cdf(x, k) > cdf2 = scipy.special.gammainc(0.5 * k, 0.5 * x) / scipy.special.gamma(0.5 * > k) > It appears to be simply a naming/expectation issue. What is being calculated with scipy.special.gammainc is not what e.g. Wikipedia calls the lower incomplete gamma function but instead is what it calls the regularized gamma function, see [1]. MATLAB's naming is consistent with Scipy's (not with Wikipedia's), see [2]. In fact, both scipy.stats.chi2.cdf and scipy.special.gammainc call the exact same underlying C function, it's just that scipy.stats.chi2.cdf divides the arguments by 2 first. You'll notice that the discrepancy-ratios you give are just equal to scipy.special.gamma(0.5 * k), k = 1, 2, ..., 20. [1] http://en.wikipedia.org/wiki/Incomplete_gamma_function#Regularized_Gamma_functions_and_Poisson_random_variables [2] http://www.mathworks.com/help/matlab/ref/gammainc.html Cheers, Chad -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 26 19:00:46 2015 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Mar 2015 23:00:46 +0000 Subject: [SciPy-User] MCTS Package (Monte Carlo Tree Search) In-Reply-To: <20150325101224.29178.16846@quirm.robotics.tu-berlin.de> References: <20150325101224.29178.16846@quirm.robotics.tu-berlin.de> Message-ID: On Wed, Mar 25, 2015 at 10:12 AM, Johannes Kulick < johannes.kulick at ipvs.uni-stuttgart.de> wrote: > > Hi all, > > over the last couple of days I wrote a lightweight monte carlo tree search > package for some experiments I want to do for my thesis. It is far from > complete etc. but I figured, that there is virtually no MCTS library around in > any language so far (did I miss anything?). So it might be useful for some. > You can find it at my github repository: > > https://github.com/hildensia/mcts > > I consider making it a "scikit.mcts" or the like, but it seems so little so far > that I'm afraid it might not worth doing it. Cool, thanks! I was about to dive in and write one myself for a personal project. This looks nice! Here is another Python implementation that I found: https://github.com/glesica/mcts-project -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From chadfulton at gmail.com Fri Mar 27 02:43:52 2015 From: chadfulton at gmail.com (Chad Fulton) Date: Thu, 26 Mar 2015 23:43:52 -0700 Subject: [SciPy-User] discrepancy in chi square cumulative distribution and gamma functions? In-Reply-To: References: Message-ID: On Thu, Mar 26, 2015 at 9:51 AM, James Jensen wrote: > Thanks, Chad. > > My reason for looking into this was that I am working with cdf values that > are very close to zero or one, such that chi2.cdf rounds them to 0.0 or 1.0. > I hope that by working in log space I could distinguish between them. So to > get the log of the cdf, I was considering using something along the lines of > > ln(lower_incomplete_gamma(k/2, x/2)) - ln(gamma(k/2)) > > and scipy.special has a gammaln function. Unfortunately from what I can tell, this will not be a viable strategy given the built-in Scipy functions, since the non-regularized function is not exposed. However the underlying calculation using power series is not difficult and you may be able to modify it to fit your needs. See http://project-mathlibs.web.cern.ch/project-mathlibs/sw/5_18_00/html/SpecFuncCephes_8cxx-source.html line 124 for the function calculating the regularized lower incomplete gamma. One line 135, you can see where they perform the regularization (i.e. subtracting the log gamma component). (I am not certain what the licensing terms of the Cephes library are.) > > I then realized that scipy.stats.chi2 has a logcdf function. But it doesn't > appear to have the additional precision I'm looking for. Is it possible that > it's implemented first computing the cdf, and then taking the logarithm of > it? If so, the precision would be lost prior to taking the logarithm. > You are correct, it simply returns the log of whatever is returned by the cdf function. Good luck, Chad From lamblinp at iro.umontreal.ca Fri Mar 27 12:13:22 2015 From: lamblinp at iro.umontreal.ca (Pascal Lamblin) Date: Fri, 27 Mar 2015 17:13:22 +0100 Subject: [SciPy-User] Announcing Theano 0.7 Message-ID: <20150327161322.GC25634@bob.blip.be> =========================== Announcing Theano 0.7 =========================== This is a release for a major version, with lots of new features, bug fixes, and some interface changes (deprecated or potentially misleading features were removed). Upgrading to Theano 0.7 is recommended for everyone, but you should first make sure that your code does not raise deprecation warnings with the version you are currently using. For those using the bleeding edge version in the git repository, we encourage you to update to the `rel-0.7` tag. What's New ---------- Highlights: * Integration of CuDNN for 2D convolutions and pooling on supported GPUs * Too many optimizations and new features to count * Various fixes and improvements to scan * Better support for GPU on Windows * On Mac OS X, clang is used by default * Many crash fixes * Some bug fixes as well Description ----------- Theano is a Python library that allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays. It is built on top of NumPy. Theano features: * tight integration with NumPy: a similar interface to NumPy's. numpy.ndarrays are also used internally in Theano-compiled functions. * transparent use of a GPU: perform data-intensive computations up to 140x faster than on a CPU (support for float32 only). * efficient symbolic differentiation: Theano can compute derivatives for functions of one or many inputs. * speed and stability optimizations: avoid nasty bugs when computing expressions such as log(1+ exp(x)) for large values of x. * dynamic C code generation: evaluate expressions faster. * extensive unit-testing and self-verification: includes tools for detecting and diagnosing bugs and/or potential problems. Theano has been powering large-scale computationally intensive scientific research since 2007, but it is also approachable enough to be used in the classroom (IFT6266 at the University of Montreal). Resources --------- About Theano: http://deeplearning.net/software/theano/ Related projects: http://github.com/Theano/Theano/wiki/Related-projects About NumPy: http://numpy.scipy.org/ About SciPy: http://www.scipy.org/ Machine Learning Tutorial with Theano on Deep Architectures: http://deeplearning.net/tutorial/ Acknowledgments --------------- I would like to thank all contributors of Theano. For this particular release, many people have helped, and to list them all would be impractical. I would also like to thank users who submitted bug reports. Also, thank you to all NumPy and Scipy developers as Theano builds on their strengths. All questions/comments are always welcome on the Theano mailing-lists ( http://deeplearning.net/software/theano/#community ) -- Pascal From kevin.gullikson at gmail.com Fri Mar 27 15:40:02 2015 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Fri, 27 Mar 2015 14:40:02 -0500 Subject: [SciPy-User] Talk at Scipy Conference Message-ID: Hey all, sorry if this is slightly off topic. I am thinking about applying to give a talk at the Scipy 2015 conference. Does anyone here know what happens if my talk is declined? At the astronomy conferences I have been to, you automatically just give a poster if your talk is not accepted. Is that the case at Scipy? Thanks! Kevin Gullikson -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Mar 27 16:53:43 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 27 Mar 2015 13:53:43 -0700 Subject: [SciPy-User] Talk at Scipy Conference In-Reply-To: References: Message-ID: Hi Kevin, On 27 March 2015 at 12:40, Kevin Gullikson wrote: > Hey all, sorry if this is slightly off topic. I am thinking about applying > to give a talk at the Scipy 2015 conference. Does anyone here know what > happens if my talk is declined? At the astronomy conferences I have been > to, you automatically just give a poster if your talk is not accepted. Is > that the case at Scipy? I know people who have submitted a talk proposal for Scipy in previous years and had it accepted as a poster, but I don't think that's automatic for everyone whose talk is declined. Hopefully someone will correct me if I'm wrong. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bblais at gmail.com Fri Mar 27 19:28:05 2015 From: bblais at gmail.com (Brian Blais) Date: Fri, 27 Mar 2015 19:28:05 -0400 Subject: [SciPy-User] speed of logpdf functions in scipy.stats Message-ID: Hello, I was playing with some MCMC methods which require logpdf for different distributions, and thought "hey! I can use the scipy.stats.distributions!". Then, when I tested them, they seemed slow. Upon comparison, I noticed a huge speed difference between these functions and my own not-so-cleverly written python only functions. Is there a way to get better performance? Am I doing something silly here, and creating unneeded objects somewhere? The simplest code which shows the issue is below. thanks, bb -- ----------------- bblais at gmail.com http://web.bryant.edu/~bblais import numpy as np from scipy.stats import distributions as D def lognormalpdf(x,mn,sig): # 1/sqrt(2*pi*sigma^2)*exp(-x^2/2/sigma^2) return -0.5*log(2*np.pi*sig**2)- (x-mn)**2/sig**2/2.0 x=np.random.rand(5) print lognormalpdf(x,0,1) print D.norm.logpdf(x,0,1) x=np.random.rand(15000) %timeit y=lognormalpdf(x,0,1) # 10000 loops, best of 3: 66.9 ?s per loop %timeit y=D.norm.logpdf(x,0,1) # 1000 loops, best of 3: 727 ?s per loop From josef.pktd at gmail.com Fri Mar 27 20:01:25 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Mar 2015 20:01:25 -0400 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 7:28 PM, Brian Blais wrote: > Hello, > > I was playing with some MCMC methods which require logpdf for > different distributions, and thought "hey! I can use the > scipy.stats.distributions!". Then, when I tested them, they seemed > slow. Upon comparison, I noticed a huge speed difference between > these functions and my own not-so-cleverly written python only > functions. Is there a way to get better performance? Am I doing > something silly here, and creating unneeded objects somewhere? The > simplest code which shows the issue is below. > > thanks, > > bb > > -- > ----------------- > > bblais at gmail.com > http://web.bryant.edu/~bblais > > import numpy as np > from scipy.stats import distributions as D > > def lognormalpdf(x,mn,sig): > # 1/sqrt(2*pi*sigma^2)*exp(-x^2/2/sigma^2) > return -0.5*log(2*np.pi*sig**2)- (x-mn)**2/sig**2/2.0 > > x=np.random.rand(5) > print lognormalpdf(x,0,1) > print D.norm.logpdf(x,0,1) > > x=np.random.rand(15000) > > %timeit y=lognormalpdf(x,0,1) > # 10000 loops, best of 3: 66.9 ?s per loop > > %timeit y=D.norm.logpdf(x,0,1) > # 1000 loops, best of 3: 727 ?s per loop You can compare norm.logpdf(x,0,1) and norm._logpdf(x,0,1) to get an estimate on the overhead of the argument checking. The leading underscore method is the distribution specific or generic implementation that does not include the generic loc scale handling and argument checking. If the "private" method is much slower than your implementation, then it might be inefficiently implemented, or have costly handling of edge cases. The overhead of the argument checking and generic loc scale handling is pretty much unavoidable in the current implementation. Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From bblais at gmail.com Fri Mar 27 20:13:14 2015 From: bblais at gmail.com (Brian Blais) Date: Fri, 27 Mar 2015 20:13:14 -0400 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 8:01 PM, wrote: > On Fri, Mar 27, 2015 at 7:28 PM, Brian Blais wrote: > You can compare > norm.logpdf(x,0,1) > and > norm._logpdf(x,0,1) > > to get an estimate on the overhead of the argument checking. I thought there would be something like that. However, when I try y=D.norm._logpdf(x,0,1) I get an error: TypeError: _logpdf() takes exactly 2 arguments (4 given) scipy version 0.15.1, anaconda distribution. thanks, bb -- ----------------- bblais at gmail.com http://web.bryant.edu/~bblais From robert.kern at gmail.com Sat Mar 28 07:35:43 2015 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 28 Mar 2015 11:35:43 +0000 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 12:13 AM, Brian Blais wrote: > > On Fri, Mar 27, 2015 at 8:01 PM, wrote: > > On Fri, Mar 27, 2015 at 7:28 PM, Brian Blais wrote: > > You can compare > > norm.logpdf(x,0,1) > > and > > norm._logpdf(x,0,1) > > > > to get an estimate on the overhead of the argument checking. > > I thought there would be something like that. However, when I try > > y=D.norm._logpdf(x,0,1) > > I get an error: TypeError: _logpdf() takes exactly 2 arguments (4 given) > > scipy version 0.15.1, anaconda distribution. He meant norm._logpdf(x). -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From bblais at gmail.com Sat Mar 28 13:01:42 2015 From: bblais at gmail.com (Brian Blais) Date: Sat, 28 Mar 2015 13:01:42 -0400 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 7:35 AM, Robert Kern wrote: > > He meant norm._logpdf(x). > ah, that makes more sense....and it's faster than my python function. however, this clearly works only in the case of mu=0, sd=1. for the normal it's easy to transform, but my goal is to have fast version of the logpdf's of the different scipy.stats distributions, where each call may have *different* distribution parameters. is there a fast _logpdf-type version for the distributions where you can specify value, scale, etc...? thanks, bb -- ----------------- bblais at gmail.com http://web.bryant.edu/~bblais From robert.kern at gmail.com Sat Mar 28 13:14:48 2015 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 28 Mar 2015 17:14:48 +0000 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 5:01 PM, Brian Blais wrote: > > On Sat, Mar 28, 2015 at 7:35 AM, Robert Kern wrote: > > > > He meant norm._logpdf(x). > > > > ah, that makes more sense....and it's faster than my python function. > however, this clearly works only in the case of mu=0, sd=1. for the > normal it's easy to transform, but my goal is to have fast version of > the logpdf's of the different scipy.stats distributions, where each > call may have *different* distribution parameters. is there a fast > _logpdf-type version for the distributions where you can specify > value, scale, etc...? The difference in overhead between norm.logpdf() and norm._logpdf() is mostly just taking care of the location and scale parameters. It's the same for every distribution, which is why we have it organized this way. There is perhaps a little more overhead than strictly necessary because it needs to handle the generic argument manipulation for those distributions that take shape parameters in addition to location and scale. If you want to write a bunch of repetitive boilerplate to do the location and scale manipulation for each distribution separately, that is likely the only way to improve on the overhead. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Mar 28 13:30:44 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 28 Mar 2015 13:30:44 -0400 Subject: [SciPy-User] speed of logpdf functions in scipy.stats In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 1:14 PM, Robert Kern wrote: > On Sat, Mar 28, 2015 at 5:01 PM, Brian Blais wrote: >> >> On Sat, Mar 28, 2015 at 7:35 AM, Robert Kern >> wrote: >> > >> > He meant norm._logpdf(x). >> > >> >> ah, that makes more sense....and it's faster than my python function. >> however, this clearly works only in the case of mu=0, sd=1. for the >> normal it's easy to transform, but my goal is to have fast version of >> the logpdf's of the different scipy.stats distributions, where each >> call may have *different* distribution parameters. is there a fast >> _logpdf-type version for the distributions where you can specify >> value, scale, etc...? > > The difference in overhead between norm.logpdf() and norm._logpdf() is > mostly just taking care of the location and scale parameters. It's the same > for every distribution, which is why we have it organized this way. There is > perhaps a little more overhead than strictly necessary because it needs to > handle the generic argument manipulation for those distributions that take > shape parameters in addition to location and scale. If you want to write a > bunch of repetitive boilerplate to do the location and scale manipulation > for each distribution separately, that is likely the only way to improve on > the overhead. generic loc scale handling would be just dist._logpdf((x - loc) / scale) - np.log(scale) There is also checking whether x is in the domain, which is relevant in cases with a lower or upper bound, like log-normal, poisson, beta, besides the generic checking for valid shape parameters. Josef > > -- > Robert Kern > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jonathanrocher at gmail.com Sat Mar 28 14:34:33 2015 From: jonathanrocher at gmail.com (Jonathan Rocher) Date: Sat, 28 Mar 2015 13:34:33 -0500 Subject: [SciPy-User] Talk at Scipy Conference In-Reply-To: References: Message-ID: Hi Kevin, I think that Thomas is correct: in the past few years, we had around 1/3 talks, 1/3 posters and 1/3 rejections. It is unfortunate, but we don't have enough poster spots to give one to anyone one who applies. Good luck with your submission and hope to see you this summer. Best, Jonathan On Fri, Mar 27, 2015 at 3:53 PM, Thomas Kluyver wrote: > Hi Kevin, > > On 27 March 2015 at 12:40, Kevin Gullikson > wrote: > >> Hey all, sorry if this is slightly off topic. I am thinking about >> applying to give a talk at the Scipy 2015 conference. Does anyone here know >> what happens if my talk is declined? At the astronomy conferences I have >> been to, you automatically just give a poster if your talk is not accepted. >> Is that the case at Scipy? > > > I know people who have submitted a talk proposal for Scipy in previous > years and had it accepted as a poster, but I don't think that's automatic > for everyone whose talk is declined. Hopefully someone will correct me if > I'm wrong. > > Thomas > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Jonathan Rocher Austin TX, USA Cell : +1 512 501 0865 http://www.linkedin.com/in/jonathanrocher ------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From rokroskar at gmail.com Mon Mar 30 10:51:15 2015 From: rokroskar at gmail.com (Rok Roskar) Date: Mon, 30 Mar 2015 16:51:15 +0200 Subject: [SciPy-User] strange error when creating csr_matrix Message-ID: I am trying to create a csr_matrix with very large indices (> 2 x 10^9). This works most of the time, but occasionally I get an error "could not convert integer scalar". Here is a small snippet that reproduces the problem: In [145]: inds Out[145]: array([ 827969148, 996833913, 1968345558, 898183169, 1811744124, 2101454109, 133039182, 898183170, 919293479, 133039089]) In [146]: vals Out[146]: array([ 1., 1., 1., 1., 1., 2., 1., 1., 1., 1.]) In [147]: max_index Out[147]: 2337713000 In [143]: csr_matrix((vals, (np.zeros(10), inds)), shape = (1, max_index+1)) ... 996 fn = _sparsetools.csr_sum_duplicates 997 M,N = self._swap(self.shape) --> 998 fn(M, N, self.indptr, self.indices, self.data) 999 1000 self.prune() # nnz may have changed ValueError: could not convert integer scalar I thought it might be that the csr_matrix uses 32-bit integers for index arrays, but this works fine: In [11]: csr_matrix(([1,1], ([0,0], [1,2**33])), shape = (1, 2**34)).nonzero() Out[11]: (array([0, 0]), array([ 1, 8589934592])) I would appreciate any ideas as to what might be the problem -- I have also opened a question on stackoverflow: http://stackoverflow.com/questions/29168699/cryptic-scipy-could-not-convert-integer-scalar-error Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Mar 30 11:25:21 2015 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 30 Mar 2015 18:25:21 +0300 Subject: [SciPy-User] strange error when creating csr_matrix In-Reply-To: References: Message-ID: 30.03.2015, 17:51, Rok Roskar kirjoitti: [clip] > In [147]: max_index > > Out[147]: > 2337713000 > > In [143]: csr_matrix((vals, (np.zeros(10), inds)), shape = (1, max_index+1)) > ... > > 996 fn = _sparsetools.csr_sum_duplicates > 997 M,N = self._swap(self.shape) > --> 998 fn(M, N, self.indptr, self.indices, self.data) > 999 > 1000 self.prune() # nnz may have changed > > ValueError: could not convert integer scalar If your max_index is that large, you must ensure your index arrays `np.zeros(10)` and `inds` are both 64-bit integer type. From nxkryptor at gmail.com Tue Mar 31 00:56:54 2015 From: nxkryptor at gmail.com (nxkryptor nxkr) Date: Tue, 31 Mar 2015 07:56:54 +0300 Subject: [SciPy-User] (no subject) Message-ID: 1 from __future__ import division 2 from scipy import optimize 3 import matplotlib.pyplot as plt 4 import numpy as np 5 6 data = open('data.dat', "r") 7 line = data.readline() 8 while line.startswith('#'): 9 line = data.readline()10 data_header = line.split("\t")11 data_header[-1] = data_header[-1].strip()12 13 14 _data_ = np.genfromtxt('data.dat', skiprows=2, delimiter='\t', names = data_header, dtype = None, unpack = True).transpose()15 data = np.array(_data_.tolist())16 m = data.shape[0]17 n = data.shape[1] - 218 print m, n19 y_data = np.empty(shape=(m, n))20 for i in range(0, m):21 for j in range(0, n):22 y_data[i, j] = (data[i, j+2])23 x = _data_['X']24 z = _data_['Z']25 26 def quadratic_fit(x, a, b, c):27 return a * x ** 2 + b * x + c28 29 fit_a = np.empty(shape = (m, 1))30 fit_b = np.empty(shape = (m, 1))31 fit_c = np.empty(shape = (m, 1)) z_fit = np.empty(shape=(m, len(z)))for j in range(m): x_fit = y_data[j, :] y_fit = z fit_a[j], fit_b[j], fit_c[j] = optimize.curve_fit(quadratic_fit, x_fit, y_fit)[0] fit_a_fit_a, fit_a_fit_b, fit_a_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_a)[0] fit_b_fit_a, fit_b_fit_b, fit_b_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_b)[0] fit_c_fit_a, fit_c_fit_b, fit_c_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_c)[0] fit_a = fit_a_fit_a * x ** 2 + fit_a_fit_b * x + fit_a_fit_c fit_b = fit_b_fit_a * x ** 2 + fit_b_fit_b * x + fit_b_fit_c fit_c = fit_c_fit_a * x ** 2 + fit_c_fit_b * x + fit_c_fit_cfor j in range(m): z_fit[j, :] = (fit_a[j] * x_fit ** 2) + (fit_b[j] * x_fit) + fit_c[j] -------------- next part -------------- An HTML attachment was scrubbed... URL: From nxkryptor at gmail.com Tue Mar 31 01:00:16 2015 From: nxkryptor at gmail.com (nxkryptor nxkr) Date: Tue, 31 Mar 2015 08:00:16 +0300 Subject: [SciPy-User] How to curve fit a set of fitting parameters in Python? Message-ID: Dear SciPy users, I am trying to curve fit a data (arbitrary) set as given below: # Data T Z X 1 X 2 X 3 X 4 X 5 100.000 1.000 1.000 1.478 1.304 1.162 0.805 200.000 1.500 2.000 2.314 2.168 2.086 1.801 300.000 2.250 3.000 3.246 3.114 3.058 2.798 400.000 3.375 4.000 4.211 4.087 4.044 3.780 500.000 5.063 5.000 5.189 5.070 5.035 4.780 At first it is curve fit using a quadratic expression Z = a * x ^ 2 + b * x + c along a constant value of T i.e. along each rows, which gives as fit parameters for each T as given below (as an example): T a b c 100.00 1.00 2.10 10.02 200.00 4.00 6.20 10.06 300.00 9.00 12.30 10.12 400.00 16.00 20.40 10.20 500.00 25.00 30.50 10.31 Now I would like to fit each fit parameters along T so that I get the equations of the form a = p * T ^ 2 + q * T + q, b = s * T + u, etc. I tried applying this using the code: from __future__ import division from scipy import optimize import matplotlib.pyplot as plt import numpy as np data = open('data.dat', "r") line = data.readline() while line.startswith('#'): line = data.readline() data_header = line.split("\t") data_header[-1] = data_header[-1].strip() _data_ = np.genfromtxt('data.dat', skiprows=2, delimiter='\t', names = data_header, dtype = None, unpack = True).transpose() data = np.array(_data_.tolist()) m = data.shape[0] n = data.shape[1] - 2 print m, n y_data = np.empty(shape=(m, n)) for i in range(0, m): for j in range(0, n): y_data[i, j] = (data[i, j+2]) x = _data_['X'] z = _data_['Z'] def quadratic_fit(x, a, b, c): return a * x ** 2 + b * x + c fit_a = np.empty(shape = (m, 1)) fit_b = np.empty(shape = (m, 1)) fit_c = np.empty(shape = (m, 1)) z_fit = np.empty(shape=(m, len(z))) for j in range(m): x_fit = y_data[j, :] y_fit = z fit_a[j], fit_b[j], fit_c[j] = optimize.curve_fit(quadratic_fit, x_fit, y_fit)[0] fit_a_fit_a, fit_a_fit_b, fit_a_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_a)[0] fit_b_fit_a, fit_b_fit_b, fit_b_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_b)[0] fit_c_fit_a, fit_c_fit_b, fit_c_fit_c, = optimize.curve_fit(quadratic_fit, x, fit_c)[0] fit_a = fit_a_fit_a * x ** 2 + fit_a_fit_b * x + fit_a_fit_c fit_b = fit_b_fit_a * x ** 2 + fit_b_fit_b * x + fit_b_fit_c fit_c = fit_c_fit_a * x ** 2 + fit_c_fit_b * x + fit_c_fit_c for j in range(m): z_fit[j, :] = (fit_a[j] * x_fit ** 2) + (fit_b[j] * x_fit) + fit_c[j] This question has already been posted on Stackoverflow . Thank you. nxkr -------------- next part -------------- An HTML attachment was scrubbed... URL: From rokroskar at gmail.com Tue Mar 31 03:34:31 2015 From: rokroskar at gmail.com (Rok Roskar) Date: Tue, 31 Mar 2015 09:34:31 +0200 Subject: [SciPy-User] strange error when creating csr_matrix Message-ID: Hi Pauli, thanks for the idea. I think the arrays are int64 by default, so this doesn't seem make a difference: In [23]: csr_matrix((vals.astype(np.int64), (np.zeros(10,dtype=np.int64), inds.astype(np.int64))), shape = (1, max_index+1)) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) ValueError: could not convert integer scalar Does it work on your machine? > Message: 5 > Date: Mon, 30 Mar 2015 16:51:15 +0200 > From: Rok Roskar > Subject: [SciPy-User] strange error when creating csr_matrix > To: scipy-user at scipy.org > Message-ID: > < > CAOufH8WVUzZSHc_RJSPequqe3YA+YH75Rb08j++yjS1s9PRsWA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I am trying to create a csr_matrix with very large indices (> 2 x 10^9). > This works most of the time, but occasionally I get an error "could not > convert integer scalar". > > Here is a small snippet that reproduces the problem: > > In [145]: inds > > Out[145]: > array([ 827969148, 996833913, 1968345558, 898183169, 1811744124, > 2101454109, 133039182, 898183170, 919293479, 133039089]) > > In [146]: vals > > Out[146]: > array([ 1., 1., 1., 1., 1., 2., 1., 1., 1., 1.]) > > In [147]: max_index > > Out[147]: > 2337713000 > > In [143]: csr_matrix((vals, (np.zeros(10), inds)), shape = (1, > max_index+1)) > ... > > 996 fn = _sparsetools.csr_sum_duplicates > 997 M,N = self._swap(self.shape) > --> 998 fn(M, N, self.indptr, self.indices, self.data) > 999 > 1000 self.prune() # nnz may have changed > > ValueError: could not convert integer scalar > > > > I thought it might be that the csr_matrix uses 32-bit integers for index > arrays, but this works fine: > > In [11]: csr_matrix(([1,1], ([0,0], [1,2**33])), shape = (1, > 2**34)).nonzero() > Out[11]: (array([0, 0]), array([ 1, 8589934592])) > > > I would appreciate any ideas as to what might be the problem -- I have also > opened a question on stackoverflow: > > http://stackoverflow.com/questions/29168699/cryptic-scipy-could-not-convert-integer-scalar-error > > > Thanks! > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/scipy-user/attachments/20150330/a90c22cc/attachment-0001.html > > ------------------------------ > > Message: 6 > Date: Mon, 30 Mar 2015 18:25:21 +0300 > From: Pauli Virtanen > Subject: Re: [SciPy-User] strange error when creating csr_matrix > To: scipy-user at scipy.org > Message-ID: > Content-Type: text/plain; charset=windows-1252 > > 30.03.2015, 17:51, Rok Roskar kirjoitti: > [clip] > > In [147]: max_index > > > > Out[147]: > > 2337713000 > > > > In [143]: csr_matrix((vals, (np.zeros(10), inds)), shape = (1, > max_index+1)) > > ... > > > > 996 fn = _sparsetools.csr_sum_duplicates > > 997 M,N = self._swap(self.shape) > > --> 998 fn(M, N, self.indptr, self.indices, self.data) > > 999 > > 1000 self.prune() # nnz may have changed > > > > ValueError: could not convert integer scalar > > If your max_index is that large, you must ensure your index arrays > `np.zeros(10)` and `inds` are both 64-bit integer type. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Tue Mar 31 12:02:01 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 31 Mar 2015 18:02:01 +0200 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: On Tue, Mar 24, 2015 at 9:39 PM, Todd wrote: > Is anyone aware of a well-maintained, simple, cross-platform python > package that can play a numpy array as sound over speakers? > > I am aware of https://wiki.python.org/moin/Audio/. However, in all the > cases there, as far as I can find they either do not support numpy arrays, > are not cross-platform, cannot playback sound at all, or are unmaintained. > There is also PySoundCard, which would do what I need but also appears to > be unmaintained (no release in over a year, and no commits in 5 months, no > release with serious bugfixes mentioned in commits). > So in terms of raw waveform playback (as opposed to music note playback), I have done some more searching and I think I have found something that works. It is the "audio.io" package (https://pypi.python.org/pypi/audio.io/). It has a recent release (late 2014), supports numpy arrays, and is cross-platform through PyAudio. It is just a VERY thin wrapper around PyAudio (less than 100 lines). However, there is no website, no issue tracker, essentially no documentation, and has several projects copied into its tarball (including setputools, about, and sh). Here are the reasonably maintained, reasonably relevant alternatives I have been able to find: PyAudio: maintained, cross-platform, doesn't support numpy. It seems to be used as a backend by a lot of other projects. audiolazy: cross-platform, supports numpy, has not seen a release since 2013 but its github repo is still seeing commits so it may have more releases in the future. Uses PyAudio. Provides a lot of other powerful audio-handling and audio-processing capabilities. PySoundCard: cross-platform, supports numpy, has not seen a release in over a year and its github repo has not seen a commit in 5 months, but another related project (PySoundFile) has seen commits and releases recently. The only option amongst these that does NOT rely on PyAudio. pydub: maintained, cross-platform, doesn't appear to support numpy but the audio output is undocumented so I can't be sure. Uses PyAudio or ffmpeg if PyAudio is not available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Mar 31 12:35:31 2015 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 31 Mar 2015 19:35:31 +0300 Subject: [SciPy-User] strange error when creating csr_matrix In-Reply-To: References: Message-ID: 31.03.2015, 10:34, Rok Roskar kirjoitti: > Hi Pauli, thanks for the idea. I think the arrays are int64 by default, so > this doesn't seem make a difference: > > In [23]: csr_matrix((vals.astype(np.int64), (np.zeros(10,dtype=np.int64), > inds.astype(np.int64))), shape = (1, max_index+1)) > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > > > > ValueError: could not convert integer scalar > > Does it work on your machine? Turns out this is actually a bug: https://github.com/scipy/scipy/pull/4678 csr_matrix constructor thinks 32-bit integers are enough, but this is not the case. From rokroskar at gmail.com Tue Mar 31 12:51:38 2015 From: rokroskar at gmail.com (Rok Roskar) Date: Tue, 31 Mar 2015 18:51:38 +0200 Subject: [SciPy-User] strange error when creating csr_matrix In-Reply-To: References: Message-ID: aha good to know, thanks -- I suppose there isn't an easy work-around for the time being? On Tue, Mar 31, 2015 at 6:35 PM, Pauli Virtanen wrote: > 31.03.2015, 10:34, Rok Roskar kirjoitti: > > Hi Pauli, thanks for the idea. I think the arrays are int64 by default, > so > > this doesn't seem make a difference: > > > > In [23]: csr_matrix((vals.astype(np.int64), (np.zeros(10,dtype=np.int64), > > inds.astype(np.int64))), shape = (1, max_index+1)) > > > --------------------------------------------------------------------------- > > ValueError Traceback (most recent call > last) > > > > > > > > ValueError: could not convert integer scalar > > > > Does it work on your machine? > > Turns out this is actually a bug: > https://github.com/scipy/scipy/pull/4678 > > csr_matrix constructor thinks 32-bit integers are enough, but this is > not the case. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Mar 31 14:07:50 2015 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 31 Mar 2015 21:07:50 +0300 Subject: [SciPy-User] strange error when creating csr_matrix In-Reply-To: References: Message-ID: 31.03.2015, 19:51, Rok Roskar kirjoitti: > aha good to know, thanks -- I suppose there isn't an easy work-around for > the time being? You can monkeypatch: import scipy import scipy.sparse if scipy.__version__ in ("0.14.0", "0.14.1", "0.15.1"): _get_index_dtype = scipy.sparse.sputils.get_index_dtype def _my_get_index_dtype(*a, **kw): kw.pop('check_contents', None) return _get_index_dtype(*a, **kw) scipy.sparse.compressed.get_index_dtype = _my_get_index_dtype scipy.sparse.csr.get_index_dtype = _my_get_index_dtype scipy.sparse.bsr.get_index_dtype = _my_get_index_dtype This will prevent the constructor from switching from int64 to int32. Be sure to include the `__version__` check. From bhmerchant at gmail.com Tue Mar 31 15:29:41 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Tue, 31 Mar 2015 12:29:41 -0700 Subject: [SciPy-User] Playing numpy array over speakers In-Reply-To: References: Message-ID: Thanks so much for updating us on what you have found! I am very glad to be able to have a chance to explore the possible "solutions" you found! On Tue, Mar 31, 2015 at 9:02 AM, Todd wrote: > On Tue, Mar 24, 2015 at 9:39 PM, Todd wrote: > >> Is anyone aware of a well-maintained, simple, cross-platform python >> package that can play a numpy array as sound over speakers? >> >> I am aware of https://wiki.python.org/moin/Audio/. However, in all the >> cases there, as far as I can find they either do not support numpy arrays, >> are not cross-platform, cannot playback sound at all, or are unmaintained. >> There is also PySoundCard, which would do what I need but also appears to >> be unmaintained (no release in over a year, and no commits in 5 months, no >> release with serious bugfixes mentioned in commits). >> > > So in terms of raw waveform playback (as opposed to music note playback), > I have done some more searching and I think I have found something that > works. It is the "audio.io" package ( > https://pypi.python.org/pypi/audio.io/). It has a recent release (late > 2014), supports numpy arrays, and is cross-platform through PyAudio. It is > just a VERY thin wrapper around PyAudio (less than 100 lines). However, > there is no website, no issue tracker, essentially no documentation, and > has several projects copied into its tarball (including setputools, about, > and sh). > > Here are the reasonably maintained, reasonably relevant alternatives I > have been able to find: > > PyAudio: maintained, cross-platform, doesn't support numpy. It seems to > be used as a backend by a lot of other projects. > > audiolazy: cross-platform, supports numpy, has not seen a release since > 2013 but its github repo is still seeing commits so it may have more > releases in the future. Uses PyAudio. Provides a lot of other powerful > audio-handling and audio-processing capabilities. > > PySoundCard: cross-platform, supports numpy, has not seen a release in > over a year and its github repo has not seen a commit in 5 months, but > another related project (PySoundFile) has seen commits and releases > recently. The only option amongst these that does NOT rely on PyAudio. > > pydub: maintained, cross-platform, doesn't appear to support numpy but the > audio output is undocumented so I can't be sure. Uses PyAudio or ffmpeg if > PyAudio is not available. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: