From ralf.gommers at gmail.com Tue Nov 1 04:09:33 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 1 Nov 2016 21:09:33 +1300 Subject: [SciPy-Dev] www.scipy.org down In-Reply-To: References: Message-ID: The open issue for this is https://github.com/scipy/scipy.org/issues/187 Ralf On Tue, Nov 1, 2016 at 3:45 PM, Warren Weckesser wrote: > I was getting the following error when I tried to go to > https://www.scipy.org: > > Secure Connection Failed > > An error occurred during a connection to www.scipy.org. SSL received a > record that exceeded the maximum permissible length. Error code: > SSL_ERROR_RX_RECORD_TOO_LONG > > The page you are trying to view cannot be shown because the > authenticity of the received data could not be verified. > Please contact the website owners to inform them of this problem. > > > Now I'm getting: > > Unable to connect > > Firefox can?t establish a connection to the server at www.scipy.org. > > The site could be temporarily unavailable or too busy. Try again in a > few moments. > If you are unable to load any pages, check your computer?s network > connection. > If your computer or network is protected by a firewall or proxy, make > sure that Firefox is permitted to access the Web. > > > Warren > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.open at rehammar.se Thu Nov 3 15:37:07 2016 From: robert.open at rehammar.se (Robert Rehammar) Date: Thu, 3 Nov 2016 20:37:07 +0100 Subject: [SciPy-Dev] Differentiate function Message-ID: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> Dear all, I implemented a simple function to differentiate an array which seems other people might like to have. It looks like: def diff(f, x, retval = 'same'): """ Differentiate f w.r.t. x and return the diff with same samples, or different depending on retval. retval can be: same: return data for same data points as the indata. This is done through interpolation at internal points and extrapolation at endpoint simple: just return the scaled diff samesimple: return with same size vector, just shifted. Last data point is repeated. new: return at new datapoints and length - 1 """ df = f[1:] - f[:-1] dx = x[1:] - x[:-1] dfodx = df/dx if retval == 'new': return dfodx, x[:-1] + dx/2 elif retval == 'same': cdfodx = np.zeros(len(f)) cdfodx[1:-1] = (dfodx[:-1] + dfodx[1:])/2 cdfodx[0] = dfodx[0]-0.5*(dfodx[1] - dfodx[0])/dx[0] cdfodx[-1] = dfodx[-1]+0.5*(dfodx[-1] - dfodx[-2])/dx[-1] return cdfodx, x elif retval == 'simple': return dfodx, x[:-1] elif retval == 'samesimple': cdfodx = np.zeros(len(f)) cdfodx[:-1] = dfodx cdfodx[-1] = dfodx[-1] return cdfodx, x And generates the following results (top plot original function, bottom different differentiates) from the code blow: x = np.linspace(0,2*np.pi,10) f = np.sin(x) plt.subplot(211) plt.plot(x, f, marker="*") plt.ylabel('$\sin(y)$') plt.grid(True) plt.subplot(212) df,xp = diff(f, x, 'simple') plt.plot(xp, df, marker="*", label='simple', linewidth=4) df,xp = diff(f, x, 'new') plt.plot(xp, df, marker="*", label='new') df,xp = diff(f, x, 'same') plt.plot(xp, df, marker="*", label='same') df,xp = diff(f, x, 'samesimple') plt.plot(xp, df, marker="*", label='samesimple') plt.ylabel('$d\sin(x)/dx$') plt.xlabel('$x$') plt.grid(True) plt.legend(loc='lower right') What do you think about adding it to scipy? Best, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blklkpdhkkdlagef.png Type: image/png Size: 51705 bytes Desc: not available URL: From evgeny.nekrasov at phystech.edu Fri Nov 4 06:40:40 2016 From: evgeny.nekrasov at phystech.edu (Evgeny Nekrasov) Date: Fri, 4 Nov 2016 13:40:40 +0300 Subject: [SciPy-Dev] scipy.sparse.coo_matrix a new method drop_zero_columns? Message-ID: Hello, I am writing to follow up on the discussion https://github.com/scipy/ scipy/issues/6754 I would be happy to get your opinions. Thank you for your time, Evgeny -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Nov 5 17:44:21 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 6 Nov 2016 10:44:21 +1300 Subject: [SciPy-Dev] scipy.sparse.coo_matrix a new method drop_zero_columns? In-Reply-To: References: Message-ID: On Fri, Nov 4, 2016 at 11:40 PM, Evgeny Nekrasov < evgeny.nekrasov at phystech.edu> wrote: > Hello, > > I am writing to follow up on the discussion https://github.com/scipy/scipy > /issues/6754 > > I would be happy to get your opinions. > Hi Evgeny, it would be helpful to add to the issue description some links to code / examples where this method is used or would be useful. Right now there's just an assertion "this is useful", which is hard to evaluate. Other questions I would have: 1. The issue has a method for COO, but I'd think we want this for all or none of the formats? 2. Is it only drop column, or would drop row also make sense? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.debuyl at kuleuven.be Sat Nov 5 17:53:03 2016 From: pierre.debuyl at kuleuven.be (Pierre de Buyl) Date: Sat, 5 Nov 2016 22:53:03 +0100 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> Message-ID: <20161105215303.GA2429@pi-x230> Dear Robert, On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: > Dear all, > > I implemented a simple function to differentiate an array which seems other > people might like to have. It looks like: > def diff(f, x, retval = 'same'): > (...) > > What do you think about adding it to scipy? I am not a SciPy developer myself but I thought that I would point out NumPy's diff function. It does not go as far as your routine in terms of options but goes most of the way. It applies the differentiation to order 'n' as an option that is similar to the recursive application. Given this existing routine, the incentive to add another one might be low. Regards, Pierre From rlucente at pipeline.com Sun Nov 6 09:02:56 2016 From: rlucente at pipeline.com (Robert Lucente - Pipeline) Date: Sun, 6 Nov 2016 09:02:56 -0500 Subject: [SciPy-Dev] Math Optimization in Power Industry Message-ID: <03de01d23836$7bce9010$736bb030$@pipeline.com> I realize that this is "off topic" but I figure if I send one and only one email it might be tolerated because of potential interest to the group. I was wondering if anyone had any interest or experience applying math optimization to the power industry or familiarity w/ the book Modern Optimization Techniques with Applications in Electric Power Systems by Soliman Abdel-Hady Soliman, Abdel-Aal Hassan Mantawy http://www.springer.com/us/book/9781461417514 Please reply directly to me at rlucente at pipeline.com From evgeny.nekrasov at phystech.edu Sun Nov 6 13:10:58 2016 From: evgeny.nekrasov at phystech.edu (Evgeny Nekrasov) Date: Sun, 6 Nov 2016 21:10:58 +0300 Subject: [SciPy-Dev] scipy.sparse.coo_matrix a new method drop_zero_columns? In-Reply-To: References: Message-ID: Dear Ralf, Thank you for your response. The popular use case for drop_zero_columns is feature selection before applying machine learning algorithm. For example a very similar technique VarianceThreshold is implemented in sklearn ( http://scikit-learn.org/stable/modules/feature_selection.html). The problem with current implementations is that it fails or takes much more resources than actually needed if the amount of zero columns is really huge. Such sparse data representations often produced by popular techniques such as feature hashing, bag of words, bag of content_ids or similar. Such techniques are implemented in sklearn ( http://scikit-learn.org/stable/modules/feature_extraction.html). Nevertheless, custom implementations often needed, and here drop_zero_columns is valuable. Other questions: 1. It would be great to have drop_zero_columns for all matrix types. I wrote about this method for COO due to it is sufficient to process data with huge amount of zero columns in efficient way. 2. I don't know popular use cases for rows. Best regards, Evgeny 2016-11-06 0:44 GMT+03:00 Ralf Gommers : > > > On Fri, Nov 4, 2016 at 11:40 PM, Evgeny Nekrasov < > evgeny.nekrasov at phystech.edu> wrote: > >> Hello, >> >> I am writing to follow up on the discussion >> https://github.com/scipy/scipy/issues/6754 >> >> I would be happy to get your opinions. >> > > Hi Evgeny, it would be helpful to add to the issue description some links > to code / examples where this method is used or would be useful. Right now > there's just an assertion "this is useful", which is hard to evaluate. > > Other questions I would have: > 1. The issue has a method for COO, but I'd think we want this for all or > none of the formats? > 2. Is it only drop column, or would drop row also make sense? > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.open at rehammar.se Mon Nov 7 01:53:07 2016 From: robert.open at rehammar.se (Robert Rehammar) Date: Mon, 7 Nov 2016 07:53:07 +0100 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <20161105215303.GA2429@pi-x230> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> Message-ID: Dear Pierre, Thank you for your reply. I am note fully aware of the difference between numpy and scipy and also not a developer, but it seems to me this function could fit in scipy where you often will want to differentiate w.r.t. a particular variable. I am aware of diff, but it is really a much more basic function. In many areas of science and engineering is the sampling points of crucial importance, and having a function giving you the appropriate sampling points can be very important for accuracy. I have been missing this function several times my self, and hence the reason why I submitted it here. Anyway, if there is no interest, I will of course not peruse the matter further. Best, Robert On 2016-11-05 22:53, Pierre de Buyl wrote: > Dear Robert, > > On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >> Dear all, >> >> I implemented a simple function to differentiate an array which seems other >> people might like to have. It looks like: >> def diff(f, x, retval = 'same'): >> (...) >> >> What do you think about adding it to scipy? > I am not a SciPy developer myself but I thought that I would point out NumPy's > diff function. It does not go as far as your routine in terms of options but > goes most of the way. It applies the differentiation to order 'n' as an option > that is similar to the recursive application. Given this existing routine, the > incentive to add another one might be low. > > Regards, > > Pierre > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From jjstickel at gmail.com Mon Nov 7 11:21:01 2016 From: jjstickel at gmail.com (Jonathan Stickel) Date: Mon, 7 Nov 2016 09:21:01 -0700 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> Message-ID: <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Robert, I think, conceptually, something like this could be of interest in SciPy, but the devil is in the details. There are a lot of ways to implement finite differences; the code you shared provides the forward/backward finite differences with optional shift in the x positions. These are first-order accurate. Your interpolation of forward differences effectively results in the second-order accurate central-difference method. Evaluation of the endpoints is a bit more tricky; your approaches are simple but lack rigor (I am not sure of the accuracy). I know of formulas for second-order accurate endpoint finite differences for equally spaced x, but I am not sure about unequally spaced data (I am sure they exist but I haven't looked them up). So, should this functionality exist in SciPy, what methods should be implemented and made available? I think many of us (including myself) have implemented our own methods that are satisfactory for our particular needs, and no one has submitted general utility finite-differences functions that provides multiple methods with appropriate mathematical rigor. Such a function should also provide higher-order derivatives (at least second, but maybe up to fourth; or perhaps nth-order methods using a series formula). Also, the name should not be "diff" to avoid confusion with numpy.diff. Regards, Jonathan On 11/6/16 23:53 , Robert Rehammar wrote: > Dear Pierre, > > Thank you for your reply. I am note fully aware of the difference > between numpy and scipy and also not a developer, but it seems to me > this function could fit in scipy where you often will want to > differentiate w.r.t. a particular variable. I am aware of diff, but it > is really a much more basic function. In many areas of science and > engineering is the sampling points of crucial importance, and having a > function giving you the appropriate sampling points can be very > important for accuracy. > > I have been missing this function several times my self, and hence the > reason why I submitted it here. Anyway, if there is no interest, I will > of course not peruse the matter further. > > Best, > > Robert > > > On 2016-11-05 22:53, Pierre de Buyl wrote: >> Dear Robert, >> >> On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >>> Dear all, >>> >>> I implemented a simple function to differentiate an array which seems >>> other >>> people might like to have. It looks like: >>> def diff(f, x, retval = 'same'): >>> (...) >>> >>> What do you think about adding it to scipy? >> I am not a SciPy developer myself but I thought that I would point out >> NumPy's >> diff function. It does not go as far as your routine in terms of >> options but >> goes most of the way. It applies the differentiation to order 'n' as >> an option >> that is similar to the recursive application. Given this existing >> routine, the >> incentive to add another one might be low. >> >> Regards, >> >> Pierre >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From evgeny.burovskiy at gmail.com Mon Nov 7 11:52:09 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Mon, 7 Nov 2016 19:52:09 +0300 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: Note that `approx_derivative` implements several finite-difference schemes, https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L179 For the moment, it's well hidden in scipy.optimize, but in the long run the idea is to offer public functionality. So far the main issue was the lack of bandwidth --- meaning if someone feels like working on it, great! On Mon, Nov 7, 2016 at 7:21 PM, Jonathan Stickel wrote: > Robert, > > I think, conceptually, something like this could be of interest in SciPy, > but the devil is in the details. There are a lot of ways to implement finite > differences; the code you shared provides the forward/backward finite > differences with optional shift in the x positions. These are first-order > accurate. Your interpolation of forward differences effectively results in > the second-order accurate central-difference method. > > Evaluation of the endpoints is a bit more tricky; your approaches are simple > but lack rigor (I am not sure of the accuracy). I know of formulas for > second-order accurate endpoint finite differences for equally spaced x, but > I am not sure about unequally spaced data (I am sure they exist but I > haven't looked them up). > > So, should this functionality exist in SciPy, what methods should be > implemented and made available? I think many of us (including myself) have > implemented our own methods that are satisfactory for our particular needs, > and no one has submitted general utility finite-differences functions that > provides multiple methods with appropriate mathematical rigor. Such a > function should also provide higher-order derivatives (at least second, but > maybe up to fourth; or perhaps nth-order methods using a series formula). > > Also, the name should not be "diff" to avoid confusion with numpy.diff. > > Regards, > Jonathan > > > > > On 11/6/16 23:53 , Robert Rehammar wrote: >> >> Dear Pierre, >> >> Thank you for your reply. I am note fully aware of the difference >> between numpy and scipy and also not a developer, but it seems to me >> this function could fit in scipy where you often will want to >> differentiate w.r.t. a particular variable. I am aware of diff, but it >> is really a much more basic function. In many areas of science and >> engineering is the sampling points of crucial importance, and having a >> function giving you the appropriate sampling points can be very >> important for accuracy. >> >> I have been missing this function several times my self, and hence the >> reason why I submitted it here. Anyway, if there is no interest, I will >> of course not peruse the matter further. >> >> Best, >> >> Robert >> >> >> On 2016-11-05 22:53, Pierre de Buyl wrote: >>> >>> Dear Robert, >>> >>> On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >>>> >>>> Dear all, >>>> >>>> I implemented a simple function to differentiate an array which seems >>>> other >>>> people might like to have. It looks like: >>>> def diff(f, x, retval = 'same'): >>>> (...) >>>> >>>> What do you think about adding it to scipy? >>> >>> I am not a SciPy developer myself but I thought that I would point out >>> NumPy's >>> diff function. It does not go as far as your routine in terms of >>> options but >>> goes most of the way. It applies the differentiation to order 'n' as >>> an option >>> that is similar to the recursive application. Given this existing >>> routine, the >>> incentive to add another one might be low. >>> >>> Regards, >>> >>> Pierre >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From robert.open at rehammar.se Tue Nov 8 02:10:22 2016 From: robert.open at rehammar.se (Robert Rehammar) Date: Tue, 8 Nov 2016 08:10:22 +0100 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: This is also a good function to have, but it seems complementary to the function I propose since it operate on functions and mine operates on arrays. Best, Robert On 2016-11-07 17:52, Evgeni Burovski wrote: > Note that `approx_derivative` implements several finite-difference schemes, > > https://github.com/scipy/scipy/blob/master/scipy/optimize/_numdiff.py#L179 > > For the moment, it's well hidden in scipy.optimize, but in the long > run the idea is to offer public functionality. So far the main issue > was the lack of bandwidth --- meaning if someone feels like working on > it, great! > > > > > On Mon, Nov 7, 2016 at 7:21 PM, Jonathan Stickel wrote: >> Robert, >> >> I think, conceptually, something like this could be of interest in SciPy, >> but the devil is in the details. There are a lot of ways to implement finite >> differences; the code you shared provides the forward/backward finite >> differences with optional shift in the x positions. These are first-order >> accurate. Your interpolation of forward differences effectively results in >> the second-order accurate central-difference method. >> >> Evaluation of the endpoints is a bit more tricky; your approaches are simple >> but lack rigor (I am not sure of the accuracy). I know of formulas for >> second-order accurate endpoint finite differences for equally spaced x, but >> I am not sure about unequally spaced data (I am sure they exist but I >> haven't looked them up). >> >> So, should this functionality exist in SciPy, what methods should be >> implemented and made available? I think many of us (including myself) have >> implemented our own methods that are satisfactory for our particular needs, >> and no one has submitted general utility finite-differences functions that >> provides multiple methods with appropriate mathematical rigor. Such a >> function should also provide higher-order derivatives (at least second, but >> maybe up to fourth; or perhaps nth-order methods using a series formula). >> >> Also, the name should not be "diff" to avoid confusion with numpy.diff. >> >> Regards, >> Jonathan >> >> >> >> >> On 11/6/16 23:53 , Robert Rehammar wrote: >>> Dear Pierre, >>> >>> Thank you for your reply. I am note fully aware of the difference >>> between numpy and scipy and also not a developer, but it seems to me >>> this function could fit in scipy where you often will want to >>> differentiate w.r.t. a particular variable. I am aware of diff, but it >>> is really a much more basic function. In many areas of science and >>> engineering is the sampling points of crucial importance, and having a >>> function giving you the appropriate sampling points can be very >>> important for accuracy. >>> >>> I have been missing this function several times my self, and hence the >>> reason why I submitted it here. Anyway, if there is no interest, I will >>> of course not peruse the matter further. >>> >>> Best, >>> >>> Robert >>> >>> >>> On 2016-11-05 22:53, Pierre de Buyl wrote: >>>> Dear Robert, >>>> >>>> On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >>>>> Dear all, >>>>> >>>>> I implemented a simple function to differentiate an array which seems >>>>> other >>>>> people might like to have. It looks like: >>>>> def diff(f, x, retval = 'same'): >>>>> (...) >>>>> >>>>> What do you think about adding it to scipy? >>>> I am not a SciPy developer myself but I thought that I would point out >>>> NumPy's >>>> diff function. It does not go as far as your routine in terms of >>>> options but >>>> goes most of the way. It applies the differentiation to order 'n' as >>>> an option >>>> that is similar to the recursive application. Given this existing >>>> routine, the >>>> incentive to add another one might be low. >>>> >>>> Regards, >>>> >>>> Pierre >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From robert.open at rehammar.se Tue Nov 8 02:20:49 2016 From: robert.open at rehammar.se (Robert Rehammar) Date: Tue, 8 Nov 2016 08:20:49 +0100 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: <353202cb-c8c0-fa92-f8f8-58030739f96f@rehammar.se> Jonathan, Please see below. On 2016-11-07 17:21, Jonathan Stickel wrote: > Robert, > > I think, conceptually, something like this could be of interest in > SciPy, but the devil is in the details. There are a lot of ways to > implement finite differences; the code you shared provides the > forward/backward finite differences with optional shift in the x > positions. These are first-order accurate. Your interpolation of > forward differences effectively results in the second-order accurate > central-difference method. Agree > > Evaluation of the endpoints is a bit more tricky; your approaches are > simple but lack rigor (I am not sure of the accuracy). I know of > formulas for second-order accurate endpoint finite differences for > equally spaced x, but I am not sure about unequally spaced data (I am > sure they exist but I haven't looked them up). If there is an interest in the function I can put a bit of more work into it. At it looks now, it is just a very fast hack that I did since I could not find it anywhere. Of course the things you point out should be treated more rigorously. For this function to be useful it is important that it can operate on unequally spaced grids since real-world data often is. > > So, should this functionality exist in SciPy, what methods should be > implemented and made available? I think many of us (including myself) > have implemented our own methods that are satisfactory for our > particular needs, and no one has submitted general utility > finite-differences functions that provides multiple methods with > appropriate mathematical rigor. Such a function should also provide > higher-order derivatives (at least second, but maybe up to fourth; or > perhaps nth-order methods using a series formula). Sure. The simplest way of doing that I guess is to do it recursively. But I can look into this and see what possibilities there are to do it more accurately. Since differentiating is a non-regulating operator, when working with real-world data, it is somethings of interest to also convolute with some windowing function to reduce noise. If the function should provide higher order derivatives, it might also need to be able to do that. > > Also, the name should not be "diff" to avoid confusion with numpy.diff. Yes of course. What do you propose? It could be differentiate, but it is a bit long. Some other possibilities derivative ndiff frac_diff fdiff Best, Robert > > Regards, > Jonathan > > > > On 11/6/16 23:53 , Robert Rehammar wrote: >> Dear Pierre, >> >> Thank you for your reply. I am note fully aware of the difference >> between numpy and scipy and also not a developer, but it seems to me >> this function could fit in scipy where you often will want to >> differentiate w.r.t. a particular variable. I am aware of diff, but it >> is really a much more basic function. In many areas of science and >> engineering is the sampling points of crucial importance, and having a >> function giving you the appropriate sampling points can be very >> important for accuracy. >> >> I have been missing this function several times my self, and hence the >> reason why I submitted it here. Anyway, if there is no interest, I will >> of course not peruse the matter further. >> >> Best, >> >> Robert >> >> >> On 2016-11-05 22:53, Pierre de Buyl wrote: >>> Dear Robert, >>> >>> On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >>>> Dear all, >>>> >>>> I implemented a simple function to differentiate an array which seems >>>> other >>>> people might like to have. It looks like: >>>> def diff(f, x, retval = 'same'): >>>> (...) >>>> >>>> What do you think about adding it to scipy? >>> I am not a SciPy developer myself but I thought that I would point out >>> NumPy's >>> diff function. It does not go as far as your routine in terms of >>> options but >>> goes most of the way. It applies the differentiation to order 'n' as >>> an option >>> that is similar to the recursive application. Given this existing >>> routine, the >>> incentive to add another one might be low. >>> >>> Regards, >>> >>> Pierre >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev From jjstickel at gmail.com Tue Nov 8 13:00:19 2016 From: jjstickel at gmail.com (Jonathan Stickel) Date: Tue, 8 Nov 2016 11:00:19 -0700 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <353202cb-c8c0-fa92-f8f8-58030739f96f@rehammar.se> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> <353202cb-c8c0-fa92-f8f8-58030739f96f@rehammar.se> Message-ID: <78c7fbf4-92d2-1927-e0c1-febf9d31c875@gmail.com> This has started to be a mess of top and bottom posting. I'll clean this up a bit and provide my response below. On 11/8/16 00:20 , Robert Rehammar wrote: > Jonathan, > > Please see below. > > > On 2016-11-07 17:21, Jonathan Stickel wrote: >> Robert, >> >> I think, conceptually, something like this could be of interest in >> SciPy, but the devil is in the details. There are a lot of ways to >> implement finite differences; the code you shared provides the >> forward/backward finite differences with optional shift in the x >> positions. These are first-order accurate. Your interpolation of >> forward differences effectively results in the second-order accurate >> central-difference method. > Agree >> >> Evaluation of the endpoints is a bit more tricky; your approaches are >> simple but lack rigor (I am not sure of the accuracy). I know of >> formulas for second-order accurate endpoint finite differences for >> equally spaced x, but I am not sure about unequally spaced data (I am >> sure they exist but I haven't looked them up). > If there is an interest in the function I can put a bit of more work > into it. At it looks now, it is just a very fast hack that I did since I > could not find it anywhere. Of course the things you point out should be > treated more rigorously. For this function to be useful it is important > that it can operate on unequally spaced grids since real-world data > often is. >> >> So, should this functionality exist in SciPy, what methods should be >> implemented and made available? I think many of us (including myself) >> have implemented our own methods that are satisfactory for our >> particular needs, and no one has submitted general utility >> finite-differences functions that provides multiple methods with >> appropriate mathematical rigor. Such a function should also provide >> higher-order derivatives (at least second, but maybe up to fourth; or >> perhaps nth-order methods using a series formula). > Sure. The simplest way of doing that I guess is to do it recursively. > But I can look into this and see what possibilities there are to do it > more accurately. > > Since differentiating is a non-regulating operator, when working with > real-world data, it is somethings of interest to also convolute with > some windowing function to reduce noise. If the function should provide > higher order derivatives, it might also need to be able to do that. >> >> Also, the name should not be "diff" to avoid confusion with numpy.diff. > Yes of course. What do you propose? It could be differentiate, but it is > a bit long. Some other possibilities > derivative > ndiff > frac_diff > fdiff > > Best, > Robert >> >> Regards, >> Jonathan >> >> >> >>>> >>>> On Thu, Nov 03, 2016 at 08:37:07PM +0100, Robert Rehammar wrote: >>>>> Dear all, >>>>> >>>>> I implemented a simple function to differentiate an array which seems >>>>> other >>>>> people might like to have. It looks like: >>>>> def diff(f, x, retval = 'same'): >>>>> (...) >>>>> >>>>> What do you think about adding it to scipy? Robert Since you seem interested in putting in the work to provide a finite-difference method function to SciPy, I encourage you to work more on it. I would be happy to help you, although you should know that I am not a developer myself. I have use cases for both differentiating experimental data as well as using finite differences in custom partial differential equation solvers, sometimes with unequal grids. I am familiar with the noise problem of finite differences of data. I implemented a Tikhonov-regularization data smoother that can be found here: https://github.com/jjstickel/scikit-datasmooth I provides an option for returning a smooth derivative (instead of the smooth signal), but the endpoints are neglected. Also, SciPy provides smoothing splines in scipy.interpolate. In both cases, a general-use finite differences function would be helpful for obtaining derivatives of the smooth signal with appropriate handling of endpoints. Another question is where to locate such a function. Perhaps in scipy.signal? For a name, I would suggest finitediff or findiff. Regards. Jonathan From pav at iki.fi Tue Nov 8 14:01:37 2016 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 8 Nov 2016 19:01:37 +0000 (UTC) Subject: [SciPy-Dev] Differentiate function References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > Note that `approx_derivative` implements several finite-difference > schemes, In addition, I'd remind of https://pypi.python.org/pypi/Numdifftools From ralf.gommers at gmail.com Thu Nov 10 02:10:22 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 10 Nov 2016 20:10:22 +1300 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen wrote: > Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > > Note that `approx_derivative` implements several finite-difference > > schemes, > > In addition, I'd remind of > > https://pypi.python.org/pypi/Numdifftools And https://github.com/scipy/scipy/wiki/Proposal:-add-finite-difference-numerical-derivatives-as-scipy.diff Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.haslwanter at gmail.com Thu Nov 10 03:19:32 2016 From: thomas.haslwanter at gmail.com (Thomas Haslwanter) Date: Thu, 10 Nov 2016 09:19:32 +0100 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: The current discussion lacks a reference to the existing Savitzky-Golay filter https://scipy.github.io/devdocs/generated/scipy.signal.savgol_filter.html which - to my understanding - should solves most of Robert's problems. thomas On Thu, Nov 10, 2016 at 8:10 AM, Ralf Gommers wrote: > > > On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen wrote: > >> Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: >> > Note that `approx_derivative` implements several finite-difference >> > schemes, >> >> In addition, I'd remind of >> >> https://pypi.python.org/pypi/Numdifftools > > > And https://github.com/scipy/scipy/wiki/Proposal:-add- > finite-difference-numerical-derivatives-as-scipy.diff > > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -- --- Thomas Haslwanter Hubertusgasse 26, A-4060 Leonding http://home.thaslwanter.at -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjstickel at gmail.com Thu Nov 10 12:32:14 2016 From: jjstickel at gmail.com (Jonathan Stickel) Date: Thu, 10 Nov 2016 10:32:14 -0700 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> Message-ID: <10437a08-8c2a-8b84-368c-08f3ba10f9e2@gmail.com> On 11/10/16 01:19 , Thomas Haslwanter wrote: > The current discussion lacks a reference to the existing Savitzky-Golay > filter > https://scipy.github.io/devdocs/generated/scipy.signal.savgol_filter.html > which - to my understanding - should solves most of Robert's problems. > > thomas > No, I don't think this addresses Robert's needs. That is simply a data smoother (and arguably inferior to other data-smoothing methods). Although it does have an option to provide a derivative, it presumes the data are equally spaced. > On Thu, Nov 10, 2016 at 8:10 AM, Ralf Gommers > wrote: > > > > On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen > wrote: > > Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > > Note that `approx_derivative` implements several finite-difference > > schemes, > > In addition, I'd remind of > > https://pypi.python.org/pypi/Numdifftools > > > > And > https://github.com/scipy/scipy/wiki/Proposal:-add-finite-difference-numerical-derivatives-as-scipy.diff > > > Ralf These are tools for finite-differences of a known function. Robert (and I) are interested in finite-differences of y vs. x vectors, whether obtained from experiment or as part of a higher-level numerical method. From tikh at triumf.ca Thu Nov 10 12:36:50 2016 From: tikh at triumf.ca (Evgeniy) Date: Thu, 10 Nov 2016 09:36:50 -0800 Subject: [SciPy-Dev] Jupyter Notebook for the beginner Message-ID: My intention is to implement a web application with Jupyter Notebook, which will start by clicking a link on some page. To start with this I need clarifications from the experts: - is it possible to run the code in Jupyter Notebook immediately after connection. Say, when the user clicks a link, then Jupyter Notebook is opened in new tab and immediately shows the result. - is it possible to put matplotlib drawing (which shows the results) at the top of the notebook? Thus, after immediate execution the users will see the top part of the screen with the results and they should not care about the rest of the screen with Python code. - is it possible to make Python code invisible for users and leave only drawing (after releasing the notebook into production for the users)? - is it possible to create GUI for Jupyter Notebook (buttons, maybe sliders, readback fields)? What can be used for this? Thanks, Evgeniy From gustavo.scalet at eldorado.org.br Thu Nov 10 12:45:38 2016 From: gustavo.scalet at eldorado.org.br (Gustavo Serra Scalet) Date: Thu, 10 Nov 2016 17:45:38 +0000 Subject: [SciPy-Dev] Jupyter Notebook for the beginner In-Reply-To: References: Message-ID: <0f9581f109fb4ba688d0612089b56590@serv030.corp.eldorado.org.br> I guess you should ask to the Jupyter group, right? https://jupyter.org/community.html This mailing list is intended to the Scipy development group > -----Original Message----- > From: SciPy-Dev [mailto:scipy-dev-bounces at scipy.org] On Behalf Of > Evgeniy > Sent: quinta-feira, 10 de novembro de 2016 15:37 > To: scipy-dev at scipy.org > Subject: [SciPy-Dev] Jupyter Notebook for the beginner > > My intention is to implement a web application with Jupyter Notebook, > which will start by clicking a link on some page. > To start with this I need clarifications from the experts: > > - is it possible to run the code in Jupyter Notebook immediately after > connection. Say, when the user clicks a link, then Jupyter Notebook is > opened in new tab and immediately shows the result. > > - is it possible to put matplotlib drawing (which shows the results) at > the top of the notebook? Thus, after immediate execution the users will > see the top part of the screen with the results and they should not care > about the rest of the screen with Python code. > > - is it possible to make Python code invisible for users and leave only > drawing (after releasing the notebook into production for the users)? > > - is it possible to create GUI for Jupyter Notebook (buttons, maybe > sliders, readback fields)? What can be used for this? > > Thanks, > Evgeniy > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From tikh at triumf.ca Thu Nov 10 12:45:03 2016 From: tikh at triumf.ca (Evgeniy) Date: Thu, 10 Nov 2016 09:45:03 -0800 Subject: [SciPy-Dev] Jupyter Notebook for the beginner In-Reply-To: <0f9581f109fb4ba688d0612089b56590@serv030.corp.eldorado.org.br> References: <0f9581f109fb4ba688d0612089b56590@serv030.corp.eldorado.org.br> Message-ID: <68aed03a-7013-48eb-04c9-6df8e56e0a0d@triumf.ca> OK, thanks On 11/10/2016 09:45 AM, Gustavo Serra Scalet wrote: > I guess you should ask to the Jupyter group, right? > https://jupyter.org/community.html > > This mailing list is intended to the Scipy development group > >> -----Original Message----- >> From: SciPy-Dev [mailto:scipy-dev-bounces at scipy.org] On Behalf Of >> Evgeniy >> Sent: quinta-feira, 10 de novembro de 2016 15:37 >> To: scipy-dev at scipy.org >> Subject: [SciPy-Dev] Jupyter Notebook for the beginner >> >> My intention is to implement a web application with Jupyter Notebook, >> which will start by clicking a link on some page. >> To start with this I need clarifications from the experts: >> >> - is it possible to run the code in Jupyter Notebook immediately after >> connection. Say, when the user clicks a link, then Jupyter Notebook is >> opened in new tab and immediately shows the result. >> >> - is it possible to put matplotlib drawing (which shows the results) at >> the top of the notebook? Thus, after immediate execution the users will >> see the top part of the screen with the results and they should not care >> about the rest of the screen with Python code. >> >> - is it possible to make Python code invisible for users and leave only >> drawing (after releasing the notebook into production for the users)? >> >> - is it possible to create GUI for Jupyter Notebook (buttons, maybe >> sliders, readback fields)? What can be used for this? >> >> Thanks, >> Evgeniy >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > From Per.Brodtkorb at ffi.no Fri Nov 11 18:03:51 2016 From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no) Date: Fri, 11 Nov 2016 23:03:51 +0000 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <10437a08-8c2a-8b84-368c-08f3ba10f9e2@gmail.com> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> <10437a08-8c2a-8b84-368c-08f3ba10f9e2@gmail.com> Message-ID: <8114F0AADAECD745AF1FC2047A5DC7ED201D4A26@HBU-POST2.ffi.no> Maybe this addresses Robert's needs: http://www.scholarpedia.org/article/Finite_difference_method#FD_formulas_in_higher-D https://github.com/pbrod/numdifftools/blob/master/numdifftools/fornberg.py Per A. -----Original Message----- From: SciPy-Dev [mailto:scipy-dev-bounces at scipy.org] On Behalf Of Jonathan Stickel Sent: 10. november 2016 18:32 To: scipy-dev at scipy.org Subject: Re: [SciPy-Dev] Differentiate function On 11/10/16 01:19 , Thomas Haslwanter wrote: > The current discussion lacks a reference to the existing > Savitzky-Golay filter > https://scipy.github.io/devdocs/generated/scipy.signal.savgol_filter.h > tml which - to my understanding - should solves most of Robert's > problems. > > thomas > No, I don't think this addresses Robert's needs. That is simply a data smoother (and arguably inferior to other data-smoothing methods). Although it does have an option to provide a derivative, it presumes the data are equally spaced. > On Thu, Nov 10, 2016 at 8:10 AM, Ralf Gommers > wrote: > > > > On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen > wrote: > > Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > > Note that `approx_derivative` implements several finite-difference > > schemes, > > In addition, I'd remind of > > https://pypi.python.org/pypi/Numdifftools > > > > And > https://github.com/scipy/scipy/wiki/Proposal:-add-finite-difference-numerical-derivatives-as-scipy.diff > > umerical-derivatives-as-scipy.diff> > > Ralf These are tools for finite-differences of a known function. Robert (and I) are interested in finite-differences of y vs. x vectors, whether obtained from experiment or as part of a higher-level numerical method. _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev From sievert.scott at gmail.com Fri Nov 11 18:57:38 2016 From: sievert.scott at gmail.com (Scott Sievert) Date: Fri, 11 Nov 2016 15:57:38 -0800 Subject: [SciPy-Dev] Differentiate function In-Reply-To: <8114F0AADAECD745AF1FC2047A5DC7ED201D4A26@HBU-POST2.ffi.no> References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> <10437a08-8c2a-8b84-368c-08f3ba10f9e2@gmail.com> <8114F0AADAECD745AF1FC2047A5DC7ED201D4A26@HBU-POST2.ffi.no> Message-ID: There?s also scipy.misc.derivative which also finds the derivative of a function. There?s not only the method of finite differences, but there?s also automatic differentiation In this, you keep track of what operations are performed on the input, then use the chain rule to find the derivative. Because it only keeps track of the functions performed, this method support if-statments, while-loops and recursion. Here?s a package that implements this method with a thin wrapper around NumPy: https://github.com/HIPS/autograd (though it looks like numdifftools also support this) Numerical Optimization 2nd edition by Stephen Wright also has more detail on chapter 8 methods to compute derivatives (and chapter 9 is on derivative free optimization!). Scott On November 11, 2016 at 5:04:12 PM, per.brodtkorb at ffi.no ( per.brodtkorb at ffi.no) wrote: Maybe this addresses Robert's needs: http://www.scholarpedia.org/article/Finite_difference_method#FD_formulas_in_higher-D https://github.com/pbrod/numdifftools/blob/master/numdifftools/fornberg.py Per A. -----Original Message----- From: SciPy-Dev [mailto:scipy-dev-bounces at scipy.org] On Behalf Of Jonathan Stickel Sent: 10. november 2016 18:32 To: scipy-dev at scipy.org Subject: Re: [SciPy-Dev] Differentiate function On 11/10/16 01:19 , Thomas Haslwanter wrote: > The current discussion lacks a reference to the existing > Savitzky-Golay filter > https://scipy.github.io/devdocs/generated/scipy.signal.savgol_filter.h > tml which - to my understanding - should solves most of Robert's > problems. > > thomas > No, I don't think this addresses Robert's needs. That is simply a data smoother (and arguably inferior to other data-smoothing methods). Although it does have an option to provide a derivative, it presumes the data are equally spaced. > On Thu, Nov 10, 2016 at 8:10 AM, Ralf Gommers > wrote: > > > > On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen > wrote: > > Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > > Note that `approx_derivative` implements several finite-difference > > schemes, > > In addition, I'd remind of > > https://pypi.python.org/pypi/Numdifftools > > > > And > https://github.com/scipy/scipy/wiki/Proposal:-add-finite-difference-numerical-derivatives-as-scipy.diff > > umerical-derivatives-as-scipy.diff> > > Ralf These are tools for finite-differences of a known function. Robert (and I) are interested in finite-differences of y vs. x vectors, whether obtained from experiment or as part of a higher-level numerical method. _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Nov 12 00:42:23 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 12 Nov 2016 00:42:23 -0500 Subject: [SciPy-Dev] Differentiate function In-Reply-To: References: <3cb3bcc3-4961-b09a-18f4-6219218ef968@rehammar.se> <20161105215303.GA2429@pi-x230> <414a0bc9-31b1-e38d-4c33-021638598582@gmail.com> <10437a08-8c2a-8b84-368c-08f3ba10f9e2@gmail.com> <8114F0AADAECD745AF1FC2047A5DC7ED201D4A26@HBU-POST2.ffi.no> Message-ID: On Fri, Nov 11, 2016 at 6:57 PM, Scott Sievert wrote: > There?s also scipy.misc.derivative > > which also finds the derivative of a function. > > There?s not only the method of finite differences, but there?s also automatic > differentiation > In this, you keep track of what operations are performed on the input, then > use the chain rule to find the derivative. Because it only keeps track of > the functions performed, this method support if-statments, while-loops and > recursion. > > Here?s a package that implements this method with a thin wrapper around > NumPy: https://github.com/HIPS/autograd (though it looks like > numdifftools also support this) > > Numerical Optimization 2nd edition by Stephen Wright also has more detail > on chapter 8 methods to compute derivatives (and chapter 9 is on derivative > free optimization!). > > Scott > > On November 11, 2016 at 5:04:12 PM, per.brodtkorb at ffi.no ( > per.brodtkorb at ffi.no) wrote: > > Maybe this addresses Robert's needs: > > http://www.scholarpedia.org/article/Finite_difference_ > method#FD_formulas_in_higher-D > > https://github.com/pbrod/numdifftools/blob/master/numdifftools/fornberg.py > > Per A. > > -----Original Message----- > From: SciPy-Dev [mailto:scipy-dev-bounces at scipy.org] On Behalf Of > Jonathan Stickel > Sent: 10. november 2016 18:32 > To: scipy-dev at scipy.org > Subject: Re: [SciPy-Dev] Differentiate function > > On 11/10/16 01:19 , Thomas Haslwanter wrote: > > The current discussion lacks a reference to the existing > > Savitzky-Golay filter > > https://scipy.github.io/devdocs/generated/scipy.signal.savgol_filter.h > > tml which - to my understanding - should solves most of Robert's > > problems. > > > > thomas > > > > > No, I don't think this addresses Robert's needs. That is simply a data > smoother (and arguably inferior to other data-smoothing methods). > Although it does have an option to provide a derivative, it presumes the > data are equally spaced. > > > > On Thu, Nov 10, 2016 at 8:10 AM, Ralf Gommers > > wrote: > > > > > > > > On Wed, Nov 9, 2016 at 8:01 AM, Pauli Virtanen > > wrote: > > > > Mon, 07 Nov 2016 19:52:09 +0300, Evgeni Burovski kirjoitti: > > > Note that `approx_derivative` implements several finite-difference > > > schemes, > > > > In addition, I'd remind of > > > > https://pypi.python.org/pypi/Numdifftools > > > > > > > > And > > https://github.com/scipy/scipy/wiki/Proposal:-add- > finite-difference-numerical-derivatives-as-scipy.diff > > > > > umerical-derivatives-as-scipy.diff> > > > > Ralf > > > These are tools for finite-differences of a known function. Robert (and > I) are interested in finite-differences of y vs. x vectors, whether > obtained from experiment or as part of a higher-level numerical method. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > To throw one more in here local polynomial regression (local kernel or windows) would be another way. AFAIR, it would be similar to Savitsky-Golay for non-equal spaced grid. IIRC, the slope coefficient of the local linear regression is the derivative of the smooth function and has relatively good edge behavior because of the local linear structure. I think higher order derivatives would follow from the coefficients of higher order polynomial regression. That would be the kernel regression analog of smoothing splines version. Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 17 00:47:39 2016 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Nov 2016 22:47:39 -0700 Subject: [SciPy-Dev] NumPy 1.12.0b1 released. Message-ID: Hi All, I'm pleased to annouce the release of NumPy 1.12.0b1. This release supports Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests submitted by 133 contributors. It is quite sizeable and rather than put the release notes inline I've attached them as a file and they may also be viewed at Github . Zip files and tarballs may also be found the Github link. Wheels and source archives may be downloaded from PyPI, which is the recommended method. This release is a large collection of fixes, enhancements, and improvements and it is difficult to select just a few as highlights. However, the following enhancements may be of particular interest - Order of operations in ``np.einsum`` now can be optimized for large speed improvements. - New ``signature`` argument to ``np.vectorize`` for vectorizing with core dimensions. - The ``keepdims`` argument was added to many functions. - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this is a milestone for PyPy's C-API compatibility layer. Thanks to all, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1.12.0-notes.rst Type: text/prs.fallenstein.rst Size: 64965 bytes Desc: not available URL: From ssouravsingh12 at gmail.com Thu Nov 17 10:33:35 2016 From: ssouravsingh12 at gmail.com (Sourav Singh) Date: Thu, 17 Nov 2016 21:03:35 +0530 Subject: [SciPy-Dev] Working on an issue In-Reply-To: References: Message-ID: Hello Everyone, I am writing this mail to ask about a pending Pull Request in SciPy. I am interested in working on this pull request https://github.com/scipy/scipy/pull/5775 which needs a contributor to finish up. I would appreciate it if someone can guide me on finishing up the pull request. Regards, Sourav -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Nov 18 04:29:14 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 18 Nov 2016 22:29:14 +1300 Subject: [SciPy-Dev] Working on an issue In-Reply-To: References: Message-ID: On Fri, Nov 18, 2016 at 4:33 AM, Sourav Singh wrote: > Hello Everyone, > > I am writing this mail to ask about a pending Pull Request in SciPy. > > I am interested in working on this pull request https://github.com/scipy/ > scipy/pull/5775 which needs a contributor to finish up. > > I would appreciate it if someone can guide me on finishing up the pull > request. > Hi Sourav, thanks for your interest. I've left some comments on that PR for how to continue. Note that it's not an easy PR to get started on, there's quite some fairly low level code there. If you picked that PR because you need the feature, go for it. If you picked it at random and find it difficult, you may want to look at issues labeled "easy-fix". Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Nov 19 00:54:39 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 19 Nov 2016 18:54:39 +1300 Subject: [SciPy-Dev] scipy.sparse.coo_matrix a new method drop_zero_columns? In-Reply-To: References: Message-ID: On Mon, Nov 7, 2016 at 7:10 AM, Evgeny Nekrasov < evgeny.nekrasov at phystech.edu> wrote: > Dear Ralf, > > Thank you for your response. The popular use case for drop_zero_columns is > feature selection before applying machine learning algorithm. For example a > very similar technique VarianceThreshold is implemented in sklearn ( > http://scikit-learn.org/stable/modules/feature_selection.html). The > problem with current implementations is that it fails or takes much more > resources than actually needed if the amount of zero columns is really > huge. Such sparse data representations often produced by popular techniques > such as feature hashing, bag of words, bag of content_ids or similar. Such > techniques are implemented in sklearn (http://scikit-learn.org/ > stable/modules/feature_extraction.html). Nevertheless, custom > implementations often needed, and here drop_zero_columns is valuable. > Other questions: > 1. It would be great to have drop_zero_columns for all matrix types. I > wrote about this method for COO due to it is sufficient to process data > with huge amount of zero columns in efficient way. > 2. I don't know popular use cases for rows. > Thanks Evgeny, clear. Looks fine to me to add drop_zero_columns for all sparse matrix types. Cheers, Ralf > > 2016-11-06 0:44 GMT+03:00 Ralf Gommers : > >> >> >> On Fri, Nov 4, 2016 at 11:40 PM, Evgeny Nekrasov < >> evgeny.nekrasov at phystech.edu> wrote: >> >>> Hello, >>> >>> I am writing to follow up on the discussion >>> https://github.com/scipy/scipy/issues/6754 >>> >>> I would be happy to get your opinions. >>> >> >> Hi Evgeny, it would be helpful to add to the issue description some links >> to code / examples where this method is used or would be useful. Right now >> there's just an assertion "this is useful", which is hard to evaluate. >> >> Other questions I would have: >> 1. The issue has a method for COO, but I'd think we want this for all or >> none of the formats? >> 2. Is it only drop column, or would drop row also make sense? >> >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholdav at gmail.com Sat Nov 19 22:41:15 2016 From: nicholdav at gmail.com (David Nicholson) Date: Sat, 19 Nov 2016 22:41:15 -0500 Subject: [SciPy-Dev] signal.spectrogram doesn't change nperseg to length of window? Message-ID: Hi scipy-dev list, I'm using spectrogram from scipy.signal with a custom window. My understanding from the docs is that when I provide an array_like for the window, the value for nperseg will change from the default of 256 to whatever the length is of the window I provide. But instead I get an error saying 'noverlap must be less than nperseg' and when I check with pdb I can see that nperseg is still the default value. I'm using a value for noverlap that's larger than the default nperseg, but less than the length of the window. I don't get the error if I call the function with 'nperseg=window.shape[0]'. I skimmed the source and don't see anything in the _spect_helper function that would change nperseg to the length of window. Am I misunderstanding the docs, or am I calling the function wrong, or is this a bug? Here's my code: from scipy.io import wavfile from scipy.signal import slepian # i.e. 'DPSS' window used for FFT from scipy.signal import spectrogram FFT_SIZE = 512 FFT_OVERLAP = FFT_SIZE - 32 win_dpss = slepian(FFT_SIZE, 4/FFT_SIZE) #shape = [rate,data] = wavfile.read('a_file_you_have.wav') f,t,spec = spectrogram(data, rate, window=win_dpss, noverlap=FFT_OVERLAP) Thanks for your help --David -- David Nicholson, Ph.D. Candidate Sober Lab , Emory Neuroscience Program. www.nicholdav.info; https://github.com/NickleDave -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Mon Nov 21 16:15:19 2016 From: larson.eric.d at gmail.com (Eric Larson) Date: Mon, 21 Nov 2016 16:15:19 -0500 Subject: [SciPy-Dev] signal.spectrogram doesn't change nperseg to length of window? In-Reply-To: References: Message-ID: It looks like you are reading the docs correctly. From looking at the code, I can confirm that it doesn't change `nperseg` based on what is passed for `window`. So it looks like a bug to me. Do you have time to look into making a Pull Request to fix it? Eric On Sat, Nov 19, 2016 at 10:41 PM, David Nicholson wrote: > Hi scipy-dev list, > > I'm using spectrogram from scipy.signal with a custom window. My > understanding from the docs is that when I provide an array_like for the > window, the value for nperseg will change from the default of 256 to > whatever the length is of the window I provide. > > But instead I get an error saying 'noverlap must be less than nperseg' and > when I check with pdb I can see that nperseg is still the default value. > I'm using a value for noverlap that's larger than the default nperseg, but > less than the length of the window. I don't get the error if I call the > function with 'nperseg=window.shape[0]'. > > I skimmed the source and don't see anything in the _spect_helper function > that would change nperseg to the length of window. > > Am I misunderstanding the docs, or am I calling the function wrong, or is > this a bug? > > Here's my code: > from scipy.io import wavfile > from scipy.signal import slepian # i.e. 'DPSS' window used for FFT > from scipy.signal import spectrogram > > FFT_SIZE = 512 > FFT_OVERLAP = FFT_SIZE - 32 > win_dpss = slepian(FFT_SIZE, 4/FFT_SIZE) #shape = > > [rate,data] = wavfile.read('a_file_you_have.wav') > f,t,spec = spectrogram(data, rate, window=win_dpss, noverlap=FFT_OVERLAP) > > Thanks for your help > --David > > -- > David Nicholson, Ph.D. Candidate > Sober Lab , Emory > Neuroscience Program. > www.nicholdav.info; > https://github.com/NickleDave > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholdav at gmail.com Tue Nov 22 10:42:38 2016 From: nicholdav at gmail.com (David Nicholson) Date: Tue, 22 Nov 2016 10:42:38 -0500 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 157, Issue 15 In-Reply-To: References: Message-ID: Hi Eric, I think I can do that over the holiday. --David On Tue, Nov 22, 2016 at 7:00 AM, wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at scipy.org > > You can reach the person managing the list at > scipy-dev-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: signal.spectrogram doesn't change nperseg to length of > window? (Eric Larson) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 21 Nov 2016 16:15:19 -0500 > From: Eric Larson > To: SciPy Developers List > Subject: Re: [SciPy-Dev] signal.spectrogram doesn't change nperseg to > length of window? > Message-ID: > HQ at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > It looks like you are reading the docs correctly. From looking at the code, > I can confirm that it doesn't change `nperseg` based on what is passed for > `window`. So it looks like a bug to me. > > Do you have time to look into making a Pull Request to fix it? > > Eric > > > On Sat, Nov 19, 2016 at 10:41 PM, David Nicholson > wrote: > > > Hi scipy-dev list, > > > > I'm using spectrogram from scipy.signal with a custom window. My > > understanding from the docs is that when I provide an array_like for the > > window, the value for nperseg will change from the default of 256 to > > whatever the length is of the window I provide. > > > > But instead I get an error saying 'noverlap must be less than nperseg' > and > > when I check with pdb I can see that nperseg is still the default value. > > I'm using a value for noverlap that's larger than the default nperseg, > but > > less than the length of the window. I don't get the error if I call the > > function with 'nperseg=window.shape[0]'. > > > > I skimmed the source and don't see anything in the _spect_helper > function > > that would change nperseg to the length of window. > > > > Am I misunderstanding the docs, or am I calling the function wrong, or is > > this a bug? > > > > Here's my code: > > from scipy.io import wavfile > > from scipy.signal import slepian # i.e. 'DPSS' window used for FFT > > from scipy.signal import spectrogram > > > > FFT_SIZE = 512 > > FFT_OVERLAP = FFT_SIZE - 32 > > win_dpss = slepian(FFT_SIZE, 4/FFT_SIZE) #shape = > > > > [rate,data] = wavfile.read('a_file_you_have.wav') > > f,t,spec = spectrogram(data, rate, window=win_dpss, noverlap=FFT_OVERLAP) > > > > Thanks for your help > > --David > > > > -- > > David Nicholson, Ph.D. Candidate > > Sober Lab , Emory > > Neuroscience Program. > > www.nicholdav.info; > > https://github.com/NickleDave > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: attachments/20161121/d77e09a5/attachment-0001.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 157, Issue 15 > ****************************************** > -- David Nicholson, Ph.D. Candidate Sober Lab , Emory Neuroscience Program. www.nicholdav.info; https://github.com/NickleDave -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Nov 26 17:41:45 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 27 Nov 2016 11:41:45 +1300 Subject: [SciPy-Dev] signal.spectrogram doesn't change nperseg to length of window? In-Reply-To: References: Message-ID: On Tue, Nov 22, 2016 at 10:15 AM, Eric Larson wrote: > It looks like you are reading the docs correctly. From looking at the > code, I can confirm that it doesn't change `nperseg` based on what is > passed for `window`. So it looks like a bug to me. > > Do you have time to look into making a Pull Request to fix it? > Opened an issue for this to not lose track of it: https://github.com/scipy/scipy/issues/6817 Ralf > Eric > > > On Sat, Nov 19, 2016 at 10:41 PM, David Nicholson > wrote: > >> Hi scipy-dev list, >> >> I'm using spectrogram from scipy.signal with a custom window. My >> understanding from the docs is that when I provide an array_like for the >> window, the value for nperseg will change from the default of 256 to >> whatever the length is of the window I provide. >> >> But instead I get an error saying 'noverlap must be less than nperseg' and >> when I check with pdb I can see that nperseg is still the default value. >> I'm using a value for noverlap that's larger than the default nperseg, but >> less than the length of the window. I don't get the error if I call the >> function with 'nperseg=window.shape[0]'. >> >> I skimmed the source and don't see anything in the _spect_helper >> function that would change nperseg to the length of window. >> >> Am I misunderstanding the docs, or am I calling the function wrong, or is >> this a bug? >> >> Here's my code: >> from scipy.io import wavfile >> from scipy.signal import slepian # i.e. 'DPSS' window used for FFT >> from scipy.signal import spectrogram >> >> FFT_SIZE = 512 >> FFT_OVERLAP = FFT_SIZE - 32 >> win_dpss = slepian(FFT_SIZE, 4/FFT_SIZE) #shape = >> >> [rate,data] = wavfile.read('a_file_you_have.wav') >> f,t,spec = spectrogram(data, rate, window=win_dpss, noverlap=FFT_OVERLAP) >> >> Thanks for your help >> --David >> >> -- >> David Nicholson, Ph.D. Candidate >> Sober Lab , Emory >> Neuroscience Program. >> www.nicholdav.info; >> https://github.com/NickleDave >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: