[SciPy-Dev] Adding algorithm for automatic differentiation of noisy data to SciPy

[Siva Sankar]

Ok, thanks for the response. I will release it as a standalone algorithm
somewhere.

BR,
Siva.

On Sat, Jul 7, 2018 at 7:56 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

> On Fri, Jul 6, 2018 at 2:17 PM, Siva Sankar <sivsankar977 at gmail.com>
> wrote:
>
>> The paper can be found here. https://arxiv.org/abs/1610.04397
>>
>
> Thanks Siva. Given that that paper is quite technical and terse, that it's
> 2 years old with only one self-citation, and that it doesn't fit nicely in
> a scipy submodule, my opinion is that we unfortunately cannot accept your
> PR. Please don't let that discourage you though - I would suggest to
> release your algorithm standalone - if it matures and gets update we can
> reconsider at some point.
>
> Cheers,
> Ralf
>
>
>>
>> Ok, I will wait before I do anything more.
>>
>> BR, Siva.
>>
>> On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>> On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977 at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Thanks for the swift response. Yes, I have based this on a paper that
>>>> the professor I am working for was working on. The paper, however, is not
>>>> yet published in a Journal yet but is available in arXiv.
>>>>
>>>
>>> Can you share a link to the paper?
>>>
>>>
>>>> The algorithm was tested with real-world data and synthetic data,
>>>> comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and
>>>> Savitzky Golay with manual parameter tuning.
>>>>
>>>
>>> Your PR adds code to scipy.misc, which is not the right place (we'll get
>>> rid of misc soon). It sounds more like it belongs in scipy.signal. That
>>> said, don't do a lot of work to move things now - first we need to decide
>>> whether this makes sense to include in SciPy. At the moment I'd say it's
>>> too early; we probably want to wait a year or two until it's clear that the
>>> paper has been accepted and gets citations that show the algorithm is
>>> valuable.
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>>> The real world test data included using an XSENS IMU to get inertial
>>>> data readings and then comparing them against those from a VectorNAV imu
>>>> which has significantly accurate readings. The XSENS IMU that was used not
>>>> being very accurate simulated the noisy measurements, while the VectorNAV
>>>> unit measurements were considered as the reference measurements. The
>>>> algorithm was then used to estimate the derivates and then results were
>>>> compared with the reference values.
>>>>
>>>> Best Regards,
>>>> Siva Sankar Kannan.
>>>>
>>>> On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Automatic differentiation of noisy data has not been very accurate
>>>>>> and does not have a lot of different algorithms that give consistent and
>>>>>> accurate results.
>>>>>>
>>>>>> I have added an algorithm that estimates the derivative of noisy data
>>>>>> using linear Gaussian state-space smoothing and square root formulas.
>>>>>>
>>>>>> Link to the pull request. https://github.com/scipy/scipy/pull/9004
>>>>>>
>>>>>> The algorithm in its current state provides the smoothed signal from
>>>>>> the noisy measurements, estimation of the first and second order
>>>>>> derivatives and optionally also provides the dense output for the same. It
>>>>>> accepts non-equally spaced data abscissas and is able to compute the state
>>>>>> parameters between the abscissas, hence being able to provide the dense
>>>>>> output.
>>>>>>
>>>>>> The algorithm was tested with data from bioanalytics and provides
>>>>>> equal or better accuracy of the derivatives compared to the other automatic
>>>>>> derivative algorithms.
>>>>>>
>>>>>> Real life readings and measurements of data in any field are prone to
>>>>>> noise, thereby making the normal differentiation algorithms less reliable.
>>>>>> Having an algorithm like this can significantly help users from a variety
>>>>>> of fields where they need a good differentiation estimation without having
>>>>>> to tweak with the parameters to differentiate data.
>>>>>>
>>>>>
>>>>> Hi Siva, thank you for trying to improve the situation with
>>>>> differentiation functionality. In SciPy we aim to include algorithms that
>>>>> are well known and have good performance - that means ideally a paper with
>>>>> enough citations showing real-world value. Have you based this on a
>>>>> publication?
>>>>>
>>>>> Cheers,
>>>>> Ralf
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180708/bd8d63ba/attachment-0001.html>