[SciPy-Dev] Adding algorithm for automatic differentiation of noisy data to SciPy

[Ralf Gommers]

On Fri, Jul 6, 2018 at 2:17 PM, Siva Sankar <sivsankar977 at gmail.com> wrote:

> The paper can be found here. https://arxiv.org/abs/1610.04397
>

Thanks Siva. Given that that paper is quite technical and terse, that it's
2 years old with only one self-citation, and that it doesn't fit nicely in
a scipy submodule, my opinion is that we unfortunately cannot accept your
PR. Please don't let that discourage you though - I would suggest to
release your algorithm standalone - if it matures and gets update we can
reconsider at some point.

Cheers,
Ralf


>
> Ok, I will wait before I do anything more.
>
> BR, Siva.
>
> On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>> On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977 at gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks for the swift response. Yes, I have based this on a paper that
>>> the professor I am working for was working on. The paper, however, is not
>>> yet published in a Journal yet but is available in arXiv.
>>>
>>
>> Can you share a link to the paper?
>>
>>
>>> The algorithm was tested with real-world data and synthetic data,
>>> comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and
>>> Savitzky Golay with manual parameter tuning.
>>>
>>
>> Your PR adds code to scipy.misc, which is not the right place (we'll get
>> rid of misc soon). It sounds more like it belongs in scipy.signal. That
>> said, don't do a lot of work to move things now - first we need to decide
>> whether this makes sense to include in SciPy. At the moment I'd say it's
>> too early; we probably want to wait a year or two until it's clear that the
>> paper has been accepted and gets citations that show the algorithm is
>> valuable.
>>
>> Cheers,
>> Ralf
>>
>>
>>> The real world test data included using an XSENS IMU to get inertial
>>> data readings and then comparing them against those from a VectorNAV imu
>>> which has significantly accurate readings. The XSENS IMU that was used not
>>> being very accurate simulated the noisy measurements, while the VectorNAV
>>> unit measurements were considered as the reference measurements. The
>>> algorithm was then used to estimate the derivates and then results were
>>> compared with the reference values.
>>>
>>> Best Regards,
>>> Siva Sankar Kannan.
>>>
>>> On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Automatic differentiation of noisy data has not been very accurate and
>>>>> does not have a lot of different algorithms that give consistent and
>>>>> accurate results.
>>>>>
>>>>> I have added an algorithm that estimates the derivative of noisy data
>>>>> using linear Gaussian state-space smoothing and square root formulas.
>>>>>
>>>>> Link to the pull request. https://github.com/scipy/scipy/pull/9004
>>>>>
>>>>> The algorithm in its current state provides the smoothed signal from
>>>>> the noisy measurements, estimation of the first and second order
>>>>> derivatives and optionally also provides the dense output for the same. It
>>>>> accepts non-equally spaced data abscissas and is able to compute the state
>>>>> parameters between the abscissas, hence being able to provide the dense
>>>>> output.
>>>>>
>>>>> The algorithm was tested with data from bioanalytics and provides
>>>>> equal or better accuracy of the derivatives compared to the other automatic
>>>>> derivative algorithms.
>>>>>
>>>>> Real life readings and measurements of data in any field are prone to
>>>>> noise, thereby making the normal differentiation algorithms less reliable.
>>>>> Having an algorithm like this can significantly help users from a variety
>>>>> of fields where they need a good differentiation estimation without having
>>>>> to tweak with the parameters to differentiate data.
>>>>>
>>>>
>>>> Hi Siva, thank you for trying to improve the situation with
>>>> differentiation functionality. In SciPy we aim to include algorithms that
>>>> are well known and have good performance - that means ideally a paper with
>>>> enough citations showing real-world value. Have you based this on a
>>>> publication?
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180707/d18fc20c/attachment.html>