[SciPy-Dev] Adding algorithm for automatic differentiation of noisy data to SciPy

[Siva Sankar]

The paper can be found here. https://arxiv.org/abs/1610.04397

Ok, I will wait before I do anything more.

BR, Siva.

On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

> On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977 at gmail.com>
> wrote:
>
>> Hello,
>>
>> Thanks for the swift response. Yes, I have based this on a paper that the
>> professor I am working for was working on. The paper, however, is not yet
>> published in a Journal yet but is available in arXiv.
>>
>
> Can you share a link to the paper?
>
>
>> The algorithm was tested with real-world data and synthetic data,
>> comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and
>> Savitzky Golay with manual parameter tuning.
>>
>
> Your PR adds code to scipy.misc, which is not the right place (we'll get
> rid of misc soon). It sounds more like it belongs in scipy.signal. That
> said, don't do a lot of work to move things now - first we need to decide
> whether this makes sense to include in SciPy. At the moment I'd say it's
> too early; we probably want to wait a year or two until it's clear that the
> paper has been accepted and gets citations that show the algorithm is
> valuable.
>
> Cheers,
> Ralf
>
>
>> The real world test data included using an XSENS IMU to get inertial data
>> readings and then comparing them against those from a VectorNAV imu which
>> has significantly accurate readings. The XSENS IMU that was used not being
>> very accurate simulated the noisy measurements, while the VectorNAV unit
>> measurements were considered as the reference measurements. The algorithm
>> was then used to estimate the derivates and then results were compared with
>> the reference values.
>>
>> Best Regards,
>> Siva Sankar Kannan.
>>
>> On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>> On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977 at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Automatic differentiation of noisy data has not been very accurate and
>>>> does not have a lot of different algorithms that give consistent and
>>>> accurate results.
>>>>
>>>> I have added an algorithm that estimates the derivative of noisy data
>>>> using linear Gaussian state-space smoothing and square root formulas.
>>>>
>>>> Link to the pull request. https://github.com/scipy/scipy/pull/9004
>>>>
>>>> The algorithm in its current state provides the smoothed signal from
>>>> the noisy measurements, estimation of the first and second order
>>>> derivatives and optionally also provides the dense output for the same. It
>>>> accepts non-equally spaced data abscissas and is able to compute the state
>>>> parameters between the abscissas, hence being able to provide the dense
>>>> output.
>>>>
>>>> The algorithm was tested with data from bioanalytics and provides equal
>>>> or better accuracy of the derivatives compared to the other automatic
>>>> derivative algorithms.
>>>>
>>>> Real life readings and measurements of data in any field are prone to
>>>> noise, thereby making the normal differentiation algorithms less reliable.
>>>> Having an algorithm like this can significantly help users from a variety
>>>> of fields where they need a good differentiation estimation without having
>>>> to tweak with the parameters to differentiate data.
>>>>
>>>
>>> Hi Siva, thank you for trying to improve the situation with
>>> differentiation functionality. In SciPy we aim to include algorithms that
>>> are well known and have good performance - that means ideally a paper with
>>> enough citations showing real-world value. Have you based this on a
>>> publication?
>>>
>>> Cheers,
>>> Ralf
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180707/c519276e/attachment-0001.html>