[SciPy-Dev] Add feature to approx_fprime (Nicolas CELLIER)
Jeffrey Manville
jeffmanvillejr at gmail.com
Sun Sep 24 14:34:42 EDT 2017
Hello Nick,
So, I am implementing Andrew Ng's Machine learning coursework in Python
which was originally in Matlab/Octave. I was using the approx_fprime to
evaluate by implementation of a back propagation algorithm.
To be honest, I didn't know about automatic differentiation. I suspect that
it can be prohibitively expensive for many machine learning algorithms, but
probably not for my project. I could use a smaller epsilon, but then I am
just getting closer and closer to the floating point error problem.
*My project:*
In my project, the "difference" between backprop and the numerical gradient
was expected to be <1e-9 with an epsilon of 1e-4. I was getting about
2.48e-5 with the current approx_fprime vs 1.65e-11 with the two sided
method and the same epsilon. The two sided was significantly more accurate.
*The problem:*
The problem with the one sided method is that it is inherently biased. The
change isn't centered around the point in question for the one sided
method.
The two methods are essentially the same method with the difference being
"where" they guess that calculated derivative is. One sided guesses that
the calculated derivative is at one side of the two points. The two sided
guesses the derivative is in the middle.
*Proof:*
I attached an excel doc that has a kind of example of the difference.
I have the project on github. Here is the branch
<https://github.com/manvillej/Python-AndrewNgML/tree/Ex4NNtraining> that I
am currently working on. If you pull it and run ex4checker.py you will see
the two sided. If you change line 3: import checker as op to import checker
as op you will see the relative difference change from the two sided to
the one sided.
Cheers,
Jeff
On Sun, Sep 24, 2017 at 12:00 PM, <scipy-dev-request at python.org> wrote:
> Send SciPy-Dev mailing list submissions to
> scipy-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman/listinfo/scipy-dev
> or, via email, send a message with subject or body 'help' to
> scipy-dev-request at python.org
>
> You can reach the person managing the list at
> scipy-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-Dev digest..."
>
>
> Today's Topics:
>
> 1. Add feature to approx_fprime (Jeffrey Manville)
> 2. Re: Add feature to approx_fprime (Nicolas CELLIER)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 23 Sep 2017 23:05:57 -0400
> From: Jeffrey Manville <jeffmanvillejr at gmail.com>
> To: scipy-dev at python.org
> Subject: [SciPy-Dev] Add feature to approx_fprime
> Message-ID:
> <CAFUATRLXKY6y=5VEANjrd2ATtZ8GHKxWEeLmZ=Tk6RkfxABEvw at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
> I have been using the approx_fprime and I noticed that the source code as
> it is only uses the one sided numerical differentiation. I had to build my
> own two sided one by copying and modifying the one sided code.
>
> One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton
> method
> Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka
> Symmetric method
>
> The two sided method tends to be more accurate, but can be more
> computationally expensive.
>
> Is this something that would be good to add? I am newer to python and I
> haven't contributed to open source yet, but it seems like a good fit.
>
> I think it would be good to pass in a kwarg called method and set the
> default to one sided, so no one's code breaks, but they can choose which
> method to use.
>
> Should I keep the decision logic at approx_fprime and have two separate
> functions for the two methods?
>
> Here's a wikipedia article on it
> <https://en.wikipedia.org/wiki/Numerical_differentiation>
> Here's a link to the source code
> <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/
> optimize.py#L633-L688>
>
> Cheers,
>
> Jeff
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/attachments/
> 20170923/87340d02/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 24 Sep 2017 09:46:02 +0200
> From: Nicolas CELLIER <eryole at gmail.com>
> To: SciPy Developers List <scipy-dev at python.org>
> Subject: Re: [SciPy-Dev] Add feature to approx_fprime
> Message-ID: <8dd96501-8952-472e-81d8-a39ed609359d at gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hm, tell me if I'm wrong, but in that case you can choose an arbitrary
> small epsilon. So accuracy should not be a problem (except in case of
> rounding error).
>
> By curiosity, what is your application and why do you think that one sided
> isn't enough?
>
> And if accuracy loss is really a problem, have you checked automatic
> differenciation?
>
> Nicolas C.
> Le 24 sept. 2017, ? 05:09, Jeffrey Manville <jeffmanvillejr at gmail.com> a
> ?crit:
>
> Hello,
>
> I have been using the approx_fprime and I noticed that the source code as
> it is only uses the one sided numerical differentiation. I had to build my
> own two sided one by copying and modifying the one sided code.
>
> One sided: F'(x) = ( F(x+h) - F(x) ) / h ? ? ? ? ? ? ? ? ? ? ? ? aka
> Newton methodTwo sided:??F'(x) = ( F(x+h) - F(x-h) ) / (2*h) ? ? ? ? ? ? ?
> ?aka Symmetric method
> The two sided method tends to be more accurate, but can be more
> computationally expensive. ?
> Is this something that would be good to add? I am newer to python and I
> haven't contributed to open source yet, but it seems like a good fit.
>
> I think it would be good to pass in a kwarg called method and set the
> default to one sided, so no one's code breaks, but they can choose which
> method to use.?
>
> Should I keep the decision logic at approx_fprime and have two separate
> functions for the two methods?
>
> Here's a wikipedia article on it
> Here's a link to the?source code
>
> Cheers,
> Jeff
>
>
>
>
>
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
> Le 24 sept. 2017 05:09, ? 05:09, Jeffrey Manville <
> jeffmanvillejr at gmail.com> a ?crit:
> >Hello,
> >
> >I have been using the approx_fprime and I noticed that the source code
> >as
> >it is only uses the one sided numerical differentiation. I had to build
> >my
> >own two sided one by copying and modifying the one sided code.
> >
> >One sided: F'(x) = ( F(x+h) - F(x) ) / h aka
> >Newton
> >method
> >Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka
> >Symmetric method
> >
> >The two sided method tends to be more accurate, but can be more
> >computationally expensive.
> >
> >Is this something that would be good to add? I am newer to python and I
> >haven't contributed to open source yet, but it seems like a good fit.
> >
> >I think it would be good to pass in a kwarg called method and set the
> >default to one sided, so no one's code breaks, but they can choose
> >which
> >method to use.
> >
> >Should I keep the decision logic at approx_fprime and have two separate
> >functions for the two methods?
> >
> >Here's a wikipedia article on it
> ><https://en.wikipedia.org/wiki/Numerical_differentiation>
> >Here's a link to the source code
> ><https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize
> /optimize.py#L633-L688>
> >
> >Cheers,
> >
> >Jeff
> >
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >SciPy-Dev mailing list
> >SciPy-Dev at python.org
> >https://mail.python.org/mailman/listinfo/scipy-dev
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/attachments/
> 20170924/60433893/attachment-0001.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
> ------------------------------
>
> End of SciPy-Dev Digest, Vol 167, Issue 26
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170924/1173a37b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Example of One-sided vs Two-sided.xlsx
Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Size: 20304 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170924/1173a37b/attachment-0001.xlsx>
More information about the SciPy-Dev
mailing list