[AstroPy] Missing some statistical method for your astronomy analyses in Python?

[Christoph Deil]

Hi Jerome,

thanks for your suggestion.

As Tom said, a fast median should be implemented in numpy.
Actually this is an existing feature request from 2009.
I've added this section for things that should go into numpy / scipy on the wiki page:
Whishlist for numpy / scipy
Here we collect features that really belong in numpy / scipy. Even if someone implements them it will take a year or two until it is released and the updated by the majority of users, so we might decide to implement these features in astropy for now.

numpy issue #1811: "median in average O(n) time" from 2009-09-01. There is a fast median in bottleneck, see speed comparison of numpy/scipy/bottleneck/pandas here. I don't know if there is a plan to merge parts of bottleneck into numpy.
numpy issue #2448, "Numerical-stable sum (similar to math.fsum)" from 2011-06-02

Christoph


On Jan 15, 2013, at 11:29 AM, Thomas Robitaille <thomas.robitaille at gmail.com> wrote:

> I think this is something that should be brought up with the Numpy
> developers, and it would probably benefit everyone to have a faster
> median calculation function (not just astronomers). At the moment,
> Numpy just uses a quicksort, but if one could achieve speedups of 5x,
> then I think the Numpy devs might be interested.
> 
> Cheers,
> Tom
> 
> On 15 January 2013 10:51, Jerome Caron <jerome_caron_astro at ymail.com> wrote:
>> Dear Christoph
>> I think the calculation of median values in Numpy is not optimal. I don't
>> know if there are other libraries that do better?
>> On my machine I get these results:
>>>>> data = numpy.random.rand(5000,5000)
>>>>> t0=time.time();print numpy.ma.median(data);print time.time()-t0
>> 0.499845739822
>> 15.1949999332
>>>>> t0=time.time();print numpy.median(data);print time.time()-t0
>> 0.499845739822
>> 4.32100009918
>>>>> t0=time.time();print aspylib.astro.get_median(data);print time.time()-t0
>> [ 0.49984574]
>> 0.90499997139
>>>>> 
>> The median calculation in Aspylib is using C code from Nicolas Devillard
>> (can be found here: http://ndevilla.free.fr/median/index.html) interfaced
>> with ctypes.
>> It could be easily re-used for other, more official packages. I think the
>> code also finds quantiles efficiently.
>> Kind regards
>> Jerome Caron
>> 
>> De : Christoph Deil <deil.christoph at googlemail.com>
>> À : "astropy at scipy.org list" <astropy at scipy.org>
>> Envoyé le : Mardi 15 janvier 2013 8h42
>> Objet : [AstroPy] Missing some statistical method for your astronomy
>> analyses in Python?
>> 
>> Dear all,
>> 
>> we would like to know which statistical methods you want to use for you
>> astronomy work with Python that are not readily available e.g. in scipy or
>> one of the existing scientific Python packages like e.g. statsmodels,
>> scikit-learn, astroML, …
>> 
>> Please visit the wiki page for what has already been suggested, then reply
>> to this email or make an addition to the
>> https://github.com/astropy/astropy/wiki/What-methods-do-we-want-in-astropy.stats?
>> wiki page directly.
>> 
>> The plan is to add these methods either to astropy.stats or, if they are not
>> astronomy-specific, to one of the existing scientific Python packages.
>> 
>> Christoph
>> 
>> PS: This discussion started as a thread on astropy-dev (
>> https://groups.google.com/d/topic/astropy-dev/Zwgafam171E/discussion ), but
>> it was suggested to also ask here.
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20130115/b9d25232/attachment.html>