[AstroPy] astropy and photutils slow

[David Nidever]

Hi Stephen,

This is what I had:

In [7]: np.__version__
Out[7]: '1.8.0rc1'

In [10]: astropy.__version__
Out[10]: u'1.2.1'

and I'm using python 2.7.

I upgraded to the newest version and I'm finding a similar
performance to what you showed.

Thanks!
David


On Tue, Sep 13, 2016 at 1:17 PM, Stephen Bailey <stephenbailey at lbl.gov>
wrote:

> What version of numpy and astropy do you have?  (numpy.__version__ and
> astropy.__version__).
>
> As another data point, using numpy/1.11.1 and astropy/1.2.1 I get much
> faster results than you:
>
> On Tue, Sep 13, 2016 at 12:05 PM, David Nidever <davidnidever at gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm pretty new to astropy and to python in general.  I've been trying out
>> some of the astropy and photutils photometry tools, but I'm finding that
>> they
>> are running pretty slow.  I wanted to double-check that I'm not doing
>> something
>> wrong.
>>
>> Below is an example of running the astropy sigma clipped statistics
>> routine
>> on a fake 2Kx4K image (since I'm working on DECam data).  This takes
>> almost 20s to run even though similar statistics using numpy take ~0.1s
>> or so.
>>
>> >>> import numpy as np
>> >>> from astropy.stats import sigma_clipped_stats
>> >>> im = np.random.rand(4096,2048)*10.0+100.0
>> >>> timeit mean, median, std = sigma_clipped_stats(im, sigma=3.0, iters=5)
>> 1 loop, best of 3: 18.9 s per loop
>>
>
> 1.84s per loop for me.  Still slow, but not 18.9s.
>
>
>> It seems like it might be the use of the numpy masked array module that is
>> slowing things down.  Just using the masked array median versus the
>> normal numpy median takes ~30x longer.
>>
>> >>> timeit np.median(im)
>> 10 loops, best of 3: 109 ms per loop
>>
>
> 103 ms for me; i.e. this isn't just that I have a super fast machine and
> you have a super slow one.
>
>
>> >>> timeit np.ma.median(im)
>> 1 loop, best of 3: 2.71 s per loop
>>
>
> 104 ms for me.  i.e. with numpy 1.11.1 I don't see a radical difference
> between masked array or not.
>
> Stephen
>
>
>>
>> I'm finding similar speed issues with the photutils routines (e.g.
>> background2D)
>> that I believe also use masked arrays.
>>
>> Is there something I can do to speed things up?
>>
>> Thanks!
>> David
>>
>> --
>> Dr. David Lee Nidever
>> Survey Data Scientist, NOAO
>> 950 N. Cherry Ave.
>> Tucson, AZ 85719
>> (520) 318-8368
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> https://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy
>
>


-- 
Dr. David Lee Nidever
Survey Data Scientist, NOAO
950 N. Cherry Ave.
Tucson, AZ 85719
(520) 318-8368
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20160913/df00bb35/attachment.html>