[AstroPy] Using quantities is slow

Tue Aug 25 01:25:05 EDT 2015

I've encountered performance problems with units before, and one issue
is that creating new composite units (e.g., u.km/u.s) comes with some
overhead.  If such operations are a significant overhead, you can set
them up in advance, e.g.:

_kms = u.km/u.s

and use that elsewhere in the code.  You can also get a small boost
from using u.Quantity(x, unit) instead of x*unit.

In [1]: %timeit 5*u.km/u.s
The slowest run took 4.81 times longer than the fastest. This could
mean that an intermediate result is being cached
10000 loops, best of 3: 40.6 µs per loop

In [2]: %timeit u.Quantity(5, u.km/u.s)
The slowest run took 4.38 times longer than the fastest. This could
mean that an intermediate result is being cached
10000 loops, best of 3: 30.1 µs per loop

In [3]: _kms = u.km/u.s

In [4]: %timeit u.Quantity(5, _kms)
The slowest run took 6.71 times longer than the fastest. This could
mean that an intermediate result is being cached
100000 loops, best of 3: 7.89 µs per loop

(thanks to Tom Robitaille who pointed all this out to me)

On Tue, Aug 25, 2015 at 2:58 AM, Paul Kuin <npkuin at gmail.com> wrote:
> I'm converting my code to use astropy.units, and decided to use it mainly at
> the interface level, and use my own set of fixed units inside the software
> units.
>
> That said, it seems to me that it would be useful if you could set some flag
> to tell the code that the units are not going to change until you unset the
> flag. Then the "y = xx.value" steps are not needed. Or perhaps this is a
> really stupid idea because I don't know the ins and outs. .. ;-)
>
> Paul
>
> On Mon, Aug 24, 2015 at 11:41 PM, Erik Bray <embray at stsci.edu> wrote:
>>
>> On 08/24/2015 06:32 PM, Derek Homeier wrote:
>> > On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com>
>> > wrote:
>> >>
>> >> I am writing an MCMC fitting code, and my current implementation uses
>> >> the (very useful) astropy quantities framework to take care of unit
>> >> conversions and such. However, the code is surprisingly slow so I profiled
>> >> it with %prun and it looks like the quantity utilities that wrap numpy
>> >> functions are taking a huge chunk of time. Is that just the price of
>> >> convenience?
>> >>
>> > I am not an expert on the quantities/units implementation, but I would
>> > imagine that they could indeed
>> > introduce a significant overhead, as the class adds a number of
>> > validations and checks for a function
>> > calling a quantity object. Especially with rather numerous operations on
>> > comparatively small arrays
>> > this could become problematic.
>> > Did you experiment with using decorators as described in
>> >
>> > http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
>> >
>> > - iiuc this would allow bypassing some of the validations?
>>
>> I don't think this is really relevant to the OP's question (though always
>> good
>> to know about).  This decorator just makes it easy to bypass writing a lot
>> of
>> boilerplate code for checking the units of its inputs--it doesn't remove
>> any
>> overhead (in fact it adds it--but this was used for cases where that
>> overhead
>> already existed--the code was just highly repetitive).
>>
>> > Another option might be to only pass plain numpy arrays to the innermost
>> > loops of your code with .value,
>> > of course at the expense of some of the convenience gained in the first
>> > place...
>>
>> Right, that's the big problem.  The thing Quantity supports (which most
>> similar
>> libraries in Python don't) is that it *is* a Numpy ndarray, and with few
>> exceptions [1] works transparently with all of Numpy's universal
>> functions.
>> However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so
>> there's some overhead associated with redoing unit conversions on every
>> function
>> call (and even if Numpy did make this easier, as it will in 1.10,
>> performing
>> unit conversions repeatedly in a loop will add significant overhead to any
>> fitting operation, so better to do the unit conversions first so that all
>> quantities are expressed in the same base units and orders of magnitude,
>> then as
>> Derek suggested pass in the raw arrays (from .value) to high performance
>> code.
>>
>> Having a mechanism to automate such conversions will be useful I think.
>>
>> Erik
>>
>>
>> [1]
>> http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
>
> --
>
> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
> phone +44-(0)1483 (prefix) -204927 (work)
> mobile +44(0)7806985366  skype ID: npkuin
> Mullard Space Science Laboratory  – University College London  –
> Holmbury St Mary – Dorking – Surrey RH5 6NT–  U.K.
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>

-- 
Adam Ginsburg
Fellow, European Southern Observatory
http://www.adamgginsburg.com/