[SciPy-Dev] scipy.integrate optimisation for pandas.TimeSeries

Mon May 6 03:47:32 EDT 2013

On Mon, May 6, 2013 at 12:25 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

>
>
>
> On Sun, May 5, 2013 at 11:24 PM, Dražen Lučanin <drazen.lucanin at gmail.com>wrote:
>
>> On Sun, May 5, 2013 at 11:37 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>>
>>>
>>> Hi Drazen, it seems to me that making scipy.integrate time series aware
>>> is a significant enlargement of the scope of that module, and I'm not sure
>>> if this is the best way to go. Wouldn't it make more sense to add this in
>>> pandas itself?
>>>
>> Hi Ralf,
>>
>> well, I see the relation scipy : pandas practically the same as scipy :
>> numpy. Pandas offers basic functionality for dealing with data
>> (time-stamped in contrast to numpy, but still low-level),
>>
>
> Hmm, not sure I agree with that - pandas is in some respects more
> high-level than scipy. It includes some plotting tools for example. To
> build on pandas, statsmodels is a better candidate than scipy (given that
> statsmodels already has pandas as a dependency). I don't see us adding a
> dependency on pandas any time soon.
>
Yes, in a broader sense. But that's the dynamic nature of pandas that I
quite like - ti recognizes and uses other libraries if they're available,
but it only really requires numpy, python-dateutil and pytz.

https://github.com/pydata/pandas#dependencies

That's exactly the sort of pattern I had in mind for adding to
scipy.integrate - detect at runtime if pandas is installed and if so, work
with it - check the data type and adapt to it. If it is a pandas.TimeSeries
use the datetime arithmetic, instead of the lower-level numpy.ndarray
integer index that requires a domain transformation in order to integrate
based on seconds.

>
>
>> while scipy adds a layer of logic useful in specific scientific domains.
>> I guess numerical integration functionality isn't present in numpy's arrays
>> for the same reason.
>>
>> If nothing else, maintaining numerical integration methods in two places
>> seems a bit redundant.
>>
>>>
>>>
>>>> If there is any time left after this, the usability of scipy.integrate
>>>> for time series integration could be further improved by adding some new
>>>> features to Pandas too [2].
>>>>
>>>
>>> I think your proposal, even including the pandas issue [2], at the
>>> moment isn't enough for 3 months of work. For example, it should take much
>>> less than the two weeks you now have for implementing the trapezoidal rule
>>> integration (also, why even do that?).
>>>
>> Yes, it is a bit hard for me to predict the amount of work necessary for
>> the tasks I suggested, since I don't yet have a realistic picture of how
>> everything works together. For this, we could have used an additional
>> couple of feedback iterations.
>>
>
> We can still do that. You can't really edit your proposal on Melange
> anymore, but it would still be helpful to improve it. You can keep a copy
> on your blog or on Github for example.
>
If you agree, we could do that, yes. I'll put up a new version somewhere
tonight.

>
>
>>  With the trapoezoidal rule milestone I meant a very minimalistic working
>> version, on top of which additional iterations can be made to cover the
>> complete feature set (all the rules, all kinds of data, missing values etc.)
>>
>> I suggested the pandas issue more as an example. Of course, if the
>> original aims were satisfied sooner than expected, I would be glad to
>> continue working on other things as well. There is plenty of work to choose
>> from, I'd say. One direction might be to see if time series support would
>> make sense in other modules as well (interpolation (although pandas has its
>> own version - it would be interesting to see if the two could be merged
>> maybe), FFT, ...)
>>
>>>
>>> Scipy.integrate actually does need some TLC (it's unfortunately one of
>>> the least well maintained modules in scipy), but my feeling is that this
>>> wouldn't be high on the list of prios even if we did want to add time
>>> series functionality.
>>>
>> Another direction, as you say, might be to go deeper into the
>> scipy.integrate module itself. There is plenty of code regarding function
>> integration that could maybe be reorganised or even ported to something
>> more modern and maintainable than Fortran - maybe Cython. Integrating
>> functions doesn't perform quite well from what I heard (I think it was a
>> StackOverflow question, can't find it now), so it might make sense to
>> brainstorm on how to improve that.
>>
>
> See this thread for example:
> http://thread.gmane.org/gmane.comp.python.scientific.devel/16667/focus=16668.
> There were other ones as well on scipy-dev and scipy-user.
>
Sorry, I don't quite follow what you mean by that thread? So, this
efficient CVODE library is wrapped in scipy already? I see that there is
this interface that dives deep into the Fortran code right now - I guess
that's it?

https://github.com/scipy/scipy/blob/master/scipy/integrate/_ode.py

Dražen

>
>
> Ralf
>
>
>
>>
>> Regarding the priorities, that's for you guys to decide. I am interested
>> in the idea, as it's something that would help me with the way I'm using
>> scipy. Scratching my itch :)
>> http://www.catb.org/esr/writings/homesteading/cathedral-bazaar/ar01s02.html
>>
>> Cheers,
>> Dražen
>>
>>>
>>> Ralf
>>>
>>>
>>>  Is there perhaps anyone willing to mentor such work?
>>>>
>>>> Regards,
>>>> Dražen Lučanin
>>>>
>>>> [1]:
>>>> http://stackoverflow.com/questions/15203623/convert-pandas-datetimeindex-to-unix-time
>>>> [2]: https://github.com/pydata/pandas/issues/2704
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20130506/1f800202/attachment.html>