[SciPy-Dev] scipy.integrate optimisation for pandas.TimeSeries

[Ralf Gommers]

On Sun, May 5, 2013 at 11:24 PM, Dražen Lučanin <drazen.lucanin at gmail.com>wrote:

> On Sun, May 5, 2013 at 11:37 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>
>>
>> Hi Drazen, it seems to me that making scipy.integrate time series aware
>> is a significant enlargement of the scope of that module, and I'm not sure
>> if this is the best way to go. Wouldn't it make more sense to add this in
>> pandas itself?
>>
> Hi Ralf,
>
> well, I see the relation scipy : pandas practically the same as scipy :
> numpy. Pandas offers basic functionality for dealing with data
> (time-stamped in contrast to numpy, but still low-level),
>

Hmm, not sure I agree with that - pandas is in some respects more
high-level than scipy. It includes some plotting tools for example. To
build on pandas, statsmodels is a better candidate than scipy (given that
statsmodels already has pandas as a dependency). I don't see us adding a
dependency on pandas any time soon.


> while scipy adds a layer of logic useful in specific scientific domains. I
> guess numerical integration functionality isn't present in numpy's arrays
> for the same reason.
>
> If nothing else, maintaining numerical integration methods in two places
> seems a bit redundant.
>
>>
>>
>>> If there is any time left after this, the usability of scipy.integrate
>>> for time series integration could be further improved by adding some new
>>> features to Pandas too [2].
>>>
>>
>> I think your proposal, even including the pandas issue [2], at the moment
>> isn't enough for 3 months of work. For example, it should take much less
>> than the two weeks you now have for implementing the trapezoidal rule
>> integration (also, why even do that?).
>>
> Yes, it is a bit hard for me to predict the amount of work necessary for
> the tasks I suggested, since I don't yet have a realistic picture of how
> everything works together. For this, we could have used an additional
> couple of feedback iterations.
>

We can still do that. You can't really edit your proposal on Melange
anymore, but it would still be helpful to improve it. You can keep a copy
on your blog or on Github for example.


> With the trapoezoidal rule milestone I meant a very minimalistic working
> version, on top of which additional iterations can be made to cover the
> complete feature set (all the rules, all kinds of data, missing values etc.)
>
> I suggested the pandas issue more as an example. Of course, if the
> original aims were satisfied sooner than expected, I would be glad to
> continue working on other things as well. There is plenty of work to choose
> from, I'd say. One direction might be to see if time series support would
> make sense in other modules as well (interpolation (although pandas has its
> own version - it would be interesting to see if the two could be merged
> maybe), FFT, ...)
>
>>
>> Scipy.integrate actually does need some TLC (it's unfortunately one of
>> the least well maintained modules in scipy), but my feeling is that this
>> wouldn't be high on the list of prios even if we did want to add time
>> series functionality.
>>
> Another direction, as you say, might be to go deeper into the
> scipy.integrate module itself. There is plenty of code regarding function
> integration that could maybe be reorganised or even ported to something
> more modern and maintainable than Fortran - maybe Cython. Integrating
> functions doesn't perform quite well from what I heard (I think it was a
> StackOverflow question, can't find it now), so it might make sense to
> brainstorm on how to improve that.
>

See this thread for example:
http://thread.gmane.org/gmane.comp.python.scientific.devel/16667/focus=16668.
There were other ones as well on scipy-dev and scipy-user.

Ralf



>
> Regarding the priorities, that's for you guys to decide. I am interested
> in the idea, as it's something that would help me with the way I'm using
> scipy. Scratching my itch :)
> http://www.catb.org/esr/writings/homesteading/cathedral-bazaar/ar01s02.html
>
> Cheers,
> Dražen
>
>>
>> Ralf
>>
>>
>>  Is there perhaps anyone willing to mentor such work?
>>>
>>> Regards,
>>> Dražen Lučanin
>>>
>>> [1]:
>>> http://stackoverflow.com/questions/15203623/convert-pandas-datetimeindex-to-unix-time
>>> [2]: https://github.com/pydata/pandas/issues/2704
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20130506/c8784d65/attachment.html>