[Numpy-discussion] Thoughts on persistence/object tracking in scientific code

[Gael Varoquaux]

On Tue, Dec 23, 2008 at 02:10:50AM +0100, Olivier Grisel wrote:
>    Interesting topic indeed. I think I have been hit with similar problems on
>    toy experimental scripts. So far the solution was always adhoc FS caches
>    of numpy arrays with manual filename management. Maybe the first step for
>    designing a generic solution would be to list some representative yet
>    simple enough use cases with real sample python code so as to focus on
>    concrete matters and avoid over engineering a general solution for
>    philosophical problems.

Yes, that's clearly a first ste: list the usecases, and the way we would
like it solved: think about the API.

My internet connection is quite random currently, and I'll probably loose
it for a week any time soon. Do you want to start such a page on the
wiki. Mark it as a sratch page, and we'll delete it later.

I should point out that joblib (on PyPI and launchpad) was a first
attempt to solve this problem, so you could have a look at it. I have
already identified things that are wrong with joblib (more on the API
side than actual bugs), so I know it is not a final solution. Figuring
out what was wrong only came from using it heavily in my work. I thing
the only way forward it to start something, use it, figure out what's
wrong, and start again...

Looking forward to your input,

Gaël