[SciPy-User] tool for running simulations

[Dan Goodman]

On 19/06/2011 22:47, Gael Varoquaux wrote:
> On Sun, Jun 19, 2011 at 10:40:29PM +0200, Dan Goodman wrote:
>> Also, the function cacheing part is quite important for what I have in
>> mind for it,
>
> Have you had a look joblib? Dag Sverre Seljebotn wants to do similar
> things than what you are talking about with it. He has a pull request to
> improve joblib to make it more suitable for that. I need to review it...

Gael, this is awesome. Almost exactly what I was looking for. A couple 
of questions:

* Is reading the data fast? At the moment I have a system built on 
Python shelves, and the performance is not great. My impression was that 
you'd built it with this in mind, so performance is probably very good.

* Can it be used on multiple computers? If not at the moment, is there 
at least a way to easily combine data produced on multiple computers? 
(e.g. just copying the contents of one directory to another)

* Can you browse the generated data easily? That's one thing I liked 
about the idea of doing it with HDF5 is that there are nice visual 
browsers and you can include metadata, search via metadata, remove parts 
of the data, etc.

* If I change the code for a function, will that cause a recompute? I'm 
guessing not, that it's done by the name/package of the function and not 
by the code. I think it's better that it doesn't cause a recompute, but 
given that having the ability to easily browse the cached data and 
remove the cache for a function would be very handy.

Dan