[SciPy-user] Presentation of pymachine, a python package for machine learning

[Matthieu Brucher]

>
> I understand the limitation. I think it is important on the frontend
> side to have a global mechanism to enable ondisk data, for streaming
> data directly from files instead of loading everything in memory. But
> then, there is a problem on the back-end side: most algorithms expects
> all their input data at once.


We have the same problem here, one image is 64Mb, we use several images at
once, not talking about deformation fields (400Mb).
What we thought, as a solution, was to write down after each filter the
result on the hard drive (simple to display to know if something went wrong,
simple to reload, ...) and to load what was needed by the next filter.
Well that does not work for too-bug-to-fit-in-memory datasets, but it is a
good-for-a-start policy.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20070514/09d8420a/attachment.html>