[Numpy-discussion] tofile speed

[Sebastian Haase]

Just a guess out of my hat:
there might be a buffer class in the standard python library... I'm
thinking of a class that implements file-I/O and collects input up to
a maximum buffer size before it copies the same byte stream to it's
output. Since I/O is more efficient if larger chunks are written this
should improve the overall performance.

How large are your data-chunks per write ? (IOW: what is len(temp.data))

HTH,
Sebastian Haase


On 7/23/07, Lars Friedrich <lfriedri at imtek.de> wrote:
> Hello everyone,
>
> I am using array.tofile successfully for a data-acqusition-streaming
> application. I mean that I do the following:
>
> for a long time:
>        temp = dataAcquisisionDevice.getData()
>        temp.tofile(myDataFile)
>
> temp is a numpy array that is used for storing the data temporarily. The
> data acquisition device is acquiring continuously and writing the data
> to a buffer from which I can read with .getData(). This works fine, but
> of course, when I turn the sample rate higher, there is a point when
> temp.toFile is too slow. The dataAcquisitionDevice's buffer will run
> full before I can fetch the data again.
>
> (temp has a size of ~Mbyte, and the for loop has a period of ~0.5
> seconds so that increasing the chunk size won't help)
>
> I have no idea how efficient array.tofile() is. Maybe it is terribly
> efficient and what I see is just the limitation of my hardware
> (harddisk). Currently I can stream with roughly 4 Mbyte/s, which is
> quite fast, I guess. However, if anyone can point me to a way to write
> my data to harddisk faster, I would be very happy!
>
> Thanks
>
> Lars
>
>
> --
> Dipl.-Ing. Lars Friedrich
>
> Photonic Measurement Technology
> Department of Microsystems Engineering -- IMTEK
> University of Freiburg
> Georges-Köhler-Allee 102
> D-79110 Freiburg
> Germany
>
> phone: +49-761-203-7531
> fax:   +49-761-203-7537
> room:  01 088
> email: lfriedri at imtek.de
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>