[Numpy-discussion] numpy fileIO

[Prashant Saxena]

I have just started doing my tests using numpy and hdpy( http://h5py.alfven..org/ ).
Hope either of these would give me what I need.

Prashant



----- Original Message ----
From: Francesc Alted <faltet at pytables.org>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Sent: Thursday, 16 October, 2008 10:03:22 PM
Subject: Re: [Numpy-discussion] numpy fileIO

A Thursday 16 October 2008, Prashant Saxena escrigué:
> Hi,
>
> I have never used numpy in my python applications until now. I am
> writing a python/openGL based tool to manipulate 3d geometrical
> data(meshes, curve, points etc.) I would be using numpy to store
> these objects(vertices, edges, faces, index etc.) at run time. One of
> my major concern is numpy's file IO capabilities. Geometrical objects
> would be stored in a structures, for example a logical structure to
> store a mesh goes like this:
>
> struct(
> name(string)
> vertex(numpy.float16)[x, y, z]
> normal(numpy.float16)[x, y, z]
> edges(numpy.int)
> st(2d numpy.float16)[u, v]
> index(numpy.int)
> )
>
> There would be different structures for curve, patch, points and rest
> of the primitives. I am sure numpy users must have encounered similar
> scenerio where you need to dump this data to a file and read it back.
> In my case, a general assumption of nearly 50-150 megs of data would
> be considered as normal size. Before I go deep into coding It would
> be great if numpy user can share their expreience for the task.
>
> I am also open for unconventional or off the route methods, provided
> they can do the job.(C/c++, 3rd party modules etc.) Here is the
> summery of IO operations I would be working on:
>
> 1. Write different structures to a file.
> 2. Read data back from file.
> 3. if structure can be tagged(int or string) then read a particular
> structure using tag, from file.

Your structure seems like can be expressed through a record array [1].  
If that is the case, just try the included tofile()/fromfile() 
utilities included in numpy for saving/retrieving them.

If you need more functionality for recarray I/O, you can have a look at 
pytables [2], which allows you to access directly the recarrays on disk 
row by row (even if compressed) or use complex expressions to quickly 
select interesting parts, among other niceties.

If your data structures can't be fit in recarray structures directly, 
you may want to try to decompose them in smaller constructs and relate 
them in file through pointers (indices to other constructs).

[1] http://www.scipy.org/RecordArrays
[2] http://www.pytables.org

Hope that helps,

-- 
Francesc Alted
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/74cd7e91/attachment.html>