[Numpy-discussion] saving incrementally numpy arrays

[Juan Fiol]

Hi, again, I can confirm that you have to load multiple times. Also I do not see differences if using or not the f.seek line
The following snippet gives the expected result. The problem is that that way I have to load as many times as I wrote. Besides that, it works. Thanks, Juan
#-----------------------------------------
import numpy as np

x = np.array([[1,2,3],[4,5,6]])
y = np.array([[7,8,9],[10,11,12]])

f = open('myfile1.npy', 'a+b')
np.save(f, x)
# f.seek(0, 2)
np.save(f, y)
f.close()

fi=open('myfile1.npy','rb')
x1 = np.load(fi)
y1 = np.load(fi)
fi.close()
#-----------------------------------------

--- On Tue, 8/11/09, Juan Fiol <fiolj at yahoo.com> wrote:

> From: Juan Fiol <fiolj at yahoo.com>
> Subject: Re: [Numpy-discussion] saving incrementally numpy arrays
> To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
> Date: Tuesday, August 11, 2009, 8:28 PM
> Hi, thanks for all the answers. I am
> checking how to use pytables now, though I probably prefer
> to do it without further dependencies. I tried opening the
> file as 'append' and then pickle the array (because looking
> to the numpy.save it looked like what they did), but to
> retrieve the data then I have to load multiple times and
> concatenate (numpy.c_[]). I did not tried Robert suggestion
> yet, but it will probably happen the same and that is what
> Keith is seeing (though I may be wrong too).
> If I do not find a suitable solution with only numpy I'll
> learn how to use pytables. Thanks and Best regards,
> Juan
> 
> --- On Tue, 8/11/09, Keith Goodman <kwgoodman at gmail.com>
> wrote:
> 
> > From: Keith Goodman <kwgoodman at gmail.com>
> > Subject: Re: [Numpy-discussion] saving incrementally
> numpy arrays
> > To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
> > Date: Tuesday, August 11, 2009, 7:46 PM
> > On Tue, Aug 11, 2009 at 11:05 AM,
> > Robert Kern<robert.kern at gmail.com>
> > wrote:
> > > On Mon, Aug 10, 2009 at 22:29, Juan Fiol<fiolj at yahoo.com>
> > wrote:
> > >> Hi, I am creating numpy arrays in chunks and
> I
> > want to save the chunks while my program creates them.
> I
> > tried to use numpy.save but it failed (because it is
> not
> > intended to append data). I'd like to know what is, in
> your
> > opinion, the best way to go. I will put a few
> thousands
> > every time but building up a file of several Gbytes. I
> do
> > not want to put into memory
> > >> all previous data each time. Also I cannot
> wait
> > until the program finishes, I must save partial
> results
> > periodically. Thanks, any help will be appreciated
> > >
> > > As others mentioned, PyTables is an excellent,
> > complete solution. If
> > > you still want to write your own, then you can
> pass an
> > open file
> > > object to numpy.save() in order to append. Just
> open
> > it with the mode
> > > 'a+b' and seek to the end.
> > >
> > >  f = open('myfile.npy', 'a+b')
> > >  f.seek(0, 2)
> > >  numpy.save(f, chunk)
> > >  f.close()
> > 
> > That looks nice. What am I doing wrong?
> > 
> > >> x = np.array([1,2,3])
> > >> y = np.array([4,5,6])
> > >>
> > >> f = open('myfile.npy', 'a+b')
> > >> np.save(f, x)
> > >> f.seek(0, 2)
> > >> np.save(f, y)
> > >> f.close()
> > >>
> > >> xy = np.load('myfile.npy')
> > >> xy
> >    array([1, 2, 3])
> > 
> > I was expecting something like array([1, 2, 3, 4, 5,
> 6]).
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > 
> 
> 
>       
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>