[SciPy-User] Populating a recarray from 0 size

[josef.pktd at gmail.com]

On Sun, Aug 22, 2010 at 2:23 PM, Sergi Pons Freixes <spons at utm.csic.es> wrote:
> On Sat, Aug 21, 2010 at 6:29 PM,  <josef.pktd at gmail.com> wrote:
>> BTW: These are numpy functions.
>
> Oh, sorry, I am so used to call numpy functions via scipy that I
> forgot about it.
>
>>
>> I don't know the answer, but it would be more efficient to correctly
>> size the empty array, if you know the size, or to build up the array
>> first as a list of tuples before converting to an array.
>
> The problem is that the size is not known. I'll think about the list of tuples.
>
>
> On Sun, Aug 22, 2010 at 8:19 AM, Åsmund Hjulstad
> <asmund.hjulstad at gmail.com> wrote:
>>
>>
>> My approach in this kind of situation has been using pytables, creating an
>> empty table with a specific dtype, and then populating the table with the
>> append method. It seems very efficient, and quite convenient in situations
>> where I eventually will need to persist data to disk somehow.
>> Reading the table into an in memory recarray is then easily done using
>> data = h5file.root.datatable[:].view(numpy.recarray)
>>
>
> Mmm, it sounds a bit overkill to use pytables just for having a
> recarray with dynamic size. Anyway, I'll have a look at it, thank you.
>
>
> I would also appreciate is a reasoning of why is scipy.resize (e.g.,
> numpy.resize) raising this exception. The idea increasing the size of
> a 0-size array doesn't sound very weird to me, so I was shocked that i
> wasn't working. Actually, I've been successfully using the method
> array.resize(newsize) previously, but now I'm "forced" to use
> scipy.resize because of the "ValueError: cannot resize an array that
> has been referenced or is referencing another array in this way."
> issue (see [1]).

I think it should be a bug report on the numpy trac, given that the
method works but not the function.

To use the method, you might also be able to make a copy yourself to
break the second reference. (Not tried out).

I still think copying the entire array whenever a new item is added is
not a good idea (very costly in a loop).

Josef

>
> If this conversation is more appropriate to be developed in the numpy
> mailing list, please let me now and I will migrate it.
>
> Regards,
> Sergi
>
> [1] http://www.scipy.org/Numpy_Example_List#head-d5585278b02f43f78eb75a4fdc337fdc12417426
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>