[Numpy-discussion] Seg fault from numpy.rec.fromarrays

[Bruce Southey]

Dan Yamins wrote:
>
>
>     > Then I attempt
>     >    >>> A = numpy.rec.fromarrays(L,names =
>     ['Aggregates','__color__'])
>
>     So what happens when you set the dtype here?
>
>
>     Since you have variable lengths of strings, numpy probably has guessed
>     incorrectly. I would also check that Col1 and Col2 are what you
>     expect,
>     especially the minimum lengths and really are strings.
>
>
> These objects to indeed seem to by exactly what I expect.   The dtype 
> appears exactly to be right.   I had the same problem when I provided 
> the correct dtype by hand, and numpy seems to be guessing right when I 
> don't.
>
>
>     Can you provide a small example that exhibits the problem?
>
>
> Well that's the problem.  I can't easily.  The code that makes the 
> example that crashes is buried fairly deeply in some other routines.  
> When I try to produce the proximate problem manually by creating what 
> should (in theory) be identical lists, at the interpreter prompt, I 
> don't get the segfault problem.   
Well you should be able to save the Col1 and Col2 lists...
Hopefully if you save these as files and read them back but I would 
guess that is what pickle is doing.


>
> I've attached a .png picture of an short interpreter session where I 
> show the result of what is returned by the routines -- the object that 
> causes fault in its behavior, how its dtype seems right, and how 
> pickling it solves the problem.  If you can open attachments, perhaps 
> looking at this would be instructive.
In the picture many of the list elements appear empty strings but it is 
not clear if these are really empty strings. The error for A[0] may 
indicate that especially if other indices work such as those to 
individual elements.

>
> I know the problem must have something to do with the way the lists 
> Col1 and Col2 are getting created -- and that somehow, in a manner I 
> don't understand and is not readily apparent by i, they are not what I 
> think they are.  But I can't find out how they're wrong.   I was 
> wondering if anybody had encountered similar problems that might help 
> me narrow my search. 
I would try:
1) create separate arrays from Col1 and Col2 and check these can be 
viewed. Then create an recarray from those.
2) create list L from subsets of the Col1 and Col2 until I found which 
entries cause it.


>
> I'm also willing to share all the code that leads up to the problem, 
> if that's the only way to identify the problem, although it would be a 
> somewhat laborious effort I think to subject you all to :)
An example is always useful. While I do not have a Mac, you can sent it 
to me on or offlist.

Bruce