[Numpy-discussion] record arrays with char*?

Tue Feb 11 12:23:22 EST 2014

On Tue, Feb 11, 2014 at 8:57 AM, Christopher Jordan-Squire
<cjordan1 at uw.edu>wrote:

> Thanks for the answers! My responses are inline.

my C is a bit weak, so forgive my misunderstanding, but:
>> {
>>  char* ind;
>>  double val, wght;
>> } data[] = { {"camera", 15, 2}, {"necklace", 100, 20}, {"vase", 90, 20},
>>                  {"pictures", 60, 30}, {"tv", 40, 40}, {"video", 15,
30}};

Here is my C weakness -- what does this struct look like in memory? i.e is
that first element a pointer, and the actually string data is somewhere
else in memory? In which case, where? and how does that memory get managed?

Or are the bytes at the begining of the struct with the string data in it?

in the "real" case, you say the char* is a string representation of a hex
value for a memory address -- so you would know a priory exactly what the
length of that sting is, so you could use a numpy struct like:

('S8,f8,f8')

i.e 8 bytes to store the string at the begining of the struct.

then & the_array[i] would be the address of the beggining of the string --
which may be what you want.

HTH,
  -Chris

> >>
> >> At the C level, data is passed to the function by directly giving its
> >> address. (i.e. the C function takes as an argument (unsigned long)
> >> data, casting the data pointer to an int)
> >
> >
> > wow -- that's prone to error! but I"m still not sure which pointer you're
> > talking about -- a pointer to this struct?
> >
>
> I mean data. data is an array of structs, but in C that's
> (essentially) the same as a pointer to a struct. So data is the
> pointer I'm referring to.
>
> >> I'd like to create something similar using record arrays, such as
> >>
> >> np.array([("camera", 15, 2), ("necklace", 100, 20), ... ],
> >> dtype='object,f8,f8').
> >
> >
> >
> >>
> >> Unfortunately this fails because
> >> (1) In cython I need to determine the address of the first element and
> >> I can't take the address of a an input whose type I don't know (the
> >> exact type will vary on the application, so more or fewer fields may
> >> be in the C struct)
> >
> >
> > still a bit confused, but if this is types as an array in Cython, you
> should
> > be abel to do somethign like:
> >
> > &the_array[i]
> >
> > to get the address of the ith element.
> >
>
> To do that I need to be able to tell cython the type of the memory
> view I give. There are very few examples for non-primitive arrays in
> cython, so I'm not sure what that would look like. Or at least I
> *think* I need to do that, based on the cython errors I'm getting.
>
> >> (2) I don't think a python object type is what I want--I need a char*
> >> representation of the string. (Unfortunately I can't test this because
> >> I haven't solved (1) -- how do you pass a record array around in
> >> cython and/or take its address?)
> >
> >
> > well, and object type will give you a pointer to a pyobject. If you know
> for
> > sure that that pyobject is a string object (probably want a bytes object
> --
> > you son't want unicode here), then you should be abel to get the address
> of
> > the underlying char array. But that would require passing something
> > different off to the C code that the address of that element.
> >
> > You could use an unsigned long for that first field, as you are assuming
> > that in the C code anyway  but I don't hink there is a way in numpy to
> set
> > that to a pointer to a char allocated elsewhere -- where would it be
> > allocated?
> >
>
> The strings are char*, the unsigned long cast was simply to cast data
> to a memory address. (The specific setup of the library was passing a
> string with the memory location in hexadecimal. This seems weird, but
> it's because the C functions being called are an intermediary to a
> simple (third-party) virtual machine running a program compiled from
> another language. It doesn't deal with pointers directly, so the other
> language is just passed the memory address, in hex and as a string,
> for where the data resides. Along with the number of elements in the
> block of memory pointed to.)
>
> > So I would give up on expecting to store the struct directly in numpy
> array,
> > and rather, put something reasonable (maybe what you have above) in the
> > numpy array, and build the C struct you need from that rather than
> passing a
> > pointer in directly.
> >
>
> That sounds reasonable. I really wanted to avoid this because, as I
> mentioned above, I'm just trying to generate the data in numpy and
> pass it to this virtual machine running a program compiled from
> another language. The form of the struct depends on what was done in
> the other language. It could easily have more or fewer fields, have
> the fields reordered, etc.. I wanted to avoid having to write wrappers
> for all such possibilities and instead use numpy record arrays as
> mapping exactly to C structs. But if that's really the only way to go
> then I guess I'll have to write struct wrappers in cython.
>
> > -Chris
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR&R            (206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115       (206) 526-6317   main reception
> >
> > Chris.Barker at noaa.gov
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140211/aa2eda90/attachment.html>