[Numpy-discussion] Question about structure arrays

[aerojockey]

Nathaniel Smith wrote
> On Sat, Nov 7, 2015 at 1:18 PM, aerojockey <

> pythondev1@

> > wrote:
>> Hello,
>>
>> Recently I made some changes to a program I'm working on, and found that
>> the
>> changes made it four times slower than before.  After some digging, I
>> found
>> out that one of the new costs was that I added structure arrays.  Inside
>> a
>> low-level loop, I create a structure array, populate it Python, then turn
>> it
>> over to some handwritten C code for processing.  It turned out that, when
>> passed a structure array as a dtype, numpy has to parse the dtype, which
>> included calls to re.match and eval.
>>
>> Now, this is not a big deal for me to work around by using ordinary
>> slicing
>> and such, and also I can improve things by reusing arrays.  Since this is
>> inner loop stuff, sacrificing readability for speed is an appropriate
>> tradeoff.
>>
>> Nevertheless, I was curious if there was a way (or any plans for there to
>> be
>> a way) to compile a struture array dtype.  I realize it's not the
>> bread-and-butter of numpy, but it turned out to be a very convenient
>> feature
>> for my use case (populating an array of structures to pass off to C).
> 
> Does it help to turn your dtype string into a dtype object and then
> pass the dtype object around? E.g.
> 
> In [1]: dt = np.dtype("i4,i4")
> 
> In [2]: np.zeros(2, dtype=dt)
> Out[2]:
> array([(0, 0), (0, 0)],
>       dtype=[('f0', '<i4'), ('f1', '<i4')])
> 
> -n


I actually don't know, since I removed the structure array part about ten
minutes after I posted.  However, I did a quick test of your suggestion, and
indeed numpy calls exec and re.match only when creating the dtype object,
not when creating the array.  So certainly it would have helped.

I wasn't actually aware you could do that with dtypes.  In fact, I was only
vaguely that there were dtype types at all.  Thanks for the suggestion.

Carl Banks



--
View this message in context: http://numpy-discussion.10968.n7.nabble.com/Question-about-structure-arrays-tp41653p41676.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.