[Numpy-discussion] vectorizing loops

[Francesc Altet]

A Wednesday 31 October 2007, Timothy Hochberg escrigué:
> On Oct 31, 2007 3:18 AM, Francesc Altet <faltet at carabos.com> wrote:
>
> [SNIP]
>
> > Incidentally, all the improvements of the PyTables flavor of
> > numexpr have been reported to the original authors, but, for the
> > sake of keeping numexpr simple, they decided to implement only some
> > of them. However, people is encouraged to try out the Pytables
> > flavor from:
>
> My original concern was that we'd start overflowing the code cache
> and slow things down. Some experiments seemed to confirm this on some
> processors, but it then turned out those were in error. At least
> that's my recollection. Because of that, and because PyTables is, as
> far as I know, the major user of numexpr, I suspect I'd be fine
> putting those changes in now. I say "suspect" since it's been a long
> time since I looked at the relevant patches, and I should probably
> look at those again before I commit myself. I just haven't had the
> free time and motivation to go back and look at the patches.  I can't
> speak for David though.

Yes, I remember that you were mainly concerned with overflowing the code 
cache, and yes, I think you can be confident this is not the case, as 
my benchmarking in many CPU's (ranging from 7 year old AMD Duron, 
Pentiums 4 and up to relatively recents AMD Opterons), doesn't show any 
slow down with the PyTables version of numexpr, but rather the 
contrary, it speeds things up, specially in contexts where booleans 
and/or strided/unaligned arrays are used, where the improvement can be 
up to 2x on recent CPU's (as shown in my previous post).

If I remember correctly, another point where you (specially David) were 
not very keen to include was the support for strings, arguing that 
numexpr is meant mainly to deal with numerical data, not textual.  
However, our experience is that adding this support was not too 
complicated (mainly due to NumPy flexibility), and can be helpful in 
some instances.  As for one, we use them for evaluating expressions 
like 'array_string == "some_string"', and this can be very convenient 
to use when you are in the middle of potentially complex boolean 
expressions that you want to evaluate *fast*.

At any rate, we would be glad if you would like to integrate our patches 
in the main numexpr, as there is not much sense to have different 
implementations of numexpr (most specially when it seems that there are 
not much users out there).  So, count on us for any question you may 
have in this regard.

Cheers,

-- 
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   Cárabos Coop. V.   Enjoy Data
 "-"