[SciPy-User] re[SciPy-user] moving for loops...

[mdekauwe]

Also I then need to remap the 2D array I make onto another grid (the world in
this case). Which again I had am doing with a loop (note numpts is a lot
bigger than my example above).

wilcoxStats_snow = np.ones((outrows, outcols), dtype=np.float32) * np.nan
for i in xrange(numpts):
        # exclude the NaN, note masking them doesn't work in the stats func
        x = data1_snow[:,i]
        x = x[np.isfinite(x)]
        y = data2_snow[:,i]
        y = y[np.isfinite(y)]
        
        # wilcox signed rank test
        # make sure we have enough samples to do the test
        d = x - y
        d = np.compress(np.not_equal(d,0), d ,axis=-1) # Keep all non-zero
differences
        count = len(d)
        if count > 10:
            z, pval = stats.wilcoxon(x, y)
            # only map out sign different data
            if pval < 0.05:
                wilcoxStats_snow[((180-1)-(rows[i]-1)),cols[i]-1] =
np.mean(x - y)

Now I think I can push the data in one move into the wilcoxStats_snow array
by removing the index, 
but I can't see how I will get the individual x and y pts for each array
member correctly without the loop, this was my attempt which of course
doesn't work!

x = data1_snow[:,:]
x = x[np.isfinite(x)]
y = data2_snow[:,:]
y = y[np.isfinite(y)]
    
# r^2
# exclude v.small arrays, i.e. we need just less over 4 years of data
if len(x) and len(y) > 50:
    pearsonsr_snow[((180-1)-(rows-1)),cols-1] = (stats.pearsonr(x, y)[0])**2


thanks.




mdekauwe wrote:
> 
> Yes as Zachary said index is only 0 to 15237, so both methods work.
> 
> I don't quite get what you mean about slicing with axis > 3. Is there a
> link you can recommend I should read? Does that mean given I have 4dims
> that Josef's suggestion would be more advised in this case?
> 
> Thanks.
> 
> 
> 
> josef.pktd wrote:
>> 
>> On Fri, May 21, 2010 at 10:55 AM, mdekauwe <mdekauwe at gmail.com> wrote:
>>>
>>> Thanks that works...
>>>
>>> So the way to do it is with np.arange(tsteps)[:,None], that was the step
>>> I
>>> was struggling with, so this forms a 2D array which replaces the the two
>>> for
>>> loops? Do I have that right?
>> 
>> Yes, but as Zachary showed, if you need the full index in a dimension,
>> then you can use slicing. It might be faster.
>> And a warning, mixing slices and index arrays with 3 or more
>> dimensions can have some surprise switching of axes.
>> 
>> Josef
>> 
>>>
>>> A lot quicker...!
>>>
>>> Martin
>>>
>>>
>>> josef.pktd wrote:
>>>>
>>>> On Fri, May 21, 2010 at 8:59 AM, mdekauwe <mdekauwe at gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to extract data from a 4D array and store it in a 2D
>>>>> array,
>>>>> but
>>>>> avoid my current usage of the for loops for speed, as in reality the
>>>>> arrays
>>>>> sizes are quite big. Could someone also try and explain the solution
>>>>> as
>>>>> well
>>>>> if they have a spare moment as I am still finding it quite difficult
>>>>> to
>>>>> get
>>>>> over the habit of using loops (C convert for my sins). I get that one
>>>>> could
>>>>> precompute the indices's i and j i.e.
>>>>>
>>>>> i = np.arange(tsteps)
>>>>> j = np.arange(numpts)
>>>>>
>>>>> but just can't get my head round how i then use them...
>>>>>
>>>>> Thanks,
>>>>> Martin
>>>>>
>>>>> import numpy as np
>>>>>
>>>>> numpts=10
>>>>> tsteps = 12
>>>>> vari = 22
>>>>>
>>>>> data = np.random.random((tsteps, vari, numpts, 1))
>>>>> new_data = np.zeros((tsteps, numpts), dtype=np.float32)
>>>>> index = np.arange(numpts)
>>>>>
>>>>> for i in xrange(tsteps):
>>>>>    for j in xrange(numpts):
>>>>>        new_data[i,j] = data[i,5,index[j],0]
>>>>
>>>> The index arrays need to be broadcastable against each other.
>>>>
>>>> I think this should do it
>>>>
>>>> new_data = data[np.arange(tsteps)[:,None], 5, np.arange(numpts), 0]
>>>>
>>>> Josef
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/removing-for-loops...-tp28633477p28633477.html
>>>>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/removing-for-loops...-tp28633477p28634924.html
>>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/removing-for-loops...-tp28633477p28640656.html
Sent from the Scipy-User mailing list archive at Nabble.com.