[Numpy-discussion] loop through values in a array and find maximum as looping
questions anon
questions.anon at gmail.com
Tue Dec 6 22:49:52 EST 2011
Something fancier I think,
I am able to compare the result with my previous method so I can easily see
I am doing something wrong.
see code below:
all_TSFC=[]
for (path, dirs, files) in os.walk(MainFolder):
for dir in dirs:
print dir
path=path+'/'
for ncfile in files:
if ncfile[-3:]=='.nc':
print "dealing with ncfiles:", ncfile
ncfile=os.path.join(path,ncfile)
ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
TSFC=ncfile.variables['T_SFC'][:]
fillvalue=ncfile.variables['T_SFC']._FillValue
TSFC=MA.masked_values(TSFC, fillvalue)
ncfile.close()
all_TSFC.append(TSFC)
a=TSFC[0]
for b in TSFC[1:]:
N.maximum(a,b,out=a)
big_array=N.ma.concatenate(all_TSFC)
Max=big_array.max(axis=0)
print "max is", Max,"a is", a
On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <shish at keba.be> wrote:
> Is 'a' a regular numpy array or something fancier?
>
>
> -=- Olivier
>
> 2011/12/6 questions anon <questions.anon at gmail.com>
>
>> thanks again my only problem though is that the out=a in the loop does
>> not seem to replace my a= outside the loop so my final a is whatever I
>> started with for a.
>> Not sure what I am doing wrong whether it is something with the loop or
>> with the command.
>>
>> On Wed, Dec 7, 2011 at 1:44 PM, <josef.pktd at gmail.com> wrote:
>>
>>> On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <shish at keba.be> wrote:
>>> > The "out=a" keyword will ensure your first array will keep being
>>> updated. So
>>> > you can do something like:
>>> >
>>> > a = my_list_of_arrays[0]
>>> > for b in my_list_of_arrays[1:]:
>>> > numpy.maximum(a, b, out=a)
>>>
>>> I didn't think of the out argument which makes it more efficient, but
>>> in my example I used Python's reduce which takes an iterable and not
>>> one huge array.
>>>
>>> Josef
>>>
>>>
>>> >
>>> > -=- Olivier
>>> >
>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>> >>
>>> >> thanks for all of your help, that does look appropriate but I am not
>>> sure
>>> >> how to loop it over thousands of files.
>>> >> I need to keep the first array to compare with but replace any greater
>>> >> values as I loop through each array comparing back to the same array.
>>> does
>>> >> that make sense?
>>> >>
>>> >>
>>> >> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <shish at keba.be>
>>> wrote:
>>> >>>
>>> >>> Thanks, I didn't know you could specify the out array :)
>>> >>>
>>> >>> (to the OP: my initial suggestion, although probably not very
>>> efficient,
>>> >>> seems to work with 2D arrays too, so I have no idea why it didn't
>>> work for
>>> >>> you -- but Nathaniel's one seems to be the ideal one anyway).
>>> >>>
>>> >>> -=- Olivier
>>> >>>
>>> >>>
>>> >>> 2011/12/6 Nathaniel Smith <njs at pobox.com>
>>> >>>>
>>> >>>> I think you want
>>> >>>> np.maximum(a, b, out=a)
>>> >>>>
>>> >>>> - Nathaniel
>>> >>>>
>>> >>>> On Dec 6, 2011 9:04 PM, "questions anon" <questions.anon at gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> thanks for responding Josef but that is not really what I am
>>> looking
>>> >>>>> for, I have a multidimensional array and if the next array has any
>>> values
>>> >>>>> greater than what is in my first array I want to replace them. The
>>> data are
>>> >>>>> contained in netcdf files.
>>> >>>>> I can achieve what I want if I combine all of my arrays using numpy
>>> >>>>> concatenate and then using the command numpy.max(myarray, axis=0)
>>> but
>>> >>>>> because I have so many arrays I end up with a memory error so I
>>> need to find
>>> >>>>> a way to get the maximum while looping.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.pktd at gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <shish at keba.be>
>>> >>>>>> wrote:
>>> >>>>>> > It may not be the most efficient way to do this, but you can do:
>>> >>>>>> > mask = b > a
>>> >>>>>> > a[mask] = b[mask]
>>> >>>>>> >
>>> >>>>>> > -=- Olivier
>>> >>>>>> >
>>> >>>>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>> >>>>>> >>
>>> >>>>>> >> I would like to produce an array with the maximum values out of
>>> >>>>>> >> many
>>> >>>>>> >> (10000s) of arrays.
>>> >>>>>> >> I need to loop through many multidimentional arrays and if a
>>> value
>>> >>>>>> >> is
>>> >>>>>> >> larger (in the same place as the previous array) then I would
>>> like
>>> >>>>>> >> that
>>> >>>>>> >> value to replace it.
>>> >>>>>> >>
>>> >>>>>> >> e.g.
>>> >>>>>> >> a=[1,1,2,2
>>> >>>>>> >> 11,2,2
>>> >>>>>> >> 1,1,2,2]
>>> >>>>>> >> b=[1,1,3,2
>>> >>>>>> >> 2,1,0,0
>>> >>>>>> >> 1,1,2,0]
>>> >>>>>> >>
>>> >>>>>> >> where b>a replace with value in b, so the new a should be :
>>> >>>>>> >>
>>> >>>>>> >> a=[1,1,3,2]
>>> >>>>>> >> 2,1,2,2
>>> >>>>>> >> 1,1,2,2]
>>> >>>>>> >>
>>> >>>>>> >> and then keep looping through many arrays and replace whenever
>>> >>>>>> >> value is
>>> >>>>>> >> larger.
>>> >>>>>> >>
>>> >>>>>> >> I have tried numpy.putmask but that results in
>>> >>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not list
>>> >>>>>> >> Any other ideas? Thanks
>>> >>>>>>
>>> >>>>>> if I understand correctly it's a minimum.reduce
>>> >>>>>>
>>> >>>>>> numpy
>>> >>>>>>
>>> >>>>>> >>> a = np.concatenate((np.arange(5)[::-1],
>>> >>>>>> >>> np.arange(5)))*np.ones((4,3,1))
>>> >>>>>> >>> np.minimum.reduce(a, axis=2)
>>> >>>>>> array([[ 0., 0., 0.],
>>> >>>>>> [ 0., 0., 0.],
>>> >>>>>> [ 0., 0., 0.],
>>> >>>>>> [ 0., 0., 0.]])
>>> >>>>>> >>> a.T.shape
>>> >>>>>> (10, 3, 4)
>>> >>>>>>
>>> >>>>>> python with iterable
>>> >>>>>>
>>> >>>>>> >>> reduce(np.maximum, a.T)
>>> >>>>>> array([[ 4., 4., 4., 4.],
>>> >>>>>> [ 4., 4., 4., 4.],
>>> >>>>>> [ 4., 4., 4., 4.]])
>>> >>>>>> >>> reduce(np.minimum, a.T)
>>> >>>>>> array([[ 0., 0., 0., 0.],
>>> >>>>>> [ 0., 0., 0., 0.],
>>> >>>>>> [ 0., 0., 0., 0.]])
>>> >>>>>>
>>> >>>>>> Josef
>>> >>>>>>
>>> >>>>>> >>
>>> >>>>>> >> _______________________________________________
>>> >>>>>> >> NumPy-Discussion mailing list
>>> >>>>>> >> NumPy-Discussion at scipy.org
>>> >>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>>>> >>
>>> >>>>>> >
>>> >>>>>> >
>>> >>>>>> > _______________________________________________
>>> >>>>>> > NumPy-Discussion mailing list
>>> >>>>>> > NumPy-Discussion at scipy.org
>>> >>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>>>> >
>>> >>>>>> _______________________________________________
>>> >>>>>> NumPy-Discussion mailing list
>>> >>>>>> NumPy-Discussion at scipy.org
>>> >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> NumPy-Discussion mailing list
>>> >>>>> NumPy-Discussion at scipy.org
>>> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> NumPy-Discussion mailing list
>>> >>>> NumPy-Discussion at scipy.org
>>> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> NumPy-Discussion mailing list
>>> >>> NumPy-Discussion at scipy.org
>>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> NumPy-Discussion mailing list
>>> >> NumPy-Discussion at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111207/c1ae50bd/attachment.html>
More information about the NumPy-Discussion
mailing list