[Numpy-discussion] loop through values in a array and find maximum as looping
questions anon
questions.anon at gmail.com
Tue Dec 6 23:54:34 EST 2011
sorry the 'all_TSFC' is for my other check of maximum using concatenate and
N.max, I know that works so I am comparing it to this method. The only
reason I need another method is for memory error issues.
I like the code I have written so far as it makes sense to me. I can't get
the extra examples I have been given to work and that is most likely
because I don't understand them, these are the errors I get :
Traceback (most recent call last):
File "d:\plot_summarystats\test_plot_remove_memoryerror_max.py", line 46,
in <module>
N.maximum(a,TSFC,out=a)
ValueError: non-broadcastable output operand with shape (106,193) doesn't
match the broadcast shape (721,106,193)
and
Traceback (most recent call last):
File "d:\plot_summarystats\test_plot_remove_memoryerror_max.py", line 45,
in <module>
if not instance(a, N.ndarray):
NameError: name 'instance' is not defined
On Wed, Dec 7, 2011 at 3:07 PM, Olivier Delalleau <shish at keba.be> wrote:
> I *think* it may work better if you replace the last 3 lines in your loop
> by:
>
> a=all_TSFC[0]
> if len(all_TSFC) > 1:
> N.maximum(a, TSFC, out=a)
>
> Not 100% sure that would work though, as I'm not entirely confident I
> understand your code.
>
>
> -=- Olivier
>
> 2011/12/6 questions anon <questions.anon at gmail.com>
>
>> Something fancier I think,
>> I am able to compare the result with my previous method so I can easily
>> see I am doing something wrong.
>> see code below:
>>
>>
>> all_TSFC=[]
>> for (path, dirs, files) in os.walk(MainFolder):
>> for dir in dirs:
>> print dir
>> path=path+'/'
>> for ncfile in files:
>> if ncfile[-3:]=='.nc':
>> print "dealing with ncfiles:", ncfile
>> ncfile=os.path.join(path,ncfile)
>> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>> TSFC=ncfile.variables['T_SFC'][:]
>> fillvalue=ncfile.variables['T_SFC']._FillValue
>> TSFC=MA.masked_values(TSFC, fillvalue)
>> ncfile.close()
>> all_TSFC.append(TSFC)
>> a=TSFC[0]
>> for b in TSFC[1:]:
>> N.maximum(a,b,out=a)
>>
>> big_array=N.ma.concatenate(all_TSFC)
>> Max=big_array.max(axis=0)
>> print "max is", Max,"a is", a
>>
>>
>> On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <shish at keba.be> wrote:
>>
>>> Is 'a' a regular numpy array or something fancier?
>>>
>>>
>>> -=- Olivier
>>>
>>> 2011/12/6 questions anon <questions.anon at gmail.com>
>>>
>>>> thanks again my only problem though is that the out=a in the loop does
>>>> not seem to replace my a= outside the loop so my final a is whatever I
>>>> started with for a.
>>>> Not sure what I am doing wrong whether it is something with the loop or
>>>> with the command.
>>>>
>>>> On Wed, Dec 7, 2011 at 1:44 PM, <josef.pktd at gmail.com> wrote:
>>>>
>>>>> On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <shish at keba.be>
>>>>> wrote:
>>>>> > The "out=a" keyword will ensure your first array will keep being
>>>>> updated. So
>>>>> > you can do something like:
>>>>> >
>>>>> > a = my_list_of_arrays[0]
>>>>> > for b in my_list_of_arrays[1:]:
>>>>> > numpy.maximum(a, b, out=a)
>>>>>
>>>>> I didn't think of the out argument which makes it more efficient, but
>>>>> in my example I used Python's reduce which takes an iterable and not
>>>>> one huge array.
>>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>> >
>>>>> > -=- Olivier
>>>>> >
>>>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>>>> >>
>>>>> >> thanks for all of your help, that does look appropriate but I am
>>>>> not sure
>>>>> >> how to loop it over thousands of files.
>>>>> >> I need to keep the first array to compare with but replace any
>>>>> greater
>>>>> >> values as I loop through each array comparing back to the same
>>>>> array. does
>>>>> >> that make sense?
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <shish at keba.be>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Thanks, I didn't know you could specify the out array :)
>>>>> >>>
>>>>> >>> (to the OP: my initial suggestion, although probably not very
>>>>> efficient,
>>>>> >>> seems to work with 2D arrays too, so I have no idea why it didn't
>>>>> work for
>>>>> >>> you -- but Nathaniel's one seems to be the ideal one anyway).
>>>>> >>>
>>>>> >>> -=- Olivier
>>>>> >>>
>>>>> >>>
>>>>> >>> 2011/12/6 Nathaniel Smith <njs at pobox.com>
>>>>> >>>>
>>>>> >>>> I think you want
>>>>> >>>> np.maximum(a, b, out=a)
>>>>> >>>>
>>>>> >>>> - Nathaniel
>>>>> >>>>
>>>>> >>>> On Dec 6, 2011 9:04 PM, "questions anon" <
>>>>> questions.anon at gmail.com>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> thanks for responding Josef but that is not really what I am
>>>>> looking
>>>>> >>>>> for, I have a multidimensional array and if the next array has
>>>>> any values
>>>>> >>>>> greater than what is in my first array I want to replace them.
>>>>> The data are
>>>>> >>>>> contained in netcdf files.
>>>>> >>>>> I can achieve what I want if I combine all of my arrays using
>>>>> numpy
>>>>> >>>>> concatenate and then using the command numpy.max(myarray,
>>>>> axis=0) but
>>>>> >>>>> because I have so many arrays I end up with a memory error so I
>>>>> need to find
>>>>> >>>>> a way to get the maximum while looping.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.pktd at gmail.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <
>>>>> shish at keba.be>
>>>>> >>>>>> wrote:
>>>>> >>>>>> > It may not be the most efficient way to do this, but you can
>>>>> do:
>>>>> >>>>>> > mask = b > a
>>>>> >>>>>> > a[mask] = b[mask]
>>>>> >>>>>> >
>>>>> >>>>>> > -=- Olivier
>>>>> >>>>>> >
>>>>> >>>>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>>>> >>>>>> >>
>>>>> >>>>>> >> I would like to produce an array with the maximum values out
>>>>> of
>>>>> >>>>>> >> many
>>>>> >>>>>> >> (10000s) of arrays.
>>>>> >>>>>> >> I need to loop through many multidimentional arrays and if a
>>>>> value
>>>>> >>>>>> >> is
>>>>> >>>>>> >> larger (in the same place as the previous array) then I
>>>>> would like
>>>>> >>>>>> >> that
>>>>> >>>>>> >> value to replace it.
>>>>> >>>>>> >>
>>>>> >>>>>> >> e.g.
>>>>> >>>>>> >> a=[1,1,2,2
>>>>> >>>>>> >> 11,2,2
>>>>> >>>>>> >> 1,1,2,2]
>>>>> >>>>>> >> b=[1,1,3,2
>>>>> >>>>>> >> 2,1,0,0
>>>>> >>>>>> >> 1,1,2,0]
>>>>> >>>>>> >>
>>>>> >>>>>> >> where b>a replace with value in b, so the new a should be :
>>>>> >>>>>> >>
>>>>> >>>>>> >> a=[1,1,3,2]
>>>>> >>>>>> >> 2,1,2,2
>>>>> >>>>>> >> 1,1,2,2]
>>>>> >>>>>> >>
>>>>> >>>>>> >> and then keep looping through many arrays and replace
>>>>> whenever
>>>>> >>>>>> >> value is
>>>>> >>>>>> >> larger.
>>>>> >>>>>> >>
>>>>> >>>>>> >> I have tried numpy.putmask but that results in
>>>>> >>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not
>>>>> list
>>>>> >>>>>> >> Any other ideas? Thanks
>>>>> >>>>>>
>>>>> >>>>>> if I understand correctly it's a minimum.reduce
>>>>> >>>>>>
>>>>> >>>>>> numpy
>>>>> >>>>>>
>>>>> >>>>>> >>> a = np.concatenate((np.arange(5)[::-1],
>>>>> >>>>>> >>> np.arange(5)))*np.ones((4,3,1))
>>>>> >>>>>> >>> np.minimum.reduce(a, axis=2)
>>>>> >>>>>> array([[ 0., 0., 0.],
>>>>> >>>>>> [ 0., 0., 0.],
>>>>> >>>>>> [ 0., 0., 0.],
>>>>> >>>>>> [ 0., 0., 0.]])
>>>>> >>>>>> >>> a.T.shape
>>>>> >>>>>> (10, 3, 4)
>>>>> >>>>>>
>>>>> >>>>>> python with iterable
>>>>> >>>>>>
>>>>> >>>>>> >>> reduce(np.maximum, a.T)
>>>>> >>>>>> array([[ 4., 4., 4., 4.],
>>>>> >>>>>> [ 4., 4., 4., 4.],
>>>>> >>>>>> [ 4., 4., 4., 4.]])
>>>>> >>>>>> >>> reduce(np.minimum, a.T)
>>>>> >>>>>> array([[ 0., 0., 0., 0.],
>>>>> >>>>>> [ 0., 0., 0., 0.],
>>>>> >>>>>> [ 0., 0., 0., 0.]])
>>>>> >>>>>>
>>>>> >>>>>> Josef
>>>>> >>>>>>
>>>>> >>>>>> >>
>>>>> >>>>>> >> _______________________________________________
>>>>> >>>>>> >> NumPy-Discussion mailing list
>>>>> >>>>>> >> NumPy-Discussion at scipy.org
>>>>> >>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>>>> >>
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> > _______________________________________________
>>>>> >>>>>> > NumPy-Discussion mailing list
>>>>> >>>>>> > NumPy-Discussion at scipy.org
>>>>> >>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>>>> >
>>>>> >>>>>> _______________________________________________
>>>>> >>>>>> NumPy-Discussion mailing list
>>>>> >>>>>> NumPy-Discussion at scipy.org
>>>>> >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> _______________________________________________
>>>>> >>>>> NumPy-Discussion mailing list
>>>>> >>>>> NumPy-Discussion at scipy.org
>>>>> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>> _______________________________________________
>>>>> >>>> NumPy-Discussion mailing list
>>>>> >>>> NumPy-Discussion at scipy.org
>>>>> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> NumPy-Discussion mailing list
>>>>> >>> NumPy-Discussion at scipy.org
>>>>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> NumPy-Discussion mailing list
>>>>> >> NumPy-Discussion at scipy.org
>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >>
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > NumPy-Discussion mailing list
>>>>> > NumPy-Discussion at scipy.org
>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>> >
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111207/a0b440c8/attachment.html>
More information about the NumPy-Discussion
mailing list