[Numpy-discussion] Memory hungry reduce ops in Numpy

[Andreas Müller]

On 11/15/2011 07:03 PM, Gael Varoquaux wrote:
> On Tue, Nov 15, 2011 at 05:57:14PM +0000, Robert Kern wrote:
>> Actually, last time I suggested it, it was brought up that the online
>> algorithms can be worse numerically. I'll try to find the thread.
> Indeed, especially for smallish datasets where the memory overhead is not
> an issue. I think that this is a common situation for which there is not
> _one_ good algorithm to solve the problem. IMHO the best way forward is
> to propose several options to the user, either with several function, or
> with a keyword argument.
>
I thought it would be possible to do the "two pass algorithm"
(as wikipedia calls it) without copying the dataset.
I guess I didn't really think it through.
That should be possible doing a custom reduce operation, right?
I don't know in how far this would have numerical problems.