[Tutor] Memory error - how to manage large data sets?

[Daniel Sarmiento]

Hi

I tried to run your code and checked (with top) the memory ussage and
it uses more than 2 Gb of memory.

I tried to modify the code a little bit to use less memory and came up
with this:

fib = {0:0,1:1}
even = []

def fibonacci(x,y):
   return x+y

for j in xrange (2,1000000):
    i = fib[j-1] + fib[j-2]
    if i % 2 == 0:
        even.append(i)
    fib = {j-1:fib[j-1], j:i}

total = reduce(fibonacci,even)
print total

First, I replaced range with xrange.
I figured that you only need the last two values in the fibonnaci
series to calculate the next value, so I replaced the fib list with a
dictionary to only store the last two values instead of the whole
series.

It looks like the progam still hangs and I did not notice any memory
imrovements when running it with 1 000 000

Am I wrong thinking that the modifications I made help use less memory?

> Date: Mon, 28 Jul 2008 22:44:08 +0530
> From: "Karthik" <rs.karthick at gmail.com>
> Subject: Re: [Tutor] Memory error - how to manage large data sets?
> To: <tutor at python.org>
> Message-ID: <488dfe6a.04506e0a.40cd.23a1 at mx.google.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Forgot to include the following information,
>
>
>
> Platform - win32
>
> Version - 2.5.1
>
>
>
> Error message:
>
>
>
> Traceback (most recent call last):
>
>  File "C:\Python25\programs\fibo.py", line 10, in <module>
>
>    if i % 2 == 0:
>
> MemoryError
>
>
>
> Code:
>
>
>
> fib = []
>
> even = []
>
> def fibonacci(x,y):
>
>    return x+y
>
> for i in range (0,1000000):
>
>    if i < 2:
>
>        fib.append(i)
>
>    else:
>
>        i = fib[i-1] + fib[i-2]
>
>        if i % 2 == 0:
>
>            fib.append(i)
>
>            even.append(i)
>
>        else:
>
>            fib.append(i)
>
> total = reduce(fibonacci,even)
>
> print total
>
>
>
> Any pointers would be of great help to me.
>
>
>
> Regards,
>
> Karthik