Upper memory limit

[Michael Hudson]

Siegfried Gonzi <siegfried.gonzi at kfunigraz.ac.at> writes:

> Is there an upper memory limit which Python cannot handle?

Your OS's.

> I have a calculation (external C function included with the help of
> SWIG; external Fortran function included with the help of F2PY; some pre
> processed Lisp files; a dozen of binary satellite data files). One run
> of the code takes about 1 hour. The execution time is not the problem!
> Memory is not a problem too, but: it consumes after 1 hour 200 MB of
> RAM. If I start the calculation in the DOS shell I can see that Python
> gives memory back to the OS (I got the advice to start the calculation
> from the DOS line), but this is only for 2 runs the case. If I start the
> calculation for a third time I have to see that Python wreaks havoc. A
> calculation would then take many, many hours due to the Python memory
> management and at the end the system is down and I would have to restart
> it (even on a newer laptop rebooting is not very quick and hence
> anoying). 

What variant of Windows are you running?  I believe Tim has a large
collection of ways to make 98's allocator (at least) go insane.

> I need some advice for my further strategy  (I am about to write my
> simulations in Fortran 90/95 and use Python only for simple plotting
> jobs):
> 
> a) Has anybody ever experienced the same problem?; I am not angry with
> the execution time I am sick of the poor memory management behavior.

Nope, but I've never slung the amounts of data you're slinging and
never on Windows.

> b) Is there semantically speaking a difference between automatic memory
> managament and garbage collection?

I wouldn't say so.

> c) Dose Python's garbage collector just simply collect and dispose all
> the memory junks at the end of the calculation (and hence once) or does
> Python collect garbage and at the same time removing old memory?

I'm not sure what you mean here (though I have this wonderfil image of
DIMMs flying out of a box in all directions...).

I suspect you need to ask whoever wrote the allocator you're using.
Python will free() (unless you're using pymalloc...) any objects it
collects.

At any rate, if the OS can't claim back a processes memory when said
process terminates, it's definitely Time To Get A Real OS.  Even Macs
can do this now...

[...]
> d) Has anybody experience with Python on Linux or free BSD?

Yes.

> I would be ready to install Linux in order to get a better Python
> behavior.

I doubt you'd get this level of bogosity.

Summary: if you're on 95/98(/Me?), get off, right now.

Cheers,
M.

-- 
  ARTHUR:  Why should he want to know where his towel is?
    FORD:  Everybody should know where his towel is.
  ARTHUR:  I think your head's come undone.
                    -- The Hitch-Hikers Guide to the Galaxy, Episode 7