[Tutor] Memory Management etc

[Kent Johnson]

Philip Smith wrote:
> Hi
>  
> I use Activestate Python (2.4.3) in a Windows 32 bit environment.
>  
> I have a problem with a large programme that uses a lot of memory 
> (numerous large arrays which are dynamically extended).  I keep getting 
> unpredictable crashes (on a machine with 1/2 GB memory) whereas on my 
> laptop (1 GB memory) it seems OK.

I'm way out of my expertise here but I'll give it a try...

Is your program pure Python or are you using C extensions (other than 
the ones in the standard library)? If it is pure Python I think this 
would be considered a Python bug and would interest the Python developers.

> The portion of the code which crashes varies a bit but curiously is 
> NEVER a part where any memory allocation is happening.

Can you post any code? If you can make a short test program that shows 
the problem that will dramatically increase your chances of getting 
useful help.

> I have to say I have noticed (the programme is basically a 
> batch-factoring programme for integers) that no matter how I tune gc I 
> can't get it to reliably free memory in between factoring each integer.

I don't think Python will ever release memory back to the OS. This has 
changed in Python 2.5, you might be interested in trying the alpha release:
http://docs.python.org/dev/whatsnew/section-other.html
http://www.python.org/download/releases/2.5/

>  
> Because this is a development programme (and for performance reasons) I 
> use global variables some of which refer to the large arrays.
>  
> My questions are:
>  
> 1)    Does the mere fact that a function cites a variable as global 
> create a reference which prevents it being collected by gc?

You mean a 'global' statement without actually using the named variable? 
I don't know if this creates a reference, but if so, the reference 
should go out of scope when the function exits.

> 5)    Does anyone have ANY suggestions please?

These links might be interesting:
http://tinyurl.com/pszzh
http://evanjones.ca/python-memory.html
http://pysizer.8325.org/

Asking on comp.lang.python will give you access to many more people 
familiar with Python internals than you will find on this list.
>  
> I'm keen to solve this because I would like to make my programme 
> generally available - in every other respect its near complete and 
> massively outperforms the only other comparable pure python module 
> (nzmath) which does the same job.
>  
> Thanks in anticipation.
>  
> Phil
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor