[PYTHONMAC-SIG] memory usage

[Tasselli Marco]

Hi everybody,

I'm just trying to figure out how Python uses memory when loading
dictionaries.

I have a master CGI program that drives all other CGI requests.
This program loads (at startup time) 3 large dictionaries for later
access by many other CGI scripts.
This is a sort of 'in-memory' database.
Everything works fine and the performance of the system is exceptional
(every request require no more than 0.080 seconds!!).
This way the need for multithreading is a not so compelling one.

The problem, however, is with the scalability of the solution.
The 3 files I upload at startup are:
-10000 recs (350K)
-800 recs (80K)
-35000 recs (3500K)

This require a huge 23M of RAM occupied after the upload!
Obviously you cannot add many new files unless you have a very high
amount of free memory.

By the way, I tried the same upload using a MacPerl script and I
succeded in using only 6M of RAM!

Is there a way to keep memory consumption lower?

best regards
Tasselli Marco


_______________
PYTHONMAC-SIG  - SIG on Python for the Apple Macintosh

send messages to: pythonmac-sig@python.org
administrivia to: pythonmac-sig-request@python.org
_______________