reading large file

[Michael Peuser]

"guillaume" <g_alleon at yahoo.fr> schrieb im Newsbeitrag
news:9bd0dd3f.0309030400.7fa2dc0f at posting.google.com...
> I have to read and process a large ASCII file containing a mesh : a
> list of points and triangles.
> The file is 100 MBytes.
>
> I first tried to do it in memory but I think I am running out of
> memory therefore I decide to use the shelve
> module to store my points and elements on disks.
> Despite the fact it is slow ... Any hint ? I think I have the same
> memory problem but I don't understand why
> since  my aPoint should be removed by the gc.

What do you expect from shelve? I should recommend you convert your data in
afirst pass into a binary format (doing all this atoi() in this pre-pass)
Then use memory mapped file access when reading it for your work pass.

But maybe you need a lot of memory for your internal structure as well. If
youe have  a small RAM <512 MB the system could do a lot of swapping. You
will notice that when processor load goes down! The cheapest solution
generally is doubling your RAM.

Kindly
Michael P