optimizing memory usage of large in memory structures

[Steven Taschuk]

Quoth Hugo Liu:
> working with corpus analysis and large knowledge bases, i often have
> to load large arrays into memory.  each array element is just a text
> string.  although the text is only 5mb, python consumes 40+ mb.  i
> would suspect that the overhead is from storing type info for each
> element.  [...]

Have you tested this?  For example, if you replace your list of
elements with a same-sized list of Nones, how much memory is
consumed?  How much memory is consumed if you load the 5MB file as
one big string?

-- 
Steven Taschuk                                                 o- @
staschuk at telusplanet.net                                      7O   )
                                                               "  (