Dictionaries as records

[Skip Montanaro]

    Bill> I have a file with 200K records and 16 fields.  This file is
    Bill> parsed and each row is put into a dictionary and the dictionary is
    Bill> added to a list.  The raw file is only about 50mb.

    Bill> I was shocked to see that my memory use jumped to 500MB!  When I
    Bill> delete the list the memory is returned to the system, so I know
    Bill> that the memory is being used in the dictionaries.

    ...

    Bill> Can someone who has faced this issue and found a workaround please
    Bill> fill me in. I know one can use a list of lists or a list of
    Bill> tuples, but had rather stick to the dictionaries because of some
    Bill> library issues.

You might want to consider storing it in an on-disk mapping object.  Check
out the anydbm, bsddb, gdbm modules and the like.  Aside from not chewing up
gobs of RAM, script startup should be faster as well, because you won't have
to parse the file and initialize the dict.

-- 
Skip Montanaro (skip at pobox.com - http://www.mojam.com/)