Reclaiming (lots of) memory

[Paul Rubin]

Nick Craig-Wood <nick at craig-wood.com> writes:
> Just for fun I converted your program to use anydbm.  It then took 135
> seconds to run (vs 12 for your original code), but only used 4 MB of
> memory top.  The values of the dbm are pickled hashes.  The keys could
> be too, but I decided to just use a tab seperated string...

But I think it used much more than 4 MB of memory if you count the
amount of system cache that held that dbm file.  Without the caching
there'd have had to be real disk seeks for all those dbm updates.  So
I think if you had a lot more data, enough that you couldn't keep the
dbm in cache any more, you'd get a horrible slowdown.  That's why you
may be better off with a sorting-based method, that only needs
sequential disk operations and not a lot of random seeks.