[Tutor] numerical simulation + SQLite
Alan Gauld
alan.gauld at btinternet.com
Mon Nov 30 23:05:50 CET 2009
"Faisal Moledina" <faisal.moledina at gmail.com> wrote
.
> At first, I used a list to store a bunch of class instances, each of
> which contained a bunch of data calculated at each time step. This
> resulted in whopping memory usage (2.75 GB RAM, 3.75 GB VM).
>
> So then I decided instead to use SQLite to store that information at
> each timestep. This seems to work well, but it gets slow over time as
> well.
You may need to be realistic in your expectations.
A database is writing to disk which will be slower than working in
memory. And a 3GB file takes a while to read/traverse, even with
indexes. It depends a lot on exactly what you are doing. If its mainly
writing it should not be much slower than writing to a flat file. If you
are doing a lot of reading - and you have used indexes - then it
should be a lot faster than a file.
But RAM - if you have enough - will always be fastest, by about 100 times.
The problem is when you run out, you revert to using files and that's
usually slower than a database...
But without details of your usage pattern and database schema
and SQL code etc it is, as you say, impossible to be specific.
--
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/
More information about the Tutor
mailing list