text file vs. cPickle vs sqlite a design question

Dag f98dawa at dd.chalmers.se
Wed Apr 11 12:40:02 EDT 2007


I have an application which works with lists of tuples of the form
(id_nr,'text','more text',1 or 0).  I'll have maybe 20-50 or so of these 
lists containing anywhere from 3 to over 30000 tuples.  The actions I
need to do is either append a new tuple to the end of the list, display 
all the tuples or display all the tuples where the last element is a 1

Basically what I'm wondering is the best way to store these data stuctures 
to disc.  As the subject mentioned I've basically got three approaches.
Store each list as a text file, pickle each list to file or shove the
whole thing into a bunch of database tables.  I can see pros and cons
with each approach.  Does anybody have any advice as to whether any of
these approaches is obviously better than any other?  On one hand I like
the text file approach since it lets me append without loading
everything into memory, on the other hand the sqlite approach makes it
easy to select stuff with SELECT * FROM foo WHERE... which could be
handy if ever need to add more advanced filtering.

Dag



More information about the Python-list mailing list