text file vs. cPickle vs sqlite a design question

[Paddy]

On Apr 11, 5:40 pm, Dag <f98d... at dd.chalmers.se> wrote:
> I have an application which works with lists of tuples of the form
> (id_nr,'text','more text',1 or 0).  I'll have maybe 20-50 or so of these
> lists containing anywhere from 3 to over 30000 tuples.  The actions I
> need to do is either append a new tuple to the end of the list, display
> all the tuples or display all the tuples where the last element is a 1
>
> Basically what I'm wondering is the best way to store these data stuctures
> to disc.  As the subject mentioned I've basically got three approaches.
> Store each list as a text file, pickle each list to file or shove the
> whole thing into a bunch of database tables.  I can see pros and cons
> with each approach.  Does anybody have any advice as to whether any of
> these approaches is obviously better than any other?  On one hand I like
> the text file approach since it lets me append without loading
> everything into memory, on the other hand the sqlite approach makes it
> easy to select stuff with SELECT * FROM foo WHERE... which could be
> handy if ever need to add more advanced filtering.
>
> Dag

If you have enough resources to keep all the lists comfortably in
memory, and you have enough disk space then I would save your data as
python text. Something like:

print "# <What the data is and how it is formatted>"
print "all_lists = []"
for i,l in enumerate(all_lists):
  print "all_lists.append( [  #", i
  for tpl in l:
    print " ", tpl, ","
  print " ])  #", i

You would then have your data saved in a format that could easily
be re-used by other programs at a later date, and that can be
examined in any text editor.

- Paddy.