SQLite or files?

[Tim Chase]

> info was stored in the simple RTF files.
> 
> However now I'd like to rewrite this program in Python (using PyQt) as
> I want to make it cross-platform and add/remove some features. Now I'm
> thinking about where to store my information. Would it be better to
> use files as I used to do or use the database, SQLite in particular?
> What will be faster and more flexible in the long run? This
> application will be in the memory most of the time so I'm also
> concerned about memory usage.

Not knowing what you do with the files, nor what sort of data 
they contain makes it a bit difficult to make suggestions in 
light of the context.  However:

- sqlite will let you perform arbitrary queries against your 
data, so if you want to aggregate or perform complex conditional 
tests, it's just SQL.

- I don't know if you're currently keeping the RTF in memory the 
whole time, or if you repeatedly reload (whether in one go, or 
streaming it) and reparse the file.  This sounds memory and/or 
processor intensive.  Using sqlite, the processing is done at the 
C-module level, the data is kept on disk and only brought into 
memory as-requested, being released when you're done with it.

- concurrently sharing a sqlite database should have minimal 
issues.  Sharing RTF files concurrently means locking/contention 
issues.  May not be an issue for you.

- sqlite comes built-in with Python2.5+ while RTF processing is 
not batteries-included from what I can tell[1]

So in the general case, I see sqlite being a notable win over 
RTF.  Depending on your data, if it's just key/value pairs of 
strings, you might look into the anydbm module[2] which has an 
even simpler interface.

-tkc

[1]
http://pyrtf.sf.net

[2]
http://docs.python.org/library/anydbm.html