Is it more CPU-efficient to read/write config file or read/write sqlite database?

[Cameron Simpson]

On 14Dec2013 10:15, Tim Chase <python.list at tim.thechases.com> wrote:
> On 2013-12-14 07:29, JL wrote:
> > I have a number of python processes which communicate with each
> > other through writing/reading config text files. The python
> > ConfigParser is used. I am wondering if it is more CPU-efficient to
> > switch to using sqlite database instead of using configuration
> > files. If the software does plenty of reading/writing, is it more
> > efficient to use config text files or sqlite database?
> 
> I'm pretty sure that the CPU aspect doesn't really play into things.
> A few thoughts:
> 
> + You'll be I/O bound most of the time.  Even if you used a ramdisk
>   to reduce disk access delays, accessing multiple .txt files requires
>   the OS to do permission-checking each time, while a single sqlite
>   file gets checked once upon opening the DB initially.
> + text-files are fragile unless you take extra pains to keep things
>   atomic
> + sqlite guarantee* atomicity, so you either see all-or-nothing
> + sqlite is also very efficient for querying

Annoyingly, sqlite:

  + only lets one process access the db at a time, taking you back to
    a similar situation as with config files

  + only lets you access the db from the same thread in which it
    was opened, outstandingly annoying; I've had to gratuitously
    refactor code because of this

  + traditionally, sqlite is extreme fsync() happy; forces a disc
    level flush on each commit - extremely slow on busy databases,
    not to mention hard of drives

> + sticking with plain-text config files is just asking for some sort
>   of race-condition or partial-file issue to come up

A locking mechanism will be required. Lockfiles work quite well.

> + sqlite may give you less CPU load is just an added benefit
> 
> * well, except on NFS shares and other places where file-locking is
>   unreliable

Backing off to config files, making a lock directory is NFS safe.
So is opening a lock file for write with zero permissions (low level
open with mode=0).

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

I drank what?!? - Socrates