Is it more CPU-efficient to read/write config file or read/write sqlite database?

[Cameron Simpson]

On 15Dec2013 18:07, Tim Chase <python.list at tim.thechases.com> wrote:
> >   + only lets one process access the db at a time, taking you back
> > to a similar situation as with config files
> 
> Is this a Python limitation?  According to the docs[1], it's not a
> sqlite limitation (except, as noted, on non-locking filesystems like
> NFS)

I stand corrected. I may have been mislead by apps that keep the
write lock excessively.

> >   + only lets you access the db from the same thread in which it
> >     was opened, outstandingly annoying; I've had to gratuitously
> >     refactor code because of this
> 
> I do believe that limitation does hold though depending on the
> build-options with which sqlite was compiled [2], though it might be
> somewhat different from within Python where the GIL could
> possibly prevent actual OS-level-thread issues.

The GIL might allow this in principle. In practice, I've had to
jump through hoops in Python to force all db ops to happen in a
particular thread.

> >   + traditionally, sqlite is extreme fsync() happy; forces a disc
> >     level flush on each commit - extremely slow on busy databases,
> >     not to mention hard of drives
> 
> I'd say this is the right thing for a DB to do.  If it comes back
> from a commit() call, it better be on that disk, barring a failure
> of the physical hardware.  If it comes back from a commit() and data
> gets lost because of a power-failure, something is wrong.

Depends on your view. People seem to treat dbs as some special form
of data storage. I don't; to me they're no different to storing
data in any other file. Do you do an fsync() every time you close
a file you've written? Of course not, it is a gratuitous performance
loss.  IMO, I've handed the data to the filesystem layer; its
integrity is now the OS's problem.

Now, if you _want_ to fsync on every commit, be my guest. It could
even be the default behaviour. But FFS let me turn it off easily.

The situation is made worse by fsync() implementations that are
equivalent to whole-filesystem sync(). (I'm looking at you, Linux
ext3!)

> > > * well, except on NFS shares and other places where file-locking
> > > is unreliable
> > 
> > Backing off to config files, making a lock directory is NFS safe.
> > So is opening a lock file for write with zero permissions (low level
> > open with mode=0).
> 
> Interesting.  I haven't used NFS in a long time for anything other
> than quick experiments, so it's nice to file this away.  Do you have
> a link to some official docs corroborating what you state?

Only that I'm fairly sure mkdir and open are NFS primitives. They
take place atomicly at the remote end. You can't mkdir something
twice (EEXIST) and an open with mode=0 creates an unwritable file.
A subsequent open-for-write-with=mode=0 will fail, lacking write
permission.
-- 
Cameron Simpson <cs at zip.com.au>

Knox's box is a 286.                 Fox in Socks does hacks and tricks
Knox's box is hard to fix.           To fix poor Knox's box for kicks.
        - David Mar <mar at physics.su.oz.au>,
          as quoted by John Mackin <john at civil.su.oz.au>