increasing the page size of a dbm store?

[Tim Chase]

On 2019-12-02 16:49, Michael Torrie wrote:
> On 12/1/19 7:50 PM, Tim Chase wrote:
> > After sparring with it a while, I tweaked the existing job so
> > that it chunked things into dbm-appropriate sizes to limp
> > through; for the subsequent job (where I would have used dbm
> > again) I went ahead and switched to sqlite and had no further
> > issues.  
> 
> How did you replace a key/value store with a relational database?
> Is a SQLite database fast enough at this sort of thing that it
> wasn't really designed for?

It was certainly slower, though it wasn't so bad once I had proper
indexing and submitted queries that pulled back multiple results in
one query.

But even with the slightly slower run-time aspects, it was still
faster than starting a job (expecting it to run to completion
overnight), having it crash, manually deleting my cache, and manually
resuming from where it left off, all multiple times.

And all said, since it was network I/O bound, once I had the
populated cache (resulting cache.db file was about 1TB...thank
goodness for transparent compression with ZFS), turnarounds took more
like 30min rather than 3 days.  More the "go work on something else
and come back" than the "let it run overnight".

-tkc