[Borgbackup] A small compression test

[Bzzzz]

On Tue, 7 Mar 2023 15:18:47 +0100
Thorsten Schöning <tschoening at am-soft.de> wrote:

> Guten Tag Bzzzz,
> am Dienstag, 7. März 2023 um 14:42 schrieben Sie:
> 
> > Normal : it is single threaded _and_ you have a lot more files to
> > scan, to compare to what's in the repo and, eventually, compress.
> 
> The only change I'm aware of was lz4 to zstd and that doesn't
> influence scan performance for changed files, that should be like
> before. It only influences CPU load and compression time of changed
> data.

It does, as you have more compressed files in a BB file, so checksums
are read faster than with lz4 because they're more concentrated.

> > I meant think about only add changed VM chunks to the repo[...]
> 
> The changes per day to the VM images are larger than the changes to
> the individually backed up files. So if X GiB are pretty fast for
> VM-images and database dumps, I'm wondering why (X-Y) GiB of data is
> that slow when backing up individual files. That doesn't make too much
> sense.

I reformulate to see if I understand correctly :
* VM images & DB dumps are many GB of changed data and backup fast,
* regular smaller files are not that often changed but backup slower.

If I have to make a guess, I'd say that if a very few readings on
either the client and the server, you have all what's needed for a
VM/DB, when for regular files, that might not dwell into the same BB
file and different areas on the HDD of the client, there's many more
head movements (hence latency), plus BB have to calculate many more
checksums when files are small than when they are made of big chunks.

Jean-Yves