[Borgbackup] Reducing backup times for raw VM images, with zfs send?

[Bzzzz]

On Thu, 19 May 2022 11:07:39 -0700
Mason Schmitt <mason at ftlcomputing.com> wrote:

OOPS, sorry for the PM (who's the creep that changed MLs policy, which
fails miserably each time there's more than one level in a thread !#@^)

twice! :(((

> > > My experience with backing up KVM sparse disk image files is that
> > > even if there have been very few changes to the VMs (mostly just
> > > writes to files in /var/log) backups still take several hours to a
> > > locally attached USB 3 disk.
> >
> > I doubt it is USB3 fault, as a SSD on a Rpi4 read speed is an
> > average of 235 MB/s.
> >
> > Did you change the --chunker-params or did you left it genuine ?
> >
>
> I didn't change the chunker-params.  Is there an optimum setting for
> backing up sparse, raw, VM images?  Is the optimal setting different
> between 1.1.x and 1.2.x?

Not that I know of, but people backupetting *<;-p) VMs usually lower
minimal and maximal chunk sizes to get a maximum benefit of chunks
deduplication, as a change in a large chunk means this chunk must be
uploaded into the BB repo.

See :

https://borgbackup.readthedocs.io/en/stable/usage/notes.html#chunker-params

https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html?highlight=chunker%20params

https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=chunker%20params

Be careful, as changing chunk size(s) needs a brand new repo, which is
logical, you can't compare chunks if their sizes are different.

Also note (2nd link I think) that you can fix the chunk size AND that
lowering chunk size requires more memory and ressources.

Sooo, it'll take some time to find the good chunk sizes for your case
- may be you should open another thread, asking people who backup VMs
which values they are using.

At home, I just have 2 VMs (virtualbox: w$7:50GB & w$XP:11GB)
--chunker-params 16,23,16,4095 works better than the default (faster
backups).

In my notes, there's a pro guy who have multiple VMs at work using :
--chunker-params 16,23,11,4095 which seems to suit quite well his needs.

> I'm still running 1.1.17, which is the latest version on EPEL for
> CentOS 7.

I use the pip3 1.1.17 version on debians bulleye.

>  However, given that 1.2 has better sparse file support and
> an improved chunker, I'm wondering whether it might be worth stepping
> outside EPEL and instead using pip.  Have you noticed a big
> performance improvement for large sparse files, like VM images, with
> 1.2?

First rule of computing : never touch it when it's working.
Second  "  "      "     : always let others deal with teething problems.

So, I can't tell about 1.2 as I do not plan to flip to this version
until the end of this year.

Jean-Yves