[Borgbackup] Reducing backup times for raw VM images, with zfs send?

[Heiko Helmle]

> -----Original Message-----
> > Sooo, it'll take some time to find the good chunk sizes for your case
> > - may be you should open another thread, asking people who backup VMs
> > which values they are using.
>
> I'm using Borg with default settings to backup multiple differently sized VMs,
> with different usage like file hosting, SVN, embedded databases, BTRFS
> snapshots etc. Overall size is around 950 GiB of data and backup size is
> between 2 and 3,5 hours mostly. Things heavily depend on changes, load on
> the backup target NAS etc. I'm reading from SSDs, forwarding thourhg
> unkonw LAN speed using SSH to some NAS-like box with not guaranteed
> speed.

Mind that it is very dependent on the format of the VM. I've found that the ProxMox backup format (VMA) (uncompressed) has nearly zero deduplication rate with default chunker settings, even when backing up the same VM twice without having it powered on in the meantime.
But buzhash,16,23,16,4095 results in extremely good deduplication for the same workload. It's still not fast though but I'm not sure if different chunker settings would change speed much - I always assume the biggest tradeoff is memory.

Same experience for mongodb dumpfiles. Default chunker settings of mongodump's standard "multiple files" format result in almost no dedup, even if run in quick succession, while 16.23.16.4095 gives surprisingly good results. But mind - in that situation too, memory consumption was not a primary concern.

Cheers
 Heiko
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Ralph Lauxmann, Ergin Cansiz, Hiroyuki Urabe, Markus Bode, Dr. Robert Plank, Takashi Nagano, Dr. Hiroshi Nakamura, Tetsuhiro Habe.