[Borgbackup] Reducing backup times for raw VM images, with zfs send?

[Heiko Helmle]

> Guten Tag Heiko Helmle,
> am Freitag, 20. Mai 2022 um 07:48 schrieben Sie:
>
> > Mind that it is very dependent on the format of the VM. I've found
> > that the ProxMox backup format (VMA) (uncompressed) has nearly zero
> > deduplication rate with default chunker settings[...]
>
> I'm directly backing up running VMs in QCOW2 format after creating ZFS
> snapshots. 950 GiB of those with 7 days history results in 1,5 TiB accumulated
> archives, which is good enough for me currently to not need to think about
> customized chunker settings.

I've had good results with QCOW2 and VMDK, but not with VMA - and unfortunately, that's what Proxmox gives me in our configuration.

> The most important thing seems to be to get a somewhat stable,
> uncompressed dump output in the first place. I have some Postgres
> database storing large binary files as large objects and Borg seems to handle
> that pretty well using the custom, but uncompressed dump format. It's only
> very few GiBs of changes per day for an otherwise
> ~200 GiB dump, again with default chunker settings.

Same with MySQL. As the dump there is plain SQL text and usually always sorted the same, dumps tend to perform well in dedeup.

I have the feeling that both Proxmox VMA and mongodump have highly varying block headers that are too close together for the default minimum chunk size of borg, resulting in lots of unique chunks.
For mongodb the obvious solution is not using mongodump at all and instead taking the original files from a volume snapshot. This has lots of advantages with the slight disadvantage that it's less portable and harder to set up.

Cheers
 Heiko
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Ralph Lauxmann, Ergin Cansiz, Hiroyuki Urabe, Markus Bode, Dr. Robert Plank, Takashi Nagano, Dr. Hiroshi Nakamura, Tetsuhiro Habe.