[Borgbackup] Reducing backup times for raw VM images, with zfs send?

[Thorsten Schöning]

Guten Tag Heiko Helmle,
am Freitag, 20. Mai 2022 um 07:48 schrieben Sie:

> Mind that it is very dependent on the format of the VM. I've found
> that the ProxMox backup format (VMA) (uncompressed) has nearly zero
> deduplication rate with default chunker settings[...]

I'm directly backing up running VMs in QCOW2 format after creating ZFS
snapshots. 950 GiB of those with 7 days history results in 1,5 TiB
accumulated archives, which is good enough for me currently to not
need to think about customized chunker settings.

That should be some worst case already in theory because most VMs use
BTRFS/ZFS with default compression already. I guess that makes dedup a
lot more difficult.

> Same experience for mongodb dumpfiles. Default chunker settings of
> mongodump's standard "multiple files" format result in almost no
> dedup, even if run in quick succession, while 16.23.16.4095 gives
> surprisingly good results. But mind - in that situation too, memory
> consumption was not a primary concern.

The most important thing seems to be to get a somewhat stable,
uncompressed dump output in the first place. I have some Postgres
database storing large binary files as large objects and Borg seems to
handle that pretty well using the custom, but uncompressed dump
format. It's only very few GiBs of changes per day for an otherwise
~200 GiB dump, again with default chunker settings.

Depending on the restore needs and things like temporary storage and
stuff, other formats like plain text or dumping into a temporary
archive using directory formats might make sense as well. Am using
plain text for small MySQL databases, seems to work pretty well, too.
Though, am trying to avoid additional copies into the file system by
piping dumps directly into named pipes read by Borg, so uncompressed,
one-file-based dumps are my favourite currently.

Mit freundlichen Grüßen

Thorsten Schöning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister für IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Geschäftsführer: Janine Galonska


Für Rückfragen stehe ich Ihnen jederzeit zur Verfügung. 

Mit freundlichen Grüßen, 

Thorsten Schöning


Telefon: +49 (0)515 94 68 - 0
Fax: 
E-Mail: TSchoening at am-soft.de

AM-Soft IT-Service - Bitstore Hameln GmbH
Brandenburger Straße 7c
31789 Hameln

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen und ist ausschliesslich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. 

This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. 

Hinweise zum Datenschutz: bitstore.group/datenschutz