[Borgbackup] use of borgbackup in fi-backup to save VM images

[Chiarello Ernest]

bonjour Thorsten, hi everyone ;

Le 04/09/2021 à 11:20, Thorsten Schöning a écrit :
> Guten Tag Ernest CHIARELLO,
> am Freitag, 3. September 2021 um 15:32 schrieben Sie:
>
>> my question is: would it be possible to replace the "cp -u" by
>> "borg create" (from /var/lib/libvirt/images to
>> /var/lib/libvirt/backups), in order to benefit from the
>> deduplication and to shorten the copy times ?
> No, because you are mixing two different topics. If you want to backup
> the individual files created by "fi-backup", then of course you can
> use Borg to do so. Simply because Borg doesn't care too much about the
> purpose of the files you are backing up.
>
> But is that logically the same as letting "fi-backup" copy its own
> files, create larger images and back those up? No. It's different. Is
> it what you need/want? Don't know, that's something only you can
> decide on your own.
>
> Anyway, the result of using "borg create" in different steps won't
> ever be the same as some "cp -u", because on one case you have content
> added to a Borg repo always, while in the other case you have a
> completely different file format maintained by "fi-backup". Most
> likely QCOW2 if I understand the docs correctly.
>
>> the creation of dummy files would then also be managed by fi-backup.
> If those are necessary at all, in my opinion those dummy files a an
> imlpementation detail for Borg and should therefore be managed by Borg
> or, as suggested, BorgMatic instead. "fi-backup" doesn't know or need
> to care about what Borg consideres changed or unchanged, when it does
> read which files again for wehatever reason etc.
OK
>> i suppose you can not save the qcow2 file of a running VM. you must
>> stop it before.
> That totally depends on the content of the image. A lot of operating
> systems, file systems and even apps like databases or else can handle
> and survive a crashing host pretty well. just think about your own
> Windows/Linux computer you use for your daily work or privately, they
> crashed most likely at some point in the past. With whcih results,
> completely data loss, new hardware necessary? Most likely not, only
> some unsaved changes might have been lost, some FSCK/CHECKDISK being
> executed and afterwards you were good to go again.
>
> That's not much different with your VMs most likely and allows you to
> backup running VMs as well. The important thing is to freeze the
> running VM for how long the backup takes and you have multiple ways to
> do so: Suspending the VM or something like file system level snapshots
> of e.g. BTRFS and ZFS.

i know zfs is used by ProxMox, as you can read here : ZFS on proxmox 
<https://pve.proxmox.com/wiki/ZFS_on_Linux>

But btrfs is not recommended according 
tohttp://www.linux-kvm.org/page/Tuning_KVM 
<http://www.linux-kvm.org/page/Tuning_KVM>

/Don't use the linux filesystem btrfs on the host for the image files. 
It will result in low IO performance. The kvm guest may even freeze when 
high IO traffic is done on the guest. /

> In my opinion, most people these days prefer the latter: Putting VMs
> into BTRFS/ZFS, create a snapshot, mount that snapshots and let
> BorgBackup directly backup that snapshot. This doesn't interrupt the
> oepration of the VM, while at the same time what you get is a so
> called "crash consistent" backup. That is enough for many needs, but
> of course you need to decide that on your own.
>
> And here's the point: When doing that approach, while BorgBackup reads
> the whole image, in that use-case it directly applies de-duplication,
> compression etc. and really only writes the changed data to its own
> repo. If that is faster or not than what "fi-backup" does in two steps
> is something you need to test on your own. Though, without knowing
> "fi-backup", in my opinion having only one step is easier in the end.
> Remember that each run of "borg create" will provide you a working VM,
> without additional steps to copy things around, merge QCOW2 images
> etc.
thank you for this excellent understanding of the situation and for your 
good advice! 😁
> Additionally have a look at what "fi-backup" writes on its own:
>
>> By default fi-backup.sh guarantees that qcow2 images format is
>> consistent, but this doesn't imply that the contained filesystems
>> are consistent too. In order to perform consistent backups, you can
>> use two different strategies:
> https://github.com/dguerri/LibVirtKvm-scripts#note
>
> So, depending on your setup, you might already backup only "crash
> consistent" state anyway. Additionally, even with the guest agend of
> QEMU installed, individual apps like databases won't know anything
> about that "fi-backup" is doing some backup "right now" and therefore
> won't do anything to make their own files consistent. The only thing
> the agent changes is that "fi-backup" is able to tell the guest-VM to
> e.g. sync pending file system changes on disk before suspending. that
> DOES NOT guarantee any consistent file format state in the VM, though.

you are quite right to recall that.

this is the reason why I use Urbackup <https://www.urbackup.org/> to 
back up data contained in VMs (files and databases).

thank you for this interesting discussion,

bonne journée,

Ernest.

> Mit freundlichen Grüßen
>
> Thorsten Schöning
>

-- 
Ernest CHIARELLO -Ernest.Chiarello at univ-fcomte.fr

UMR6049 ThéMA - CNRS / Université de Franche-Comté
Service de gestion et production de l'information
32 rue Mégevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

USR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210907/34f741db/attachment.html>