[Borgbackup] No de-duplication on big VM raw-backups/images

[Heiko Helmle]

You're right.

Borg's default chunker params and proxmox are not working together at all.

I did some benchmarking when setting up our proxmox and we settled for 16,23,16,4095.

Results were similar - gaining from almost no dedup to almost full dedup.

The same values work pretty well for mongodump's output too!

Would it make sense to gather experience values with the chunker params somewhere (wiki?), so people could look up some good starting params for their specific workload?

The current default values seem to be optimized for vmdk and raw VM volume data by the looks of it.

Best Regards
 Heiko

-----Original Message-----
From: Borgbackup <borgbackup-bounces+heiko.helmle=horiba.com at python.org> On Behalf Of Mateusz Kijowski
Sent: Mittwoch, 19. Februar 2020 13:24
To: Stefan Bauer <sb at plzk.de>
Cc: borgbackup at python.org
Subject: Re: [Borgbackup] No de-duplication on big VM raw-backups/images

Hi, I have the same use-case,

see this thread https://mail.python.org/pipermail/borgbackup/2017q4/000940.html

TL;DR; version is: it's most likely the chunking parameters that prevent you from efficient dedupe. The chunker-params I currently use
are: 10,22,16,4095

Regards,

Mateusz

śr., 19 lut 2020 o 11:07 Stefan Bauer <sb at plzk.de> napisał(a):
>
> Hi,
>
>
> use case is sending proxmox backups to remote site for disaster-recovery.
>
>
> given is a single file in a directory:
>
>
> -rw-r--r-- 1 root root 5.5G Feb 19 10:36
> vzdump-qemu-202-2020_02_19-10_02_38.vma
>
>
> It's uncompressed. First borg backup with --compression zlib,6 sends ~ 2,7GB to remote site. That is good.
>
>
> Next day, we have two files in the directory. Almost no changes between the files.
>
>
> -rw-r--r-- 1 root root 5.5G Feb 19 10:36
> vzdump-qemu-202-2020_02_19-10_02_38.vma
>
> -rw-r--r-- 1 root root 5.5G Feb 20 10:36
> vzdump-qemu-202-2020_02_20-10_02_38.vma
>
>
> Now, borg sends _again_ the complete ~ 2,7GB of the new file to remote site. Why?
>
>
> I would expect that _at least_, not all again, will be transfered.
>
>
> Am i doing something wrong?
>
>
> Thank you.
>
>
> Stefan
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
_______________________________________________
Borgbackup mailing list
Borgbackup at python.org
https://mail.python.org/mailman/listinfo/borgbackup
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima. Junichi Tajika