[Borgbackup] Deduplication of tar files - doesn't seem to be giving good performance

[William Gogan]

Thomas Waldmann wrote:
>> Background: Our VM tool kicks out a .tar file per container. It
>> compresses (lzo) the .tar. For discussion purposes, let's pretend it's
>> called vm.tar.lzo
>
> Please provide a tar listing so we can see how many / how big files 
> are in there. Without that, one can only speculate...
I can't give you a listing, but I can tell you this, which should help: 
This tar is created (using the command below) against a brand-new Redhat 
OS install with no user data on it yet, and minimal services. It is 
approx 1GB, and is mostly small files of type OS. I apologize that this 
isn't exactly what you asked for, but I'm not permitted to give a 
specific listing of data due to some work policies, even though this is 
just a blank install.
>
> Also, the specific format of the "vm disk file(s)" in here would be 
> interesting.
The VM 'disk file' is actually just a straight tar file, created with 
the following process:
1) LVM snapshot is taken
2) Tar is created against the snapshot using `tar cpf - --totals 
--sparse --numeric-owner --acls --xattrs --xattrs-include=user.* 
--xattrs-include=security.capability --warning=no-xattr-write 
--one-file-system --warning=no-file-ignored 
--directory=/storage/dump/vzdump-lxc-111-2016_04_21-10_39_19.tmp 
./etc/vzdump/pct.conf --directory=/mnt/vzsnap0 --no-anchored 
--exclude=lost+found --anchored --exclude=./var/log/?* 
--exclude=./tmp/?* --exclude=./var/tmp/?* --exclude=./var/run/?*.pid ./`
3) That tar file is compressed using lzo.

As previously mentioned, I lzop -d the file passing it to borg.

Pretty much the only benefit this .tar file gives me, vs pointing borg 
against the mounted LVM snapshot itself, is that should a disaster 
occur, the recovery process relies on providing the VM server with the 
.tar file of each VM.

A potential workaround to this would be to have borg work on the LVM 
mount itself, and then, during the restore process, I *might* (subject 
to testing) be able to run this tar command against the borg restore, in 
order to re-create the .tar file expected 'on demand' that can be 
consumed by the VM server. This feels a little wiggly, but I'll do some 
checking if all else fails.


-- 
William Gogan
Convey Studio / Custom. Digital. Branding.
719.278.3736
conveystudio.com <http://www.conveystudio.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20160421/66abedb3/attachment.html>