[Borgbackup] A small compression test

[Bzzzz]

On Wed, 8 Mar 2023 13:03:59 +0800
William Kenworthy <billk at iinet.net.au> wrote:

Thanks for the advices :)

Jean-Yves

>  From experience:
> 
>      1. borg repos on a network file system (moosefs in my case) can
> be very very slow
> 
>      2. Borg has to read a complete VM image before it can calculate 
> checksums - and if you store the VM's on a network filesystem it is
> time consuming just to read 500Mb of data in one image let alone
> process it and then have to go on to do a number of other images.
> 
>      3. Consider if you can avoid large VM's and use the OS files 
> natively on a filesystem/partition, or backup the inside of the VM 
> rather than the image - the borg algorithms skip files they see as
> not having changed from metadata (but does do a safety recheck after
> a certain number of runs - see docs).  VM's by their nature have to
> be read in their entirety every time to figure out what has changed,
> even if its just one byte of data in it.  I have found that reading a
> VM images` contents a much faster operation after the first time.  If
> (as in my case) both the VM's and the repos are on a network
> filesystem, you will need to carefully consider where the work
> (reading files and calculating checksums) is to be done - reading
> multiple VM and storage images of many hundreds of megabytes will
> take time and cant be avoided.  The good news is borg is still faster
> than most other backup systems even in this scenario.
> 
>      4. Consider paralleling as much as possible - running borgbackup
> on multiple hosts pushing into individual repos at the same time
> takes only a little longer than doing 1 backup. e.g. doing it
> serially is 1+1+1+1 etc., while parallel would be something like 1.5
> in total.  Note that in my case, this is also leveraging the internal
> parallelisation of moosefs running on a number of separate hosts.
> 
> ** I found I reached the limits of my moosefs filesystem storing
> decades of email, hundreds of thousands of photos, borg repos and
> other files which it did quite well until I went too far for my
> hardware :(  Moving millions of smaller files in to loopback mounted
> images solved that problem, at the expense of blowing out a 15 minute
> backup sequence to many hours.  Backing up the files by reading into
> the image made quite a large timesaving.
> 
> BillK