[Borgbackup] lzma, 3-2-1 rule

[Boris Kirkorowicz]

Hi,

Am 02.04.23 um 19:19 schrieb Thorsten Schöning:
> Guten Tag Boris Kirkorowicz,
> am Freitag, 31. März 2023 um 11:36 schrieben Sie:
> 
>> This is new to me, maybe I overlooked something when choosing "-C
>> lzma,9".
> 
> You most likely missed the following warning:
> 
>> Giving levels above 6 is pointless and counterproductive because it
>> does not compress better due to the buffer size used by borg - but
>> it wastes lots of CPU cycles and RAM.

to explain how I got to chose lzma,9: I did some testings by my own with 
a sample dir tree (7,5 GB) to compare compression methods.

> Compression	Time		Repo Size	Ratio
> lz4		00:51:55	5,8 GB		77 %
> zlib,6	00:43:31	4,4 GB		59 %
> lzma,1	00:51:51	4,5 GB		60 %
> lzma,6	01:02:50	4,2 GB		56 %
> lzma,9	01:00:11	4,2 GB		56 %

In my sample, lzma,9 was slightly faster than lzma,6 without any other 
difference, and it compressed better than the other tested methods. 
Unfortunately, I did not test all compression methods, just these above.

Since I plan so transfer the repos to a remote site, I preferred the 
best compression ratio to save bandwidth. Additionally, I logged the 
processor load and found that the load is mostly at 60~80% with rare 
peaks to 100%. Since the server has nothing else to do during backup, 
this seems to me as a practicable setting.


> I don't think that reading LZMA will be removed too soon, using it for
> new data might be. OTOH, different compression methods can be mixed
> freely, so you don't need to care too much:

I need to save the data for at least 10 years, so it is important to be 
sure that I can read the backups after that long time. Every period 
below 10 years would be too soon for me. What do you think: does lzma 
survive this period, at least reading?


>> It is no problem to mix different compression methods in one
>> repo[...]
> 
> https://manpages.debian.org/testing/borgbackup/borg-compression.1.en.html
> 
> There was a thread recently about compression performance, regarding
> that you should definitely stop using LZMA for performance reasons.

OK, I'll complete my testings with the other compression methods to pick 
another one.


>> Second question: the article says that it is no good idea to
>> fulfill the 3-2-1 rule by copying the borg repositories, since
>> errors might be copied. This sounds plausible, but on the other hand
>> it would multiply the time of backup. In my case it could take more
>> than one year, so I wonder how to do this. Any advice how to handle this?
> 
> Don't waste backup time with LZMA anymore and run multiple individual
> backups to different repos one after another. Besides that and
> depending on the performance of your source system, you might run
> multiple BORG processes concurrently as long as they use different
> target repos. If I remember correctly you discussed that for your
> initial backup yourself already.

That's what I do: I split the data into reasonable trees and invoke borg 
for each tree in parallel. So every borg process runs in e separate 
core. I did my very best, but three of these trees remain that large 
that it takes so much time. I'll have a second look at it, but I don't 
expect to find big improvements.


-- 
Mit freundlichem Gruß                                 Best regards
                                  Kirkorowicz