[borgbackup] Borg speed tuning on large files

[Alex Gorbachev]

Hi Thomas,

Here are my test results for a large area with ~2TB database backups per
day:

Tool Parameters Data size (apparent) Repo size Hrs Ratio C Rat C MB/s gzip
c3 2308843696 560376600 22 24% 4.1 7 Attic First Run default 2251760621
531964928 48 24% 4.2 3 Attic Next Run default 2308843696 234398336 32 10%
9.9 2 Borg First Run C0,19,23,21,4095 2330579192 2354907008 26 101% 1 25 Borg
Next Run C0,19,23,21,4095 2270686256 1341393408 18 59% 1.7 21 Borg First Run
C3,19,23,21,4095 2270686256 568351360 33 25% 4 5 Borg Next Run
C3,19,23,21,4095 2268472600 302165632 23 13% 7.5 4 Borg Next Run
C1,19,23,21,4095 2247244128 422037120 24 19% 5.3 5
Here is a picture in case the text does not come through well:

[image: Inline image 1]

Oddly, compression setting of 1 took longer than C3.  C0 shows the actual
dedup capability of this data.  My business goal here is to get the data in
within a day, so about 12 hours or so.

Best regards,
Alex




On Tue, Aug 25, 2015 at 6:45 PM, Thomas Waldmann <tw at waldmann-edv.de> wrote:

> > No encryption.  Tested with compression and without - observed the
> > behavior you described below, but the volume of data pretty much
> > requires compression to properly function.  I tried compression levels
> > of 0 and 3.
>
> Maybe try 1 until lz4 is available. That's relatively fast and still
> compresses.
>
> > Completely understood, and I am assuming starting multiple borg
> > processes in parallel for each file is not a good idea?
>
> You can run up to N borg in parallel (if N is your cpu core count), but
> only if the target repo of each is a different one, otherwise they will
> block each other.
>
>
>
> --
>
> GPG ID: FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20150828/9bb4ea2f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 16748 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20150828/9bb4ea2f/attachment.png>