[Borgbackup] borg 1.1 performance (and also 1.0)

[Thomas Waldmann]

Had some time and played around, looking for maximum borg performance I
could see on my systems, also comparing 1.1 to 1.0.

I used a standard ubuntu iso, so you can do the same measurement.

System used for this:
Xeon E5-2667 v2 (oldie, but goldie)
64GB RAM (mostly unused for this)
Samsung 960 Pro SSD (PCIe x4, NVME)
Ubuntu Linux 16.04 64bit
ext4 filesystem

==============================================================================

borg 1.1.1.dev36+g40186a3 (close to what will be in 1.1.1 release soon)

$ BORG_PASSPHRASE=secret borg init -e authenticated-blake2 repo
$ sudo dropcache
$ BORG_PASSPHRASE=secret borg create --stats repo::ubuntu-iso
ubuntu-16.04.3-desktop-amd64.iso

------------------------------------------------------------------------------
Archive name: ubuntu-iso
Archive fingerprint:
6c5c62ee1fba8ba51a66c5be12de0eeeaa04935d2a926cb5daa4a112a3f61f53
Time (start): Fri, 2017-10-20 02:30:15
Time (end):   Fri, 2017-10-20 02:30:24
Duration: 8.06 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
               Original size      Compressed size    Deduplicated size
This archive:        1.59 GB              1.58 GB              1.58 GB
All archives:        1.59 GB              1.58 GB              1.58 GB

                       Unique chunks         Total chunks
Chunk index:                     598                  598
------------------------------------------------------------------------------

197 MB/s borg create throughput. Yay!

If you get a faster speed here, let us know.


$ sudo dropcache
$ BORG_PASSPHRASE=secret borg create --stats --files-cache=disabled
repo::ubuntu-iso2 ubuntu-16.04.3-desktop-amd64.iso

------------------------------------------------------------------------------
Archive name: ubuntu-iso2
Archive fingerprint:
b5929010bbef035e3f5a97cdb077f7ee720f93d0bb741e0bb24e383d7c0e59f4
Time (start): Fri, 2017-10-20 02:42:12
Time (end):   Fri, 2017-10-20 02:42:17
Duration: 5.07 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
               Original size      Compressed size    Deduplicated size
This archive:        1.59 GB              1.58 GB                473 B
All archives:        3.18 GB              3.17 GB              1.58 GB

                       Unique chunks         Total chunks
Chunk index:                     599                 1196
------------------------------------------------------------------------------

This was just testing how fast it gets when it has to read, chunk, hash,
but no data is written to the repo: 313 MB/s


==============================================================================

borg 1.0.7:

$ ./borg-1.0.7 init -e none repo10
$ sudo dropcache
$ ./borg-1.0.7 create -v --stats repo10::ubuntu-iso
ubuntu-16.04.3-desktop-amd64.iso

------------------------------------------------------------------------------
Archive name: ubuntu-iso
Archive fingerprint:
0714f2c23c0471b3b82344afbb8a99c89e41275b24f715cb8c380d7e67df1349
Time (start): Fri, 2017-10-20 02:57:51
Time (end):   Fri, 2017-10-20 02:58:05
Duration: 14.11 seconds
Number of files: 1
------------------------------------------------------------------------------
               Original size      Compressed size    Deduplicated size
This archive:        1.59 GB              1.59 GB              1.59 GB
All archives:        1.59 GB              1.59 GB              1.59 GB

                       Unique chunks         Total chunks
Chunk index:                     592                  592
------------------------------------------------------------------------------

Although this repo is not authenticated, borg 1.0 is slower: 113 MB/s


$ sudo dropcache
$ ./borg-1.0.7 create -v --stats --no-files-cache repo10::ubuntu-iso2
ubuntu-16.04.3-desktop-amd64.iso

------------------------------------------------------------------------------
Archive name: ubuntu-iso2
Archive fingerprint:
f1f72b7ef1f4941ad447ef1e0f62a6faaa70b7e52c280648ef17e77a924edf92
Time (start): Fri, 2017-10-20 02:58:25
Time (end):   Fri, 2017-10-20 02:58:33
Duration: 8.02 seconds
Number of files: 1
------------------------------------------------------------------------------
               Original size      Compressed size    Deduplicated size
This archive:        1.59 GB              1.59 GB                277 B
All archives:        3.18 GB              3.18 GB              1.59 GB

                       Unique chunks         Total chunks
Chunk index:                     593                 1184
------------------------------------------------------------------------------

Reading / chunking / hashing also slower, about 200 MB/s.

==============================================================================

dropcache script:

# free pagecache, dentries and inodes
sync
echo 3 > /proc/sys/vm/drop_caches

==============================================================================

Notes:

- Don't be disappointed if your daily backup does not run that fast.
- Small files will always be much slower due to access time and metadata
processing overhead.
- HDDs and also most SSDs are slower than the one used here.
- The CPU used is some years old, but has 25MB Cache and can turbo to
4.0 GHz.
- I did not use encryption here. With repokey-blake2 it is 144 MB/s.
- The borg 1.0 repo was uncompressed, borg 1.1 tried lz4 compression
(but as the input data was compressed already, that did not win much).

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393