[Borgbackup] backup speed

[Thomas Waldmann]

> I'm backing up about 2 1/2 TB of data across the net to some cloud-based
> storage.

rsync.net or are there others also?

How many files (filesystem objects) are we talking about?

> It's borg client to Borg server (v1.07) and the initial part of
> the transfer was  fast (12 Mb per second) and now it's running slow (2
> Mb per second).

Hard to say if that indicates a problem. Some infos / ideas:

- a lot of small files naturally give lower throughput than large files,
so throughput varies.

- compression might not work for some files, work normally for others
and hilariously for sparse files (all those 0 bytes compress really good).

- if you start from 0, borg hashindex is small, hash operations are very
fast. with growing amounts of chunks, the hash table grows bigger, RAM
consumption rises (make sure you have enough RAM - if you run out of RAM
and starts to swap, it gets really slow).

- if the index is big, starting a transaction takes more time than for a
small index

- by default, it does checkpoints every 5 minutes. if your connection is
stable, you can use a longer interval

- I have the suspicion that sometimes the hashtable performance gets
slow (depends on the data fed into it),
https://github.com/borgbackup/borg/issues/536 - a developer is working
on a better hashtable implementation and also related performance tests
/ statistics.

> I believe the initial transfer never completed

You can use -v --show-rc to log the return code.

> and I've tried a couple
> times to finish the complete transfer but the backup has run for days
> transferring one iso image.

How do you know it is that one iso image all the time?

> eric at schist:~$ borg check -v esj at 192.168.73.232:backups/schist
> Remote: Starting repository check
> Remote: Completed repository check, no problems found.
> Starting archive consistency check...
> Analyzing archive 2016-08-26.checkpoint (11/11)
> Analyzing archive 2016-08-23.checkpoint (10/11)
> Analyzing archive 2016-08-22.checkpoint (9/11)
> Analyzing archive 2016-08-21.checkpoint (8/11)
> Analyzing archive 2016-08-20.checkpoint (7/11)
> Analyzing archive 2016-08-18.checkpoint (6/11)
> Analyzing archive 2016-05-07 (5/11)
> Analyzing archive 2016-04-13 (4/11)
> Analyzing archive 2016-04-05 (3/11)
> Analyzing archive 2016-03-31 (2/11)
> Analyzing archive 2016-03-27.checkpoint (1/11)
> Archive consistency check complete, no problems found.

After(!) you have finished your backup(s) successfully, you can delete
the checkpoints.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393