[Borgbackup] Backup process of a huge archive is sometimes fast, and sometimes very slow

[Thomas Waldmann]

> Most of the time, the process is quick enough, only take 15 minutes to
> complete:
> 
> Time (start): Thu, 2019-05-16 03:53:42
> Time (end):   Thu, 2019-05-16 10:55:10
> Duration: 7 hours 1 minutes 27.98 seconds

Guess you inserted the wrong log here, this was also a slow run.

> This archive:              726.59 GB            706.02 GB              4.60 

Also, the unit for the rightmost column got truncated, but would be
important.

> I obviously do not control what files/folder are changed in the tree. 

Run the borg create with --list option, so you'll see the status for
each file.

I suspect that for the slow runs, borg detects many (all?) files as
potentially changed.

That can be either a content change or a metadata change size / ctime /
inode number (size of course also means content change in any case,
ctime could be also just a metadata change like acls or xattrs and
inodes could just be unstable due to the filesystem you are using -
network filesystems often have unstable inodes).

ls -i shows the inode number for a file.

> I run two borg backups in parallel to save to two different backup
> server. They process the files at the same time wether the result is
> fast or slow.

Not the best setup for time measurements, though.
Also, when you have a cloud server, there might be a lot of other
circumstances influencing your measurement.

Maybe not the root cause for this huge difference, just saying.

> The source server is on AWS, it's an EBS running on a m5.large server.

source filesystem with the data you backup is ...?


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393