[Borgbackup] Backup process of a huge archive is sometimes fast, and sometimes very slow

[Damien Gustave]

Hello all,

I backup a huge folder of +700 GB and daily backups of this.

Most of the time, the process is quick enough, only take 15 minutes to
complete:

Time (start): Thu, 2019-05-16 03:53:42
Time (end):   Thu, 2019-05-16 10:55:10
Duration: 7 hours 1 minutes 27.98 seconds
Number of files: 1650426
Utilization of max. archive size: 0%
---------
                       Original size      Compressed size    Deduplicated
This archive:              726.59 GB            706.02 GB              4.60
All archives:               45.60 TB             44.16 TB            729.28

                       Unique chunks         Total chunks
Chunk index:                 1692888            127730487

But other times, it takes several hours like this:

Time (start): Thu, 2019-05-16 03:53:42
Time (end):   Thu, 2019-05-16 10:55:10
Duration: 7 hours 1 minutes 27.98 seconds
Number of files: 1650426
Utilization of max. archive size: 0%
---------
                       Original size      Compressed size    Deduplicated
This archive:              726.59 GB            706.02 GB              4.60
All archives:               45.60 TB             44.16 TB            729.28

                       Unique chunks         Total chunks
Chunk index:                 1692888            127730487

I obviously do not control what files/folder are changed in the tree.

I run two borg backups in parallel to save to two different backup server.
They process the files at the same time wether the result is fast or slow.

The source server is on AWS, it's an EBS running on a m5.large server.

Do you have any idea why such a difference exists ?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190517/d2dc0198/attachment.html>