[Borgbackup] Decreasing speed of borg check command?

[Marcin Zajączkowski]

On 2019-04-04 23:57, Thomas Waldmann wrote:
>> I would like to consult one observation. Performing a 600GB+ repo check,
>> the HDD read speed started at ~110MB/s (USB3 HDD drive), but over time
>> tends to decease up until ~60MB/s after 2 hours. Disk transfer data from
>> dstat.
> 
> Depending on size and internals of your target HDD, part of the problem
> might be that data transfer speed on outer tracks is faster than on
> inner tracks.

It could be. Some sources points that data on outer cylinders could be
up to 50% faster that the inner ones.

> I don't think there is a memory leak (at least not a significant one).
> 
> But when building the in-memory data structures, speed may vary
> depending e.g. on size and fill ratio of the hashtables. While doing
> that, it uses more and more memory, but that is used memory, not leaked
> memory.

That's something what I meant thinking about "some kind of memory leak" :).

> You might also observe some property of the python interpreter's memory
> manager: it does not like to give back memory to the OS memory
> management, but rather keeps it.
> 
>> I may try reproduce that behavior later on this month if needed. Borg
>> 1.1.8, Fedora 29.
> 
> Was it this borg version you used for your measurement?

Yes, the backup was created and checked with Borg 1.1.8.

> Long ago there was a bug in borg that led to performance problems in the
> hash tables while they filled up (if you want to search for it:
> "tombstones").
> 
> Also watch whether your RAM is enough. If it uses more than present,
> performance goes down the drain due to paging.

I don't have memory data collected, but I would say that was at least
few GBs of memory available for Borg. I will check it in the another try.

Marcin

-- 
https://blog.solidsoft.info/ - Working code is not enough