[Borgbackup] How to fix corrupted index or hints?

Mon Nov 28 14:02:36 EST 2022

Hi Thomas and borg users

On 11/17/22 23:27, Thomas Waldmann wrote:
>> I am running now: borg check --verify-data --repair --debug 
>> --progress --show-version --show-rc
>
> That will be rather slow, but very thorough.

Sorry for the late reply. I have results of the command above with 
following log. I did not delete the hints and index files before. I had 
hope that rsync --delete would fix the job, but...

-----
borg check --repair --debug --progress --show-version --show-rc
using builtin fallback logging configuration
borgbackup version 1.1.16
35 self tests completed in 0.18 seconds
SSH command line: ['ssh', '-i', '/home/user/.ssh/user at client', 
'user at server', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 38 self tests completed in 0.32 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/mnt/borg'
Remote: Resolved repository path to '/mnt/borg'
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: Remote: Starting 
repository check
Remote: Verified integrity of /mnt/borg/index.11057
Remote: Verified integrity of /mnt/borg/hints.11057
Remote: Replaying segments   0%
...
Remote: Replaying segments  95%
Remote: Verified integrity of /mnt/borg/index.11061
Remote: Read committed index of transaction 11061
Remote: Cleaned up 1 uncommitted segment files (== everything after 
segment 11061).
Remote: Segment transaction is    11061
Remote: Determined transaction is 11061
Remote: Found 11004 segments
Remote: Checking segments 0.0%
Remote: checking segment file /mnt/borg/data/0/1...
...
Remote: checking segment file /mnt/borg/data/5/11061...
Remote: finished segment check at segment 11061
Remote: Starting repository index check
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /mnt/borg/index.11061
TAM-verified manifest
Analyzing archive client-20110614-224401 (1/68)
...
Analyzing archive client-20221103-223444 (66/68)
Analyzing archive client-20221112-144451.checkpoint (67/68)
Analyzing archive client-20221112-220336.checkpoint (68/68)
RepositoryCache: current items 3849, size 289.73 MB / 998.12 MB, 23086 
hits, 3849 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit
Writing Manifest.
Remote: Cleaned up 0 uncommitted segment files (== everything after 
segment 11061).
Remote: Verified integrity of /mnt/borg/hints.11061
Committing repo (may take a while, due to compact_segments)...
Remote: check_free_space: required bytes 361371686, free bytes 117087985664
Remote: Compaction started (threshold is 10%).
Remote: compacting segment 2000 with usage count -84 (maybe freeable: 
197.06% [276754300 bytes])
Remote: Traceback (most recent call last):

   File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
     res = f(**args)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
in commit
     self.compact_segments(threshold)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
in compact_segments
     assert segments[segment] == 0, 'Corrupted segment reference count - 
corrupted index or hints'

AssertionError: Corrupted segment reference count - corrupted index or hints

RemoteRepository: 213.41 kB bytes sent, 321.02 MB bytes received, 3939 
messages sent
Traceback (most recent call last):

   File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
     res = f(**args)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
in commit
     self.compact_segments(threshold)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
in compact_segments
     assert segments[segment] == 0, 'Corrupted segment reference count - 
corrupted index or hints'

AssertionError: Corrupted segment reference count - corrupted index or hints

Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP 
PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.0  Python: CPython 3.10.6 msgpack: 1.0.3 fuse: 
pyfuse3 3.2.0 [pyfuse3,llfuse]
Borg server: PID: 1717358  CWD: /home/user
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', 
'/mnt/borg']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug'
Platform: Linux client 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 
(2022-10-21) x86_64
Linux: Unknown Linux
Borg: 1.1.16  Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 806530  CWD: /home/user
sys.argv: ['/usr/bin/borg', 'check', '--repair', '--debug', 
'--progress', '--show-version', '--show-rc']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2
-----

The log does not tell me any news and reported the same strange errors 
"compacting segment 2000 with usage count -84" and "Corrupted segment 
reference count - corrupted index or hints"

Further the "rsync -a --delete backup/ repo/" did not help because the 
cronjob for the rsync backup job with ~"rsync -a repo/ backup/" was not 
disabled and created another remote copy :-/

Any hints to proceed to get a writable repo again?

>> And if borg reports that all segments are valid what corrupts the 
>> segment reference count?
>
> Could be that although each segment file looks valid (== all the crc32 
> checks of its entries are ok), the overall state when looking at all 
> segments in their on-disk order is not valid and is not produced by 
> normal borg operations?

But I assume that borgs internal structure is comparable to git tree 
which I do know. So if the archive information with the root entry, file 
tree and the file leafs are OK, all data should be fine. There might be 
additional objects which are not referenced but overall the data should 
be there and safe. Isn't it?

So can I reconstruct/rebuilt the repo, eg. by the new transfer command 
of borg 2? Is there an alternative for borg 1.2?

>>> I just checked in the code: borg check --repair rebuilds the hints 
>>> and index from information found in the segment files only.
>>
>>
>> So it should be fine to delete hints and index files on the repo and 
>> call borg check --repair. Than the "usage count" and "freeable space" 
>> should be fine?
>
> Usually yes.

I never tried to delete integrity file, too. But would it be worth it 
delete index, hints and integrity files and run "borg check --repair"?

Thank you for your help and time

Sebastian