[Borgbackup] How to fix corrupted index or hints?

[Sebastian Felis]

Hi

first of all: Thank you so much for this awesome backup software! It has 
such a great feature list and I am using it for my machines and backups 
for years now without a hassle.


Now I struggle to restore a borg repository ona broken partition from 
another repo backup. Read checks seems to be fine with


Remote: Starting repository index check
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /mnt/borg/index.11050
TAM-verified manifest


but writing to the repo failes with following error:

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
in compact_segments
     assert segments[segment] == 0, 'Corrupted segment reference count - 
corrupted index or hints'

AssertionError: Corrupted segment reference count - corrupted index or hints


I tried to remove the hints file on the repo side, deleted the client 
cache via "borg delete --cache-only" and run "borg check --debug 
--progress --repair":


...
Remote: checking segment file /mnt/borg/data/5/11050...
Remote: finished segment check at segment 11050
Remote: Starting repository index check
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /mnt/borg/index.11050
TAM-verified manifest
Analyzing archive client-20110614-224401 (1/66)
...
Analyzing archive client-20221103-223444 (66/66)
RepositoryCache: current items 3847, size 289.49 MB / 999.87 MB, 22926 
hits, 3847 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit
Writing Manifest.
Remote: Cleaned up 0 uncommitted segment files (== everything after 
segment 11050).
Remote: Verified integrity of /mnt/borg/hints.11050
Committing repo (may take a while, due to compact_segments)...
Remote: check_free_space: required bytes 361371486, free bytes 117106585600
Remote: Compaction started (threshold is 10%).
Remote: compacting segment 2000 with usage count -84 (maybe freeable: 
197.06% [276754300 bytes])
Remote: Traceback (most recent call last):

   File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
     res = f(**args)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
in commit
     self.compact_segments(threshold)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
in compact_segments
     assert segments[segment] == 0, 'Corrupted segment reference count - 
corrupted index or hints'

AssertionError: Corrupted segment reference count - corrupted index or hints

RemoteRepository: 213.05 kB bytes sent, 320.78 MB bytes received, 3935 
messages sent
Traceback (most recent call last):

   File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
     res = f(**args)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
in commit
     self.compact_segments(threshold)

   File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
in compact_segments
     assert segments[segment] == 0, 'Corrupted segment reference count - 
corrupted index or hints'

AssertionError: Corrupted segment reference count - corrupted index or hints

Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP 
PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.0  Python: CPython 3.10.6 msgpack: 1.0.3 fuse: 
pyfuse3 3.2.0 [pyfuse3,llfuse]
Borg server: PID: 1086983  CWD: /home/sebastian
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', 
'/mnt/borg']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug'
Platform: Linux client 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 
(2022-09-29) x86_64
Linux: Unknown Linux
Borg: 1.1.16  Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 9384  CWD: /home/sebastian
sys.argv: ['/usr/bin/borg', 'check', '--debug', '--progress', '--repair']

SSH_ORIGINAL_COMMAND: None


I am wondering why the full repository check reports "no problems found" 
but later the segments are corrupted? Is there a way to restore the 
segment references and fix that problem?

Client runs borg version 1.1.16 on Debian/amd64
Server runs borg version 1.2.0 on Ubuntu/aarch64 (Pi4)

I could restore other repos in the same way without problems, but the 
biggest failed and I would like to fix it.


Best regards

Sebastian