[Borgbackup] How to fix corrupted index or hints?

[Thomas Waldmann]

> first of all: Thank you so much for this awesome backup software!

You're welcome!

> Now I struggle to restore a borg repository ona broken partition from 
> another repo backup. Read checks seems to be fine with
> 
> 
> Remote: Starting repository index check
> Remote: Finished full repository check, no problems found.
> Starting archive consistency check...
> Remote: Verified integrity of /mnt/borg/index.11050
> TAM-verified manifest

Seems like the end of the log is missing here?

Does it also say "Finished archive check" and does it terminate with rc 
0? You can give --show-rc to output it or do it directly afterwards on 
the shell with echo $? .

> but writing to the repo failes with following error:
> 
>    File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
> in compact_segments
>      assert segments[segment] == 0, 'Corrupted segment reference count - 
> corrupted index or hints'
> 
> AssertionError: Corrupted segment reference count - corrupted index or 
> hints

Yeah, there seems to be some inconsistency between these.

> I tried to remove the hints file on the repo side, deleted the client 
> cache via "borg delete --cache-only" and run "borg check --debug 
> --progress --repair":
> 
> 
> ...
> Remote: checking segment file /mnt/borg/data/5/11050...
> Remote: finished segment check at segment 11050
> Remote: Starting repository index check
> Remote: Finished full repository check, no problems found.
> Starting archive consistency check...
> Remote: Verified integrity of /mnt/borg/index.11050
> TAM-verified manifest
> Analyzing archive client-20110614-224401 (1/66)
> ...
> Analyzing archive client-20221103-223444 (66/66)
> RepositoryCache: current items 3847, size 289.49 MB / 999.87 MB, 22926 
> hits, 3847 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit
> Writing Manifest.
> Remote: Cleaned up 0 uncommitted segment files (== everything after 
> segment 11050).
> Remote: Verified integrity of /mnt/borg/hints.11050
> Committing repo (may take a while, due to compact_segments)...
> Remote: check_free_space: required bytes 361371486, free bytes 117106585600
> Remote: Compaction started (threshold is 10%).
> Remote: compacting segment 2000 with usage count -84 (maybe freeable: 
> 197.06% [276754300 bytes])

Uhoh, that looks weird. :-( ^^^

> Remote: Traceback (most recent call last):
> 
>    File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
>      res = f(**args)
> 
>    File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
> in commit
>      self.compact_segments(threshold)
> 
>    File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
> in compact_segments
>      assert segments[segment] == 0, 'Corrupted segment reference count - 
> corrupted index or hints'
> 
> AssertionError: Corrupted segment reference count - corrupted index or 
> hints
> 
> RemoteRepository: 213.05 kB bytes sent, 320.78 MB bytes received, 3935 
> messages sent
> Traceback (most recent call last):
> 
>    File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
>      res = f(**args)
> 
>    File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, 
> in commit
>      self.compact_segments(threshold)
> 
>    File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, 
> in compact_segments
>      assert segments[segment] == 0, 'Corrupted segment reference count - 
> corrupted index or hints'
> 
> AssertionError: Corrupted segment reference count - corrupted index or 
> hints
> 
> Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP 
> PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64
> Borg server: Linux: Unknown Linux
> Borg server: Borg: 1.2.0  Python: CPython 3.10.6 msgpack: 1.0.3 fuse: 
> pyfuse3 3.2.0 [pyfuse3,llfuse]
> Borg server: PID: 1086983  CWD: /home/sebastian
> Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', 
> '/mnt/borg']
> Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug'
> Platform: Linux client 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 
> (2022-09-29) x86_64
> Linux: Unknown Linux
> Borg: 1.1.16  Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
> PID: 9384  CWD: /home/sebastian
> sys.argv: ['/usr/bin/borg', 'check', '--debug', '--progress', '--repair']
> 
> SSH_ORIGINAL_COMMAND: None

That's bad, I would have thought that your approach (deleting hints and 
cache, running borg check --repair) would fix the issue.

I'll try to reproduce / find that issue, thanks for reporting it.

Please open an issue about this on the issue tracker on github so we can 
better keep track of it.

If possible, please keep a copy of the repository it the state where it 
is possible to reproduce this issue - just in case we need it for debugging.

> I am wondering why the full repository check reports "no problems found" 
> but later the segments are corrupted?

As of now, I do not think the segment files are corrupted. At least not 
in a way like an entry's crc32 not matching the entry contents any more, 
that would have been discovered by the borg check (repository part).