[Borgbackup] Inconsistenses in repository

Wed May 1 16:14:00 EDT 2024

Hi ML,

we have a borg repository on a CIFS share and save each night the images from about 10 virtual machines running on a linux host in this repository.
On Friday 19th we had some problems:
Fri 19 Apr 2024 06:00:01 AM CEST
/mnt/domains/reflinks
/mnt/domains
Fri 19 Apr 2024 06:00:18 AM CEST
/mnt/domains/reflinks
Killed stale lock ha-idg-3.scidom.de at 140947408721621.41751-0.
Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0.   <=====
Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0.   <=====
Creating archive at "/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup::2024-04-19T06:00:18"
 ...

After the new archive is created we have a prune and a compact:
This is the shell script:
...
## copy with borg to repo
cd $REFLINK_ROOT
pwd
time $BORG create -v -e 'fm:*reflink.*' --debug-topic=files_cache --files-cache=mtime,size --list --stats --show-rc ::{now} $REFLINK_ROOT
$DATE

## clean up borg repository
$BORG prune  -v --list --keep-daily=10 --keep-weekly=3 --keep-monthly=2 $BORG_REPO
$DATE

## free space in repository
$BORG compact -v $BORG_REPO
$DATE
 ...

After the prune I got the following errors:
segment 10365 not found, but listed in compaction data
segment 10366 not found, but listed in compaction data
segment 10368 not found, but listed in compaction data
segment 10369 not found, but listed in compaction data
segment 10370 not found, but listed in compaction data
segment 10371 not found, but listed in compaction data
segment 10372 not found, but listed in compaction data
segment 10376 not found, but listed in compaction data
 ...

and later on it seems the CIFS share is gone:
Exception ignored in: <function Repository.__del__ at 0x7fe50cd79e40>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 189, in __del__
    self.close()
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 478, in close
    self.lock.release()
  File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 417, in release
    self._roster.modify(EXCLUSIVE, REMOVE)
  File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 320, in modify
    self.save(roster)
  File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 291, in save
    with open(self.path, "w") as f:
         ^^^^^^^^^^^^^^^^^^^^
BlockingIOError: [Errno 11] Resource temporarily unavailable: '/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup/lock.roster'
Local Exception
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Then backup didn't run for some days because of missing share.
On Tuesday 23. it ran again:
Tue 23 Apr 2024 06:00:01 AM CEST
/mnt/domains/reflinks
/mnt/domains
Tue 23 Apr 2024 06:00:12 AM CEST
/mnt/domains/reflinks
Killed stale lock ha-idg-3.scidom.de at 140947408721621.22956-0.
Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 22956 thread 0.  <====
Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 22956 thread 0.  <====
Creating archive at "/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup::2024-04-23T06:00:12"
 ...
and later:
segment 10365 not found, but listed in compaction data
segment 10366 not found, but listed in compaction data
segment 10367 not found, but listed in compaction data
segment 10368 not found, but listed in compaction data
segment 10369 not found, but listed in compaction data
segment 10370 not found, but listed in compaction data
segment 10371 not found, but listed in compaction data
segment 10372 not found, but listed in compaction data
segment 10376 not found, but listed in compaction data
segment 10380 not found, but listed in compaction data
segment 10381 not found, but listed in compaction data
segment 10382 not found, but listed in compaction data
segment 10383 not found, but listed in compaction data
segment 10384 not found, but listed in compaction data
segment 10385 not found, but listed in compaction data
 ...

I stopped backup and ran a check:
ha-idg-3:~ # borg check -v --repository-only /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup
Starting repository check
Data integrity error: Segment entry checksum mismatch [segment 308, offset 22670132]

ID: 029179b026cc8f09fa5e23bc7c3a3a6fb414f8a227ca700a5883caa07ef80aef rebuilt index: <not found>      committed index: (308, 262593301)
ID: 04c3f35611a375a2ca3b3128142c43b4a063e75d805036e579f7af56eb2f027f rebuilt index: <not found>      committed index: (308, 409460324)
ID: c6a6706c52e25e504223a18c60614938b9a3adf04797519b4113d9ffb3cf44f5 rebuilt index: <not found>      committed index: (308, 265936385)
ID: 075720111838023b2a75f2a784b9e7c35d1fdaf0eb97465a530d9c91564a345a rebuilt index: <not found>      committed index: (308, 243931306)
ID: 814b94f08ced5bd6f344c78ae2858e1306b3994cf79b14097c666594f9490dfe rebuilt index: <not found>      committed index: (308, 25291226)
ID: 9a762ee1d2fff73f324a6cbf6158ff28d0d107eed3cd43965c222e6e95eddbc7 rebuilt index: <not found>      committed index: (308, 62284036)
ID: 925fcb4fef1ed3d94251b8516b93d8aff57e4eeed7f422a7f8234f2371408647 rebuilt index: <not found>      committed index: (308, 383059665)
ID: ac64c8159f6cc29ee0b6c84a7c703a9fd28ae617fbc020593d24a5ad39ba1970 rebuilt index: <not found>      committed index: (308, 143633698)
ID: 5256b0e3d5e94614360e0061b129b4e1cce461f226acfb96759c25c6f260b59e rebuilt index: <not found>      committed index: (308, 65795290)
ID: c0d2aa3a80cd48d9225c4c682d6f76475a4cfef60ec6951896ff19b73bb05a40 rebuilt index: <not found>      committed index: (308, 34193602)
ID: 24a743ca77c263642ceadd3297c1b586e44fac2a8f2662091e3a7ef81877dc80 rebuilt index: <not found>      committed index: (308, 217599204)
ID: dd879f4358e68924a94330cbca8902e45ca8a8c4b3a6485074a23d2f267bfbe2 rebuilt index: <not found>      committed index: (308, 187281410)
ID: 9668a4cf9ff6ff418fd38a23cadac3f77c3259cb642584f0e6668237e96d34e3 rebuilt index: <not found>      committed index: (308, 374603048)
ID: 9b0cced61139edeb97f139a925e85b2c8de069ffaf232e3a0785bbf6733ae0f8 rebuilt index: <not found>      committed index: (308, 57958293)
ID: ab5f034eb3fc4cddc47cf365bfb72a8040ac7ee329c39615275aa3d16380ba90 rebuilt index: <not found>      committed index: (308, 68281313)
ID: d3a7aca4090cd665e25f7e78824eb199e41c4bd1b93dfc3652c5ac8d0cbb70d0 rebuilt index: <not found>      committed index: (308, 247205158)
ID: bb89b6e8c815db9b480fd89174e7eaec5e5e19e4a0362ffeb135b668985799a7 rebuilt index: <not found>      committed index: (308, 223093111)
ID: 297e8433db31f24c0a1b50ac6dd8c12a1b61946a861694a33c7d08b3e6636fd3 rebuilt index: <not found>      committed index: (308, 106577195)
ID: 28a51b1f1cab62cc623c18d3c99c6b5387d0bccd04c0822b2ef0af3bb038ab7e rebuilt index: <not found>      committed index: (308, 184955172)
ID: 7b0801b397a1dc0f65cd4a31462376304f717e9c5c2cf288a01730f1a3dd6c3f rebuilt index: <not found>      committed index: (308, 201633797)
ID: 2a3fb8e8130cb59f45d5007a01d56547bf4cd6457d22acde8b24ddfc7515a4bd rebuilt index: <not found>      committed index: (308, 391620053)
ID: da7ac05318ec71729499c7b8ada5a4b0e94b28315de4c1b50f2632edb30b8de3 rebuilt index: <not found>      committed index: (308, 172609869)
ID: 7fd68f779a508141dfb01970e373e64ce090fd3392a8094b75891e9f3cc1bf99 rebuilt index: <not found>      committed index: (308, 380962649)
ID: 1d372916dd850c78f6e96d4d2e21967cebcd452ebcdd4eab7a3b40640d9be947 rebuilt index: <not found>      committed index: (308, 442149004)
ID: 390c8cdcd4c6ce136e7a85464b6c2c7bc2e81c03239a09baa546d7cdca23fedc rebuilt index: <not found>      committed index: (308, 359002924)
ID: 71c4c640558f452b9f72fb134250f5076dc850adca9b7bf450dc64385393a92e rebuilt index: <not found>      committed index: (308, 97954003)
ID: 68ec58c684eebfe28c8781b4143aca2dc72befab4bfcf194f428b9a638169218 rebuilt index: <not found>      committed index: (308, 339232051)
ID: 8c4ff1ce8cefd319e9acbadf7a1feb636a3ae162f7e8ceaa818060df8ced2b7f rebuilt index: <not found>      committed index: (308, 412782109)
ID: cea7bdb839223f09c9e58055991a672d19d88916daf68ae2e2b4dcb4f0b70ed1 rebuilt index: <not found>      committed index: (308, 67604237)
ID: ce010d06878a35fd852311ab9dc898f593e4293d51d662220922fd02cc754e8f rebuilt index: <not found>      committed index: (308, 206317329)
ID: 5f45b322d2fb2a445b9cf0215644ae32931518ebfea932546a0a7f4e5dffc0a1 rebuilt index: <not found>      committed index: (308, 482695602)
ID: d25f02cb4c3c1908f80f9aac0d4ca206aa7d6c6832688eaacffe8342bc43ce3f rebuilt index: <not found>      committed index: (308, 377248244)
ID: 31724c2b99395dcfb1f4c9c06b7de0ffd3cd044ecd93ae31e75161c401ca95e0 rebuilt index: <not found>      committed index: (308, 483826130)
ID: 125137778b3be4aa0ab571238d423d746706594e7e299aa0aa555801a6b2d628 rebuilt index: <not found>      committed index: (308, 77357122)
ID: 5d7a4b0e60149ea01416417f3352105e309ecb016a53ec63e5f586d0765c9d84 rebuilt index: <not found>      committed index: (308, 41528123)
ID: 8c0ada8e5462dc62df980857a17123c78762b98dbd3d5763ce3bfc89f216c5df rebuilt index: <not found>      committed index: (308, 179583845)
ID: 081b2c80a040cd43f1f71e32bafed7b080a4be78a303c8381dc8cc90dc97865f rebuilt index: <not found>      committed index: (308, 150334327)
ID: db813c478c93448bd4267b3ef14466516a58a3c8b2362466aff749bd7d75e98f rebuilt index: <not found>      committed index: (308, 303280742)
ID: 424fb1bc91182aa7302dbe1e5dd162447b514722637990b6311f9ad22eeaa653 rebuilt index: <not found>      committed index: (308, 146563636)
ID: 366591550587e2012cf8fcc8a68323ed1702c16478276209aa8cc73695a77935 rebuilt index: <not found>      committed index: (308, 501967452)
ID: 422ee15e8175fd7232661bb4eeb3be2d585a856499280b5304eb8672b0b57d88 rebuilt index: <not found>      committed index: (308, 101032139)
ID: 5eb4458656d450a4a59cf5e5524cc22af819b7a7e2db948a26826a0b86a8f9ce rebuilt index: <not found>      committed index: (308, 250294158)
ID: 580ecd751046cf6fead53f41b2a79aea70afe56407f83f3bc439297cae0ad6e5 rebuilt index: <not found>      committed index: (308, 409331104)
ID: 19bd8f50fb4697a7da19d731ade0502a704f255bcff37f886199b2ab530c1fb6 rebuilt index: <not found>      committed index: (308, 201109717)
ID: c77a6f062e8ba01c4f89d492473ab1607d183a73015c85ecdcf7afc4a59ae5b3 rebuilt index: <not found>      committed index: (308, 105297415)
ID: d456aeb2458f623dc37565a6698de21cb6a139a0b85b7a769abb893f9c4d04ce rebuilt index: <not found>      committed index: (308, 266703181)
ID: 3a64a7dd27482555fc8175915ead9009d0b3064256f947d35ac57a0f082333d5 rebuilt index: <not found>      committed index: (308, 125491747)
ID: a97b4d091ddcb2263e903facfc59efd7e7f28c48b4ca3092cdd63efa22e9d0ab rebuilt index: <not found>      committed index: (308, 159246928)
ID: 38e2708d3bbcc1ded3eeca1f6b2e5eac36f05be24a0960c73223fa178b717685 rebuilt index: <not found>      committed index: (308, 178140904)
ID: d7d80eed37c85cb339bca153a01a5945f2596522c1c433424159d663285d863c rebuilt index: <not found>      committed index: (308, 496276696)
ID: dbe36ca001287c987d6d6bfbdd5e2d60ff3496b73897be1ca41d08eb71192c32 rebuilt index: <not found>      committed index: (308, 408211219)
ID: 8610030553de9ce59f46610c18cb11aa9609a502923000d5098c886c3f800036 rebuilt index: <not found>      committed index: (308, 177790291)
Finished full repository check, errors found.

Then I checked the archive metadata:
ha-idg-3:~ # borg check -vp --show-rc --archives-only /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup
Starting archive consistency check...
Enter passphrase for key /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup:
Analyzing archive 2024-01-31T05:00:02 (1/15)
Analyzing archive 2024-02-12T05:00:05 (2/15)
Analyzing archive 2024-03-24T06:00:01 (3/15)
Analyzing archive 2024-03-31T06:00:03 (4/15)
Analyzing archive 2024-04-07T06:00:11 (5/15)
Analyzing archive 2024-04-11T06:00:14 (6/15)
Analyzing archive 2024-04-12T06:00:13 (7/15)
Analyzing archive 2024-04-13T06:00:20 (8/15)
Analyzing archive 2024-04-14T06:00:15 (9/15)
Analyzing archive 2024-04-15T06:00:14 (10/15)
Analyzing archive 2024-04-16T06:00:10 (11/15)
Analyzing archive 2024-04-17T11:11:55 (12/15)
Analyzing archive 2024-04-18T06:00:08 (13/15)
Analyzing archive 2024-04-19T06:00:18 (14/15)
Analyzing archive 2024-04-23T06:00:12 (15/15)
8035 orphaned objects found!
Archive consistency check complete, problems found.
terminating with warning status, rc 1

currently I'm running a data consistency check, which has now passed 42%.

I read the manpage and the help.
I understand what a repository and an archive is.
But what is a segment and what is a chunk ?

And do you have any advice what I could do ?
I tried to check the archive metadata separately for each archiv to find out which one is broken, but
an archive metadata check needs to check all archives.

Bernd

--

Bernd Lentes
SystemAdministrator
Institute of Metabolism and Cell Death
Helmholtz Zentrum München
Building 25 office 122
Bernd.lentes at helmholtz-munich.de
+49 89 3187 1241

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias H. Tschöp, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671
Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias H. Tschöp, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671