From tgries at online.de Thu Apr 18 18:24:28 2024 From: tgries at online.de (Thomas Gries) Date: Fri, 19 Apr 2024 00:24:28 +0200 Subject: [Borgbackup] Full repository check, errors found: Index object count mismatch. Message-ID: <5bc36cc9-2ae4-4839-b47b-8ea7145560f3@online.de> 2024-04-18 20:20:16,834 - vorta.borg.borg_job - INFO - finished segment check at segment 4011 2024-04-18 20:20:17,064 - vorta.borg.borg_job - INFO - Starting repository index check 2024-04-18 20:20:17,066 - vorta.borg.borg_job - ERROR - Index object count mismatch. 2024-04-18 20:20:17,067 - vorta.borg.borg_job - ERROR - committed index: 1808190 objects 2024-04-18 20:20:17,068 - vorta.borg.borg_job - ERROR - rebuilt index:?? 1808191 objects 2024-04-18 20:20:17,503 - vorta.borg.borg_job - WARNING - ID: 44d1149947bf7eec5ab719e2b83c1d894d6263e5718d0fd108e03a30d02380ac rebuilt index: (662, 290401399) committed index: 2024-04-18 20:20:17,911 - vorta.borg.borg_job - ERROR - Finished full repository check, errors found. 2024-04-18 20:20:18,110 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 11 2024-04-18 20:20:18,110 - vorta.borg.jobs_manager - DEBUG - No more jobs for site: 11 borg 1.2.7 vorta 0.9.1 Today I ran the "Pr?fen" Function (verify) in vorta and got for some unknown reason "ERROR - Index object count mismatch". What can I do? What is the suggested action I should do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Thu Apr 18 19:11:30 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 19 Apr 2024 01:11:30 +0200 Subject: [Borgbackup] Full repository check, errors found: Index object count mismatch. In-Reply-To: <5bc36cc9-2ae4-4839-b47b-8ea7145560f3@online.de> References: <5bc36cc9-2ae4-4839-b47b-8ea7145560f3@online.de> Message-ID: <869c90f1-42da-466c-a6af-c0c487e41ae8@waldmann-edv.de> > borg 1.2.7 > > Today I ran the "Pr?fen" Function (verify) in vorta and got for some > unknown reason "ERROR - Index object count mismatch". > What can I do? What is the suggested action I should do? If the repo was created and used with a borg version < 1.2.5, you MUST first read and follow the repo upgrade steps at the top of the 1.2.7 changelog: https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 After that, run borg check --repair on the repo. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From tgries at online.de Fri Apr 19 10:33:56 2024 From: tgries at online.de (Thomas Gries) Date: Fri, 19 Apr 2024 16:33:56 +0200 Subject: [Borgbackup] Full repository check, errors found: Index object count mismatch. In-Reply-To: <869c90f1-42da-466c-a6af-c0c487e41ae8@waldmann-edv.de> References: <5bc36cc9-2ae4-4839-b47b-8ea7145560f3@online.de> <869c90f1-42da-466c-a6af-c0c487e41ae8@waldmann-edv.de> Message-ID: <012bdd2d-e3e8-4835-b4ef-957861d74e52@online.de> Am 19.04.24 um 01:11 schrieb Thomas Waldmann: >> borg 1.2.7 >> >> Today I ran the "Pr?fen" Function (verify) in vorta and got for some >> unknown reason "ERROR - Index object count mismatch". >> What can I do? What is the suggested action I should do? > > If the repo was created and used with a borg version < 1.2.5, you MUST > first read and follow the repo upgrade steps at the top of the 1.2.7 > changelog: > > https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 > > > After that, run borg check --repair on the repo. > > thanks; problem *_solved_*. Vorta reports now: Check completed. INFO: Archive consistency check completed, no problems found.) *Feedback and two suggestions for improvement to you, Thomas:* SUGGESTION 1: * the borg check -repair function ends in my case with: 1 orphaned objects found! Archive consistency check complete, problems found. (see below) Here I suggest in cases like mine - when check -repair actually repaired something - to add saying "*and repaired 1 problem"* 1 orphaned objects found. Archive consistency check complete, problems found*and repaired 1 problem.* SUGGESTION 2: In borgbackup >= 1.2.6 add a text explaining the most like reason and fix *"check information on https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811"* for error message 2024-04-18 20:20:17,066 - vorta.borg.borg_job - ERROR - Index object count mismatch. add 2024-04-18 20:20:17,066 - vorta.borg.borg_job - ERROR - Index object count mismatch*(check information on https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 ) *Regards Tom ------------------------------------------------------------------------ Log (excerpt): BORG_WORKAROUNDS=ignore_invalid_archive_tam borg info --debug /media/benutzer/borg2tb/samosbackup 2>&1 | grep TAM | grep -i manifest^C samos-20230211-0038 Sat, 2023-02-11 00:38:09 tam:verified samos-20230331-0232 Fri, 2023-03-31 02:32:53 tam:verified samos-20230410-0902 Mon, 2023-04-10 09:02:16 tam:verified samos-20230519-1133 Fri, 2023-05-19 11:34:02 tam:verified samos-20230611-2239 Sun, 2023-06-11 22:39:54 tam:verified samos-20230718-1056 Tue, 2023-07-18 10:56:20 tam:verified samos-20230827-2352 Sun, 2023-08-27 23:52:56 tam:verified samos-20230907-1840 Thu, 2023-09-07 18:40:02 tam:verified samos-20231024-1101 Tue, 2023-10-24 11:01:45 tam:verified samos-20231230-2357 Sat, 2023-12-30 23:57:12 tam:verified samos-20240125-1958 Thu, 2024-01-25 19:58:32 tam:verified samos-20240224-2311 Sat, 2024-02-24 23:11:56 tam:verified samos-20240325-0923 Mon, 2024-03-25 09:23:03 tam:verified samos-20240414-2316 Sun, 2024-04-14 23:16:02 tam:verified samos-20240418-1235 Thu, 2024-04-18 12:35:21 tam:verified^C borg check --repair /media/benutzer/borg2tb/samosbackup/ This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! Type 'YES' if you understand this and want to continue: YES (Enter passphrase for key:) 1 orphaned objects found! Archive consistency check complete, problems found. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Fri Apr 19 16:14:34 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 19 Apr 2024 22:14:34 +0200 Subject: [Borgbackup] Full repository check, errors found: Index object count mismatch. In-Reply-To: <012bdd2d-e3e8-4835-b4ef-957861d74e52@online.de> References: <5bc36cc9-2ae4-4839-b47b-8ea7145560f3@online.de> <869c90f1-42da-466c-a6af-c0c487e41ae8@waldmann-edv.de> <012bdd2d-e3e8-4835-b4ef-957861d74e52@online.de> Message-ID: > SUGGESTION 1: > > * the borg check -repair function ends in my case with: > > 1 orphaned objects found! > Archive consistency check complete, problems found. > (see below) > > > Here I suggest in cases like mine - when check -repair actually repaired > something - to add saying "*and repaired 1 problem"* > > 1 orphaned objects found. > Archive consistency check complete, problems found*and repaired 1 > problem.* Good idea, can you file a github issue? > SUGGESTION 2: > > In borgbackup >= 1.2.6 add a text explaining the most like reason and fix > *"check information on > https://borgbackup.readthedocs.io/en/1.2.7/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811"* > > for error message > 2024-04-18 20:20:17,066 - vorta.borg.borg_job - ERROR - Index object > count mismatch. It's not like the CVE issue and the object count mismatch are related. I just pointed out you must follow the repo upgrade steps first because otherwise borg might consider some archives without a valid tam as crap and would DELETE them. Archives without a valid tam may exist in old repos due to: - a bug in older borg versions, especially in borg rename and recreate. - an attack by a malicious entity (see the upgrade notes for more details) -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From tgries at online.de Sat Apr 20 10:58:00 2024 From: tgries at online.de (Thomas Gries) Date: Sat, 20 Apr 2024 16:58:00 +0200 Subject: [Borgbackup] Question regarding "Size" versus actual filesize Message-ID: <8bb25ede-b645-493d-b23e-e42322bf2098@online.de> My borgbackup (here the vorta display) shows (after recalculation) /_sizes_/in the GB range, which do not correspond to the actual used file size on disk (which is about ~440 GB). Is there any explanation somewhere in the borg-documentation of what the Gr??e/Size means - if there is no correspondence to the used disk space? I am puzzled.... -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sun Apr 21 12:45:21 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 21 Apr 2024 18:45:21 +0200 Subject: [Borgbackup] Question regarding "Size" versus actual filesize In-Reply-To: <8bb25ede-b645-493d-b23e-e42322bf2098@online.de> References: <8bb25ede-b645-493d-b23e-e42322bf2098@online.de> Message-ID: > My borgbackup (here the vorta display) shows (after recalculation) > /_sizes_/in the GB range, which do not correspond to the actual used > file size on disk (which is about ~440 GB). > > Is there any explanation somewhere in the borg-documentation of what the > Gr??e/Size means - if there is no correspondence to the used disk space? > I am puzzled.... Did you maybe forget to add the "borg compact" call, which is required to actually free space since borg 1.2? -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From ed at edgewood.to Fri Apr 26 16:19:09 2024 From: ed at edgewood.to (Ed Blackman) Date: Fri, 26 Apr 2024 16:19:09 -0400 Subject: [Borgbackup] Consistent unexpectedly large deduplicated size Message-ID: <1714160678.e0708d@strabo.garrett> I have a couple of backups going to the same repo. They are in the same repo because files sometimes move between backup sets. I have different prefixes for the backups so that I can have different prune policies. One of them has a consistent unexpectedly large deduplicated size, much larger than the sum size of the new files being added. Local and server borg are version 1.2.4 as packaged by Debian 12. Common command line options to both backups: /usr/bin/borg create --chunker-params 19,23,21,4095 --stats --list --show-rc --keep-exclude-tags --exclude-caches --exclude-if-present .nobackup --noatime Backup A (the one with the unexpected size) had 45,364,409 bytes in 300 new files. Options specific to this backup were: --compression=lz4 --patterns-from=/home/backups/borg/snapshot/borg-tmp.20240426.1iExpB/hometmp-patterns server:/path::a_backup=host at 2024-04-26T04:18:02 home Stats output for Backup A included: Number of files: 316114 Original size Compressed size Deduplicated size This archive: 32.14 GB 28.46 GB 215.98 MB All archives: 43.55 TB 39.14 TB 667.32 GB Unique chunks Total chunks Chunk index: 1069818 73111633 Backup B had 4,226,032 bytes in 10 new files. Options specific to this backup were: --compression=zlib --exclude-from=/etc/backup/borg.all.exclude server:/path::b_backup=host at 2024-04-26T04:18:02 home/user/projects Stats output for Backup B included: Number of files: 79413 Original size Compressed size Deduplicated size This archive: 6.10 GB 5.08 GB 6.02 kB All archives: 43.52 TB 39.12 TB 667.14 GB Unique chunks Total chunks Chunk index: 1069619 72877206 I'm not sure why backup A's deduplicated size is so much larger than the size of the new files in it. Are there any ideas about how to find out? -- Ed?Blackman From bernd.lentes at helmholtz-muenchen.de Wed May 1 16:14:00 2024 From: bernd.lentes at helmholtz-muenchen.de (Bernd Lentes) Date: Wed, 1 May 2024 20:14:00 +0000 Subject: [Borgbackup] Inconsistenses in repository Message-ID: Hi ML, we have a borg repository on a CIFS share and save each night the images from about 10 virtual machines running on a linux host in this repository. On Friday 19th we had some problems: Fri 19 Apr 2024 06:00:01 AM CEST /mnt/domains/reflinks /mnt/domains Fri 19 Apr 2024 06:00:18 AM CEST /mnt/domains/reflinks Killed stale lock ha-idg-3.scidom.de at 140947408721621.41751-0. Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0. <===== Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0. <===== Creating archive at "/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup::2024-04-19T06:00:18" ... After the new archive is created we have a prune and a compact: This is the shell script: ... ## copy with borg to repo cd $REFLINK_ROOT pwd time $BORG create -v -e 'fm:*reflink.*' --debug-topic=files_cache --files-cache=mtime,size --list --stats --show-rc ::{now} $REFLINK_ROOT $DATE ## clean up borg repository $BORG prune -v --list --keep-daily=10 --keep-weekly=3 --keep-monthly=2 $BORG_REPO $DATE ## free space in repository $BORG compact -v $BORG_REPO $DATE ... After the prune I got the following errors: segment 10365 not found, but listed in compaction data segment 10366 not found, but listed in compaction data segment 10368 not found, but listed in compaction data segment 10369 not found, but listed in compaction data segment 10370 not found, but listed in compaction data segment 10371 not found, but listed in compaction data segment 10372 not found, but listed in compaction data segment 10376 not found, but listed in compaction data ... and later on it seems the CIFS share is gone: Exception ignored in: Traceback (most recent call last): File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 189, in __del__ self.close() File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 478, in close self.lock.release() File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 417, in release self._roster.modify(EXCLUSIVE, REMOVE) File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 320, in modify self.save(roster) File "/usr/lib64/python3.11/site-packages/borg/locking.py", line 291, in save with open(self.path, "w") as f: ^^^^^^^^^^^^^^^^^^^^ BlockingIOError: [Errno 11] Resource temporarily unavailable: '/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup/lock.roster' Local Exception Traceback (most recent call last): File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Then backup didn't run for some days because of missing share. On Tuesday 23. it ran again: Tue 23 Apr 2024 06:00:01 AM CEST /mnt/domains/reflinks /mnt/domains Tue 23 Apr 2024 06:00:12 AM CEST /mnt/domains/reflinks Killed stale lock ha-idg-3.scidom.de at 140947408721621.22956-0. Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 22956 thread 0. <==== Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 22956 thread 0. <==== Creating archive at "/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup::2024-04-23T06:00:12" ... and later: segment 10365 not found, but listed in compaction data segment 10366 not found, but listed in compaction data segment 10367 not found, but listed in compaction data segment 10368 not found, but listed in compaction data segment 10369 not found, but listed in compaction data segment 10370 not found, but listed in compaction data segment 10371 not found, but listed in compaction data segment 10372 not found, but listed in compaction data segment 10376 not found, but listed in compaction data segment 10380 not found, but listed in compaction data segment 10381 not found, but listed in compaction data segment 10382 not found, but listed in compaction data segment 10383 not found, but listed in compaction data segment 10384 not found, but listed in compaction data segment 10385 not found, but listed in compaction data ... I stopped backup and ran a check: ha-idg-3:~ # borg check -v --repository-only /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup Starting repository check Data integrity error: Segment entry checksum mismatch [segment 308, offset 22670132] ID: 029179b026cc8f09fa5e23bc7c3a3a6fb414f8a227ca700a5883caa07ef80aef rebuilt index: committed index: (308, 262593301) ID: 04c3f35611a375a2ca3b3128142c43b4a063e75d805036e579f7af56eb2f027f rebuilt index: committed index: (308, 409460324) ID: c6a6706c52e25e504223a18c60614938b9a3adf04797519b4113d9ffb3cf44f5 rebuilt index: committed index: (308, 265936385) ID: 075720111838023b2a75f2a784b9e7c35d1fdaf0eb97465a530d9c91564a345a rebuilt index: committed index: (308, 243931306) ID: 814b94f08ced5bd6f344c78ae2858e1306b3994cf79b14097c666594f9490dfe rebuilt index: committed index: (308, 25291226) ID: 9a762ee1d2fff73f324a6cbf6158ff28d0d107eed3cd43965c222e6e95eddbc7 rebuilt index: committed index: (308, 62284036) ID: 925fcb4fef1ed3d94251b8516b93d8aff57e4eeed7f422a7f8234f2371408647 rebuilt index: committed index: (308, 383059665) ID: ac64c8159f6cc29ee0b6c84a7c703a9fd28ae617fbc020593d24a5ad39ba1970 rebuilt index: committed index: (308, 143633698) ID: 5256b0e3d5e94614360e0061b129b4e1cce461f226acfb96759c25c6f260b59e rebuilt index: committed index: (308, 65795290) ID: c0d2aa3a80cd48d9225c4c682d6f76475a4cfef60ec6951896ff19b73bb05a40 rebuilt index: committed index: (308, 34193602) ID: 24a743ca77c263642ceadd3297c1b586e44fac2a8f2662091e3a7ef81877dc80 rebuilt index: committed index: (308, 217599204) ID: dd879f4358e68924a94330cbca8902e45ca8a8c4b3a6485074a23d2f267bfbe2 rebuilt index: committed index: (308, 187281410) ID: 9668a4cf9ff6ff418fd38a23cadac3f77c3259cb642584f0e6668237e96d34e3 rebuilt index: committed index: (308, 374603048) ID: 9b0cced61139edeb97f139a925e85b2c8de069ffaf232e3a0785bbf6733ae0f8 rebuilt index: committed index: (308, 57958293) ID: ab5f034eb3fc4cddc47cf365bfb72a8040ac7ee329c39615275aa3d16380ba90 rebuilt index: committed index: (308, 68281313) ID: d3a7aca4090cd665e25f7e78824eb199e41c4bd1b93dfc3652c5ac8d0cbb70d0 rebuilt index: committed index: (308, 247205158) ID: bb89b6e8c815db9b480fd89174e7eaec5e5e19e4a0362ffeb135b668985799a7 rebuilt index: committed index: (308, 223093111) ID: 297e8433db31f24c0a1b50ac6dd8c12a1b61946a861694a33c7d08b3e6636fd3 rebuilt index: committed index: (308, 106577195) ID: 28a51b1f1cab62cc623c18d3c99c6b5387d0bccd04c0822b2ef0af3bb038ab7e rebuilt index: committed index: (308, 184955172) ID: 7b0801b397a1dc0f65cd4a31462376304f717e9c5c2cf288a01730f1a3dd6c3f rebuilt index: committed index: (308, 201633797) ID: 2a3fb8e8130cb59f45d5007a01d56547bf4cd6457d22acde8b24ddfc7515a4bd rebuilt index: committed index: (308, 391620053) ID: da7ac05318ec71729499c7b8ada5a4b0e94b28315de4c1b50f2632edb30b8de3 rebuilt index: committed index: (308, 172609869) ID: 7fd68f779a508141dfb01970e373e64ce090fd3392a8094b75891e9f3cc1bf99 rebuilt index: committed index: (308, 380962649) ID: 1d372916dd850c78f6e96d4d2e21967cebcd452ebcdd4eab7a3b40640d9be947 rebuilt index: committed index: (308, 442149004) ID: 390c8cdcd4c6ce136e7a85464b6c2c7bc2e81c03239a09baa546d7cdca23fedc rebuilt index: committed index: (308, 359002924) ID: 71c4c640558f452b9f72fb134250f5076dc850adca9b7bf450dc64385393a92e rebuilt index: committed index: (308, 97954003) ID: 68ec58c684eebfe28c8781b4143aca2dc72befab4bfcf194f428b9a638169218 rebuilt index: committed index: (308, 339232051) ID: 8c4ff1ce8cefd319e9acbadf7a1feb636a3ae162f7e8ceaa818060df8ced2b7f rebuilt index: committed index: (308, 412782109) ID: cea7bdb839223f09c9e58055991a672d19d88916daf68ae2e2b4dcb4f0b70ed1 rebuilt index: committed index: (308, 67604237) ID: ce010d06878a35fd852311ab9dc898f593e4293d51d662220922fd02cc754e8f rebuilt index: committed index: (308, 206317329) ID: 5f45b322d2fb2a445b9cf0215644ae32931518ebfea932546a0a7f4e5dffc0a1 rebuilt index: committed index: (308, 482695602) ID: d25f02cb4c3c1908f80f9aac0d4ca206aa7d6c6832688eaacffe8342bc43ce3f rebuilt index: committed index: (308, 377248244) ID: 31724c2b99395dcfb1f4c9c06b7de0ffd3cd044ecd93ae31e75161c401ca95e0 rebuilt index: committed index: (308, 483826130) ID: 125137778b3be4aa0ab571238d423d746706594e7e299aa0aa555801a6b2d628 rebuilt index: committed index: (308, 77357122) ID: 5d7a4b0e60149ea01416417f3352105e309ecb016a53ec63e5f586d0765c9d84 rebuilt index: committed index: (308, 41528123) ID: 8c0ada8e5462dc62df980857a17123c78762b98dbd3d5763ce3bfc89f216c5df rebuilt index: committed index: (308, 179583845) ID: 081b2c80a040cd43f1f71e32bafed7b080a4be78a303c8381dc8cc90dc97865f rebuilt index: committed index: (308, 150334327) ID: db813c478c93448bd4267b3ef14466516a58a3c8b2362466aff749bd7d75e98f rebuilt index: committed index: (308, 303280742) ID: 424fb1bc91182aa7302dbe1e5dd162447b514722637990b6311f9ad22eeaa653 rebuilt index: committed index: (308, 146563636) ID: 366591550587e2012cf8fcc8a68323ed1702c16478276209aa8cc73695a77935 rebuilt index: committed index: (308, 501967452) ID: 422ee15e8175fd7232661bb4eeb3be2d585a856499280b5304eb8672b0b57d88 rebuilt index: committed index: (308, 101032139) ID: 5eb4458656d450a4a59cf5e5524cc22af819b7a7e2db948a26826a0b86a8f9ce rebuilt index: committed index: (308, 250294158) ID: 580ecd751046cf6fead53f41b2a79aea70afe56407f83f3bc439297cae0ad6e5 rebuilt index: committed index: (308, 409331104) ID: 19bd8f50fb4697a7da19d731ade0502a704f255bcff37f886199b2ab530c1fb6 rebuilt index: committed index: (308, 201109717) ID: c77a6f062e8ba01c4f89d492473ab1607d183a73015c85ecdcf7afc4a59ae5b3 rebuilt index: committed index: (308, 105297415) ID: d456aeb2458f623dc37565a6698de21cb6a139a0b85b7a769abb893f9c4d04ce rebuilt index: committed index: (308, 266703181) ID: 3a64a7dd27482555fc8175915ead9009d0b3064256f947d35ac57a0f082333d5 rebuilt index: committed index: (308, 125491747) ID: a97b4d091ddcb2263e903facfc59efd7e7f28c48b4ca3092cdd63efa22e9d0ab rebuilt index: committed index: (308, 159246928) ID: 38e2708d3bbcc1ded3eeca1f6b2e5eac36f05be24a0960c73223fa178b717685 rebuilt index: committed index: (308, 178140904) ID: d7d80eed37c85cb339bca153a01a5945f2596522c1c433424159d663285d863c rebuilt index: committed index: (308, 496276696) ID: dbe36ca001287c987d6d6bfbdd5e2d60ff3496b73897be1ca41d08eb71192c32 rebuilt index: committed index: (308, 408211219) ID: 8610030553de9ce59f46610c18cb11aa9609a502923000d5098c886c3f800036 rebuilt index: committed index: (308, 177790291) Finished full repository check, errors found. Then I checked the archive metadata: ha-idg-3:~ # borg check -vp --show-rc --archives-only /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup Starting archive consistency check... Enter passphrase for key /mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup: Analyzing archive 2024-01-31T05:00:02 (1/15) Analyzing archive 2024-02-12T05:00:05 (2/15) Analyzing archive 2024-03-24T06:00:01 (3/15) Analyzing archive 2024-03-31T06:00:03 (4/15) Analyzing archive 2024-04-07T06:00:11 (5/15) Analyzing archive 2024-04-11T06:00:14 (6/15) Analyzing archive 2024-04-12T06:00:13 (7/15) Analyzing archive 2024-04-13T06:00:20 (8/15) Analyzing archive 2024-04-14T06:00:15 (9/15) Analyzing archive 2024-04-15T06:00:14 (10/15) Analyzing archive 2024-04-16T06:00:10 (11/15) Analyzing archive 2024-04-17T11:11:55 (12/15) Analyzing archive 2024-04-18T06:00:08 (13/15) Analyzing archive 2024-04-19T06:00:18 (14/15) Analyzing archive 2024-04-23T06:00:12 (15/15) 8035 orphaned objects found! Archive consistency check complete, problems found. terminating with warning status, rc 1 currently I'm running a data consistency check, which has now passed 42%. I read the manpage and the help. I understand what a repository and an archive is. But what is a segment and what is a chunk ? And do you have any advice what I could do ? I tried to check the archive metadata separately for each archiv to find out which one is broken, but an archive metadata check needs to check all archives. Bernd -- Bernd Lentes SystemAdministrator Institute of Metabolism and Cell Death Helmholtz Zentrum M?nchen Building 25 office 122 Bernd.lentes at helmholtz-munich.de +49 89 3187 1241 Helmholtz Zentrum M?nchen ? Deutsches Forschungszentrum f?r Gesundheit und Umwelt (GmbH) Ingolst?dter Landstra?e 1, D-85764 Neuherberg, https://www.helmholtz-munich.de Gesch?ftsf?hrung: Prof. Dr. med. Dr. h.c. Matthias H. Tsch?p, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir?in Prof. Dr. Veronika von Messling Registergericht: Amtsgericht M?nchen HRB 6466 | USt-IdNr. DE 129521671 Helmholtz Zentrum M?nchen ? Deutsches Forschungszentrum f?r Gesundheit und Umwelt (GmbH) Ingolst?dter Landstra?e 1, D-85764 Neuherberg, https://www.helmholtz-munich.de Gesch?ftsf?hrung: Prof. Dr. med. Dr. h.c. Matthias H. Tsch?p, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir?in Prof. Dr. Veronika von Messling Registergericht: Amtsgericht M?nchen HRB 6466 | USt-IdNr. DE 129521671 From ed at edgewood.to Wed May 1 16:14:26 2024 From: ed at edgewood.to (Ed Blackman) Date: Wed, 1 May 2024 16:14:26 -0400 Subject: [Borgbackup] Consistent unexpectedly large deduplicated size In-Reply-To: <1714160678.e0708d@strabo.garrett> References: <1714160678.e0708d@strabo.garrett> Message-ID: <1714594402.a39c12@strabo.garrett> Any suggestions on how I can debug this? Ed On Fri, Apr 26, 2024 at 04:19:09PM -0400, Ed Blackman wrote: > I have a couple of backups going to the same repo. They are in the same repo because files sometimes move between backup sets. I have different prefixes for the backups so that I can have different prune policies. > > One of them has a consistent unexpectedly large deduplicated size, much larger than the sum size of the new files being added. > > Local and server borg are version 1.2.4 as packaged by Debian 12. > > Common command line options to both backups: > /usr/bin/borg create --chunker-params 19,23,21,4095 --stats --list > --show-rc --keep-exclude-tags --exclude-caches --exclude-if-present > .nobackup --noatime > > Backup A (the one with the unexpected size) had 45,364,409 bytes in 300 new files. Options specific to this backup were: > --compression=lz4 > --patterns-from=/home/backups/borg/snapshot/borg-tmp.20240426.1iExpB/hometmp-patterns > server:/path::a_backup=host at 2024-04-26T04:18:02 home > > Stats output for Backup A included: > > Number of files: 316114 > Original size Compressed size Deduplicated size > This archive: 32.14 GB 28.46 GB 215.98 MB > All archives: 43.55 TB 39.14 TB 667.32 GB > > Unique chunks Total chunks > Chunk index: 1069818 73111633 > > Backup B had 4,226,032 bytes in 10 new files. Options specific to this backup were: > --compression=zlib --exclude-from=/etc/backup/borg.all.exclude > server:/path::b_backup=host at 2024-04-26T04:18:02 home/user/projects > > Stats output for Backup B included: > Number of files: 79413 > Original size Compressed size Deduplicated size > This archive: 6.10 GB 5.08 GB 6.02 kB > All archives: 43.52 TB 39.12 TB 667.14 GB > > Unique chunks Total chunks > Chunk index: 1069619 72877206 > > > I'm not sure why backup A's deduplicated size is so much larger than the size of the new files in it. Are there any ideas about how to find out? > > -- > Ed?Blackman > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Ed?Blackman From tw at waldmann-edv.de Wed May 1 18:37:53 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 2 May 2024 00:37:53 +0200 Subject: [Borgbackup] Inconsistenses in repository In-Reply-To: References: Message-ID: > Killed stale lock ha-idg-3.scidom.de at 140947408721621.41751-0. > Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0. <===== > Removed stale exclusive roster lock for host ha-idg-3.scidom.de at 140947408721621 pid 41751 thread 0. <===== borg kills stale locks only if it can be really sure they are stale (== invalid and not needed any more). So, killing the lock is only fixing a leftover lock from a long dead borg process. > After the prune I got the following errors: > segment 10365 not found, but listed in compaction data ... That's strange. Either the compaction data is a bit off (that would be a minor issue) or segment files are gone that really should be still there (major issue). > and later on it seems the CIFS share is gone: > BlockingIOError: [Errno 11] Resource temporarily unavailable: '/mnt/nas/Daten/AG_BioInformatik/Technik/borg_backup/lock.roster' That might point to a severe issue with your NAS (hw or sw or network connection), which could be the root cause for all the troubles. Be aware that trying to "borg check --repair" the repo without fixing the root cause of the issues first might make the situation even worse. > Data integrity error: Segment entry checksum mismatch [segment 308, offset 22670132] That is bad. It means that a segment entry (could be a file content chunk, could be an archive metadata stream chunk) failed the crc32 check. Somehow you have data corruption there. Check the RAM and CPU of the NAS. Also check the SMART state of your disk(s). Check the power supply. > ID: 029179b026cc8f09fa5e23bc7c3a3a6fb414f8a227ca700a5883caa07ef80aef rebuilt index: committed index: (308, 262593301) That means that the on-disk committed repo index has an entry for a chunk with that ID and the in-memory rebuilt index hasn't. That means that the rebuild process did not process that chunk when rebuilding the index from the segment files. Could be because the segment file containing it is gone or because the crc32 check failed and the segment entry for that ID was considered invalid. > 8035 orphaned objects found! That could be a relatively harmless issue - or not. It just means that these objects are not used any more. But one reason because they are not used any more could be that metadata that pointed to them (used them) is now missing. > But what is a segment and what is a chunk ? Files are read and cut into smaller pieces (chunks) by borg's chunker using misc. algorithms (default: buzhash). E.g. if you have a VM disk of 100GB, borg might cut that file into ~50000 chunks of ~2MB each. A segment file (short: segment) contains some segment entries. A segment entry can be one of these types: - PUT + chunkid + chunkdata - DEL + chunkid - COMMIT Additionally, each segment entry has a crc32 checksum and an overall size. More precise infos about this can be found in the docs, "internals" section. > And do you have any advice what I could do ? Check for NAS hw/sw/network issues and fix them first. Make sure the network and NAS works correctly and stable. Then you can try "borg check --repair --progress -v REPO". It will try to rescue whatever is still there and bring the repo into a consistent state. It can't do wonders though and won't bring lost or corrupt data back. > I tried to check the archive metadata separately for each archiv to find out which one is broken, but > an archive metadata check needs to check all archives. Some checks can only be done for the whole repo / all archives. Especially the refcounting / orphans check, of course. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Wed May 1 19:03:26 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 2 May 2024 01:03:26 +0200 Subject: [Borgbackup] Consistent unexpectedly large deduplicated size In-Reply-To: <1714160678.e0708d@strabo.garrett> References: <1714160678.e0708d@strabo.garrett> Message-ID: > One of them has a consistent unexpectedly large deduplicated size, much larger than the sum size of the new files being added. Besides file content data, borg also backs up file metadata. Usually this isn't much, but in some cases it could contribute significantly to the overall size: - large xattrs or ACLS changing - files which you think are still "the same" (because that have same content data) have different metadata now and produce a different borg metadata stream (not well-deduplicating against the metadata streams of past backups). the usual reasons for this are: archiving atime, different mount point / different base dir, chown/chmod -R, different mtime ("touch"). - file discovery: dir tree traversal order, unstable inode numbers - file size vs. disk usage: sparse files can be very large size and use little disk space. you have excluded atimes from the backup archive, so it is not that in your specific case. > Are there any ideas about how to find out? You can check "borg debug ..." whether it has enough to analyze this. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From bernd.lentes at helmholtz-muenchen.de Sun May 5 07:21:58 2024 From: bernd.lentes at helmholtz-muenchen.de (Bernd Lentes) Date: Sun, 5 May 2024 11:21:58 +0000 Subject: [Borgbackup] Inconsistenses in repository In-Reply-To: References:

Message-ID: >-----Original Message----- >From: Borgbackup muenchen.de at python.org> On Behalf Of Thomas Waldmann >Sent: Thursday, May 2, 2024 12:38 AM >To: borgbackup at python.org >Subject: Re: [Borgbackup] Inconsistenses in repository >Then you can try "borg check --repair --progress -v REPO". >It will try to rescue whatever is still there and bring the repo into a >consistent state. It can't do wonders though and won't bring lost or >corrupt data back. Does it make sense to make a copy of the repo before checking ? Can I make a copy with a simple file copy (cp -av * ...) ? Bernd Helmholtz Zentrum M?nchen ? Deutsches Forschungszentrum f?r Gesundheit und Umwelt (GmbH) Ingolst?dter Landstra?e 1, D-85764 Neuherberg, https://www.helmholtz-munich.de Gesch?ftsf?hrung: Prof. Dr. med. Dr. h.c. Matthias H. Tsch?p, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir?in Prof. Dr. Veronika von Messling Registergericht: Amtsgericht M?nchen HRB 6466 | USt-IdNr. DE 129521671 From bernd.lentes at helmholtz-muenchen.de Sun May 5 07:23:45 2024 From: bernd.lentes at helmholtz-muenchen.de (Bernd Lentes) Date: Sun, 5 May 2024 11:23:45 +0000 Subject: [Borgbackup] Inconsistenses in repository In-Reply-To: References:

Message-ID: >-----Original Message----- >From: Borgbackup muenchen.de at python.org> On Behalf Of Thomas Waldmann >Sent: Thursday, May 2, 2024 12:38 AM >To: borgbackup at python.org >Subject: Re: [Borgbackup] Inconsistenses in repository > A segment entry can be one of these types: >- PUT + chunkid + chunkdata >- DEL + chunkid >- COMMIT What does PUT, DEL or COMMIT mean ? Bernd Helmholtz Zentrum M?nchen ? Deutsches Forschungszentrum f?r Gesundheit und Umwelt (GmbH) Ingolst?dter Landstra?e 1, D-85764 Neuherberg, https://www.helmholtz-munich.de Gesch?ftsf?hrung: Prof. Dr. med. Dr. h.c. Matthias H. Tsch?p, Dr. Michael Frieser | Aufsichtsratsvorsitzende: MinDir?in Prof. Dr. Veronika von Messling Registergericht: Amtsgericht M?nchen HRB 6466 | USt-IdNr. DE 129521671 From jorge.fabregas at gmail.com Sun May 5 12:00:31 2024 From: jorge.fabregas at gmail.com (=?UTF-8?Q?Jorge_F=C3=A1bregas?=) Date: Sun, 5 May 2024 12:00:31 -0400 Subject: [Borgbackup] Borg 2 - CLI Initial Impressions Message-ID: Hi everyone, I've been testing borg 2.0.0b8 (my first time with borg 2) and noticed the absence of the quintessential ASCII Summary Table when using --stats. I think this was kind of "eye candy" for someone new to the wonders of deduplication :) Now, we only get the Original and Deduplicated size for the particular archive. To see the old "All archives" summary, we need to use "rinfo". I find this interesting because when I first discovered borg many years ago, I found the original summary confusing because it gave the impression, at least to me, that borg: - compresses first and then deduplicates, while it actually deduplicates first and then compresses. - compresses all the time because "Compressed Size" showed data for every archive, even when chunks could have already been deduplicated/compressed previously. Are these two issues the main reasons for removing the old Summary, or was it just to make it more parsable? One final thing. I understand that compression reporting for a particular archive can be misleading, but why isn't there a "Compressed Size" item for the repository as a whole when doing "rinfo"? I could only infer it by looking at the "Storage quota," which kept me wondering. Thanks! -- Jorge From tw at waldmann-edv.de Sun May 5 12:21:18 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 5 May 2024 18:21:18 +0200 Subject: [Borgbackup] Inconsistenses in repository In-Reply-To: References:

Message-ID: <1dff30e7-3a06-418e-8d99-ccb06914a96d@waldmann-edv.de> >> Then you can try "borg check --repair --progress -v REPO". >> It will try to rescue whatever is still there and bring the repo into a >> consistent state. It can't do wonders though and won't bring lost or >> corrupt data back. > > Does it make sense to make a copy of the repo before checking ? > Can I make a copy with a simple file copy (cp -av * ...) ? If you can't risk losing the repo contents (or in general: making the situation worse somehonw), yes, making a backup of the repo makes sense. cp -a or rsync -avH are good ways to copy. rsync is superior in case something gets interrupted because you can the just re-issue the same rsync command. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sun May 5 14:09:53 2024 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 5 May 2024 20:09:53 +0200 Subject: [Borgbackup] Borg 2 - CLI Initial Impressions In-Reply-To: References: Message-ID: <1ec60088-a931-4725-9995-d817da3bda9e@waldmann-edv.de> > I've been testing borg 2.0.0b8 (my first time with borg 2) and noticed > the absence of the quintessential ASCII Summary Table when using > --stats. These are the reasons why some of the stats stuff is different now: - in general, updating the precise stats in borg is quite a pain. it's everywhere in the code and often it is in the way of easy changes / improvements of the code. - at some places, borg had lists or hashes containing (chunk id, chunk size, chunk csize) information. i had to drop the "csize" (compressed size of a chunk) because it also was in the way - e.g. of easy, repo-wide recompression without having to update the csize information at all the places where it was stored. - borg was often giving somehow unrelated infos. e.g. if you created an archive, it also gave you global repo-wide stats and not just the stats of the archive you just created. if that can be done quick, it's nice, but if it is slow (e.g. if it is a big repo, with lots of archives), it can be an unnecessary pain. > One final thing. I understand that compression reporting for a > particular archive can be misleading, but why isn't there a "Compressed > Size" item for the repository as a whole when doing "rinfo"? I could > only infer it by looking at the "Storage quota," which kept me wondering. Likely that was also dropped due to the csize removal. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jorge.fabregas at gmail.com Sun May 5 21:18:49 2024 From: jorge.fabregas at gmail.com (=?UTF-8?Q?Jorge_F=C3=A1bregas?=) Date: Sun, 5 May 2024 21:18:49 -0400 Subject: [Borgbackup] Borg 2 - CLI Initial Impressions In-Reply-To: <1ec60088-a931-4725-9995-d817da3bda9e@waldmann-edv.de> References: <1ec60088-a931-4725-9995-d817da3bda9e@waldmann-edv.de> Message-ID: On 5/5/24 2:09 PM, Thomas Waldmann wrote: > These are the reasons why some of the stats stuff is different now: Thanks for sharing the insights. If it's beneficial for the project's future, then that's definitely a positive direction. Thank you Thomas. - - Jorge From borgbackup_mailing_list at trendymail.fr Sun May 5 23:20:51 2024 From: borgbackup_mailing_list at trendymail.fr (borgbackup_mailing_list at trendymail.fr) Date: Mon, 6 May 2024 05:20:51 +0200 Subject: [Borgbackup] Simple message to say thanks! Message-ID: Hello! Just a quick message to say many, many thanks for Borg. :) @Thomas: discovered 6/8 months ago (better late than never...), your tool changed my "life". I use it to backup: - OpenVZ disk images - mails (5 years retention) - databases (dedup is awesome) - websites (ctime support is great) - and so on... # --- Tuning "--chunker-params" is a bit complicated but gives very good results. At this time, I use Borg 1.2.8 because "borg-linuxold64" binary runs with with OpenVZ 7. But, inside VM/CT, I plan to use Borg 1.4.x and 2.X as soon as a stable version will be available. # --- Sorry, I did not participate to beta versions but I have some non-critical data to backup (~800 GB). This way, I will discover Borg 2.x and report issues (or questions) if this is necessary and useful. Have a great night. :)