From tausciam at gmail.com Sat Oct 5 22:57:33 2019 From: tausciam at gmail.com (Michael Staggs) Date: Sat, 05 Oct 2019 21:57:33 -0500 Subject: [Borgbackup] extract restores deleted files Message-ID: <2089453.G7Ic7iLKPE@tausciam-solus> Here's the gist of what happened: I had backups for a few weeks. I got a new m.2 card and made a new backup right then. Then, I unplugged my old m.2 card and plugged in the new one. I put a base system on it and installed borg. I did a borg extract of the backup I had just made. Files that I had deleted days ago showed back up on my computer. Files I had moved to other locations showed up in their original locations. But, new files I had added since the backup before last also showed up. Any idea what went wrong? I typed the command to extract the archive I had just made. Thanks -- Sent from KMail[1] for Solus Linux[2] [Plasma Desktop] -------- [1] https://kde.org/applications/office/org.kde.kmail2 [2] https://getsol.us/home/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sun Oct 6 09:19:54 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 6 Oct 2019 15:19:54 +0200 Subject: [Borgbackup] extract restores deleted files In-Reply-To: <2089453.G7Ic7iLKPE@tausciam-solus> References: <2089453.G7Ic7iLKPE@tausciam-solus> Message-ID: <641a4eaf-cab0-1832-12ae-95b1642c8e7f@waldmann-edv.de> > I had backups for a few weeks. I got a new m.2 card and made a new > backup right then. Then, I unplugged my old m.2 card and plugged in the > new one. I put a base system on it and installed borg. I did a borg > extract of the backup I had just made. > > ? > > Files that I had deleted days ago showed back up on my computer. Files I > had moved to other locations showed up in their original locations. But, > new files I had added since the backup before last also showed up. > > Any idea what went wrong? I typed the command to extract the archive I > had just made. No idea, but without the specific and complete sequence of commands, that is hard to say. In general, borg does always do a full backup of whatever you give it (especially, there is no full / differential operation like with traditional backup tools). So if you backup the directory /m2 (as root user), the archive will contain all files below /m2 - no more, no less. It is impossible that the archive contains files that were not present any more / were not present yet below /m2 at the time of archive creation. They are not there, so how should they get into the archive? The contents of other archives that you made previously or later is not relevant when extracting one specific archive, the extracted files will exactly be the ones contained in the archive you are extracting. So what I suspect is that what you intended to do was not really what you have done. Ideas about what could have gone wrong (just guessing): - wildcards going wild - the shell will expand wildcards like * if you did not use quotes appropriately - maybe the mountpoint of your old m.2 card was not empty and contained an old state of the stuff also contained on the m.2 filesystem? maybe you did not mount the new m.2 filesystem before restoring the borg archive into the not-empty mountpoint directory? - maybe you did multiple extraction attempts without starting fresh from an empty directory? borg does not support transforming a non-empty extraction target directory into a specific state, you will get a mix up of whatever you extracted. - maybe you use multiple repositories / multiple backup media and confused them? or archive names? Hope that helps. If you still think there is some malfunction in borg, please try to reproduce it, write a script with all steps needed to reproduce and file a bug in the issue tracker with the script. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tausciam at gmail.com Sun Oct 6 10:48:30 2019 From: tausciam at gmail.com (Michael Staggs) Date: Sun, 06 Oct 2019 09:48:30 -0500 Subject: [Borgbackup] extract restores deleted files In-Reply-To: <641a4eaf-cab0-1832-12ae-95b1642c8e7f@waldmann-edv.de> References: <2089453.G7Ic7iLKPE@tausciam-solus> <641a4eaf-cab0-1832-12ae-95b1642c8e7f@waldmann-edv.de> Message-ID: <1630238.54HeQ47Udl@solus_laptop> On Sunday, October 6, 2019 8:19:54 AM CDT Thomas Waldmann wrote: > borg does not support transforming a non-empty extraction target > directory into a specific state, you will get a mix up of whatever you > extracted. Thank you. That's what I did and that's where my error was. I was thinking that I would get an exact copy of the backup borg had if I ran extract. Now I realize it adds those files back to the directories. It does not create a mirror image. In that case, it was entirely user error. It's still usable and no data was lost. I just have some extra stuff to get rid of, so no big deal. Thanks From kevin.elliott at javajedi.com Sat Oct 12 18:10:14 2019 From: kevin.elliott at javajedi.com (Kevin Elliott) Date: Sat, 12 Oct 2019 23:10:14 +0100 Subject: [Borgbackup] Prune and Transient files Message-ID: Hi folks, Something I am not understanding about Pruning, hope you can help please. Lets say I create a new archive every hour of the day, and throughout the day some files are created and some hours later they are deleted, e.g. file X created at 3am and deleted at 6pm. With a prune policy to keep-daily, is it possible to lose files? What happens to that transient file X that only existed intra-day, with daily pruning of archives is it lost? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From imperator at jedimail.de Sun Oct 13 02:42:32 2019 From: imperator at jedimail.de (Sascha Ternes) Date: Sun, 13 Oct 2019 08:42:32 +0200 Subject: [Borgbackup] Prune and Transient files In-Reply-To: References: Message-ID: Hi Kevin! Am 13.10.19 um 00:10 schrieb Kevin Elliott: > Something I am not understanding about Pruning, hope you can help > please. Lets say I create a new archive every hour of the day, and > throughout the day some files are created and some hours later they are > deleted, e.g. file X created at 3am and deleted at 6pm. With a prune > policy to keep-daily, is it possible to lose files? What happens to that > transient file X that only existed intra-day, with daily pruning of > archives is it lost?? You got it right. But it's the same with any time scale. Every file, after it is deleted, will eventually "get lost" if you prune your repo. Roughly speaking: Borg prune will keep the latest backup in all time scales. Doc says: borg prune deletes "all archives not matching any of the specified retention options". You may --keep-hourly 48 and --keep-daily 7 to hold back hourly backups for 2 days and daily backups for 7 days. If you constantly create more backups, pruning will delete oldest backups. In this example, 48 hours after you deleted the transient file, it will be removed from your repo. (Side note: Stricly speaking, Borg does not backup up "files", it backs up unique chunks of files, but that makes no difference in the result.) Sascha From billk at iinet.net.au Sun Oct 13 06:28:22 2019 From: billk at iinet.net.au (Bill Kenworthy) Date: Sun, 13 Oct 2019 18:28:22 +0800 Subject: [Borgbackup] re-storing old archives into new ones keeping the metadata information (e.g., time as considered by borg prune). Message-ID: <5354e4e8-1443-04d8-ba14-b79fcad1ac5c@iinet.net.au> Hi, ??? I am moving some old archives into new borg repos by extracting them then storing them into the new archive under the same name.? How can I set the metadata in the new archive to match the old so borg prune will consider them properly? BillK From billk at iinet.net.au Sun Oct 13 08:22:58 2019 From: billk at iinet.net.au (Bill Kenworthy) Date: Sun, 13 Oct 2019 20:22:58 +0800 Subject: [Borgbackup] re-storing old archives into new ones keeping the metadata information (e.g., time as considered by borg prune). In-Reply-To: <5354e4e8-1443-04d8-ba14-b79fcad1ac5c@iinet.net.au> References: <5354e4e8-1443-04d8-ba14-b79fcad1ac5c@iinet.net.au> Message-ID: <5835793d-ef72-0853-a9a3-731f420fd9fb@iinet.net.au> As usual after sending an email, I re-read the docs and noticed the "|--timestamp" option ... that's what I want!| | | |BillK| | | | | On 13/10/19 6:28 pm, Bill Kenworthy wrote: > Hi, > > ??? I am moving some old archives into new borg repos by extracting them > then storing them into the new archive under the same name.? How can I > set the metadata in the new archive to match the old so borg prune will > consider them properly? > > > BillK > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From jdc at uwo.ca Sun Oct 13 10:09:38 2019 From: jdc at uwo.ca (Dan Christensen) Date: Sun, 13 Oct 2019 14:09:38 +0000 Subject: [Borgbackup] Prune and Transient files In-Reply-To: (Sascha Ternes's message of "Sun, 13 Oct 2019 08:42:32 +0200") References: Message-ID: <8736fw7n4t.fsf@uwo.ca> Kevin, I like the --keep-within option and use "--keep-within 7d" so that nothing in the past week is pruned at all. The other rules then take effect for earlier backups. Dan On Oct 13, 2019, Sascha Ternes wrote: > Hi Kevin! > > Am 13.10.19 um 00:10 schrieb Kevin Elliott: >> Something I am not understanding about Pruning, hope you can help >> please. Lets say I create a new archive every hour of the day, and >> throughout the day some files are created and some hours later they are >> deleted, e.g. file X created at 3am and deleted at 6pm. With a prune >> policy to keep-daily, is it possible to lose files? What happens to that >> transient file X that only existed intra-day, with daily pruning of >> archives is it lost?? > > You got it right. But it's the same with any time scale. Every file, > after it is deleted, will eventually "get lost" if you prune your repo. > > Roughly speaking: Borg prune will keep the latest backup in all time scales. > Doc says: borg prune deletes "all archives not matching any of the > specified retention options". > You may --keep-hourly 48 and --keep-daily 7 to hold back hourly backups > for 2 days and daily backups for 7 days. If you constantly create more > backups, pruning will delete oldest backups. > In this example, 48 hours after you deleted the transient file, it will > be removed from your repo. > > (Side note: Stricly speaking, Borg does not backup up "files", it backs > up unique chunks of files, but that makes no difference in the result.) > > Sascha > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From jonikelee at gmail.com Fri Oct 18 19:55:13 2019 From: jonikelee at gmail.com (jonikelee at gmail.com) Date: Fri, 18 Oct 2019 19:55:13 -0400 Subject: [Borgbackup] borg delete failure with stack trace Message-ID: <20191018195513.16eceed2@lapdog> I am testing out borg, and accidentally did: borg create borgrepo::test4 ~ after 3 previous test archives were created using an --exclude-if-present .nobackup option. When I noticed the missing --exclude-if-present in the above create (because it was taking too long), I ^C'ed it, then tried: borg delete borgrepo::test4 and got this failure (<...> obfuscations added): Local Exception Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in main exit_code = archiver.run(args) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in run return set_ec(func(args)) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1257, in do_delete return self._delete_archives(args, repository) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1304, in _delete_archives logger.info(msg.format(format_archive(manifest.archives[archive_name]), i, len(archive_names))) File "/usr/lib/python3.7/site-packages/borg/helpers.py", line 244, in __getitem__ raise KeyError KeyError Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux Borg: 1.1.10 Python: CPython 3.7.4 msgpack: 0.5.6 PID: 10547 CWD: <...> sys.argv: ['/usr/bin/borg', 'delete', 'borgrepo::test4'] SSH_ORIGINAL_COMMAND: None Is this borgrepo now fried? If not, how can I recover it? From tw at waldmann-edv.de Sat Oct 19 16:52:16 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 19 Oct 2019 22:52:16 +0200 Subject: [Borgbackup] borg delete failure with stack trace In-Reply-To: <20191018195513.16eceed2@lapdog> References: <20191018195513.16eceed2@lapdog> Message-ID: <79c4c83c-adeb-85ae-68a8-cfbe2892288a@waldmann-edv.de> On 10/19/19 1:55 AM, jonikelee at gmail.com wrote: > I am testing out borg, and accidentally did: > > borg create borgrepo::test4 ~ > > after 3 previous test archives were created using an > --exclude-if-present .nobackup option. When I noticed the missing > --exclude-if-present in the above create (because it was taking too > long), I ^C'ed it, then tried: As the new archive was not finished / committed yet, the repo did not contain test4 yet. > borg delete borgrepo::test4 So you tried to delete a non-existing archive here. > and got this failure (<...> obfuscations added): > > Local Exception > Traceback (most recent call last): > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in main > exit_code = archiver.run(args) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in run > return set_ec(func(args)) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper > return method(self, args, repository=repository, **kwargs) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1257, in do_delete > return self._delete_archives(args, repository) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1304, in _delete_archives > logger.info(msg.format(format_archive(manifest.archives[archive_name]), i, len(archive_names))) > File "/usr/lib/python3.7/site-packages/borg/helpers.py", line 244, in __getitem__ > raise KeyError > KeyError Uuh, yeah, that is a rather unpretty way to say "archive test4 not found". I opened an issue for this: https://github.com/borgbackup/borg/issues/4793 > Is this borgrepo now fried? If not, how can I recover it? Everything is fine. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jonikelee at gmail.com Sun Oct 20 10:03:34 2019 From: jonikelee at gmail.com (jonikelee at gmail.com) Date: Sun, 20 Oct 2019 10:03:34 -0400 Subject: [Borgbackup] borg delete failure with stack trace In-Reply-To: <79c4c83c-adeb-85ae-68a8-cfbe2892288a@waldmann-edv.de> References: <20191018195513.16eceed2@lapdog> <79c4c83c-adeb-85ae-68a8-cfbe2892288a@waldmann-edv.de> Message-ID: <20191020100334.58aa9bd5@lapdog> On Sat, 19 Oct 2019 22:52:16 +0200 Thomas Waldmann wrote: > ... > > Is this borgrepo now fried? If not, how can I recover it? > > Everything is fine. > > The repo looked too big, as if it had at least partially done that mistaken create without excluding .nobackup. Although a check --verify-data did not complain. I decided to start from scratch for various unrelated reasons. However, I made a different mistake this time, and the repo won't allow creations of any more archives as a result. Here are the details: $ borg init --encryption none borgrepo $ borg create --exclude-if-present '.nobackup' --stats borgrepo::first /home/me $ borg create --exclude-if-present '.nobackup' --stats borgrepo::second /home/me all worked, then the mistake (the actual user name wasn't 'me', so the actual typo wasn't 'notme' - it was a single letter omission): $ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/notme /home/notme: [Errno 2] No such file or directory: '/home/notme' ---> ^C'ed it here - immediately after the error message, and retried without the mistake: $ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/me Archive third already exists it made the archive, or part of it, even though it had no source: $ borg list borgrepo first Sat, 2019-10-19 17:43:39 [32e1e882ef4948ba8a59abcfbfb655df71704f88366506ba307b8b8d12bc52a1] second Sat, 2019-10-19 17:45:12 [b0459333ddece1ce236aecf55f666c5ab639c596f088d86c2fed09596b448a07] third Sun, 2019-10-20 08:54:22 [e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4] but won't let go: $ borg delete borgrepo::third Local Exception Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in main exit_code = archiver.run(args) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in run return set_ec(func(args)) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1257, in do_delete return self._delete_archives(args, repository) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1310, in _delete_archives repository.commit(save_space=args.save_space) File "/usr/lib/python3.7/site-packages/borg/repository.py", line 438, in commit self.check_free_space() File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in check_free_space compact_working_space = sum(self.io.segment_size(segment) - free for segment, free in self.compact.items()) File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in compact_working_space = sum(self.io.segment_size(segment) - free for segment, free in self.compact.items()) File "/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in segment_size return os.path.getsize(self.segment_filename(segment)) File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or directory: '/<...>/borgrepo/data/0/7' Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux Borg: 1.1.10 Python: CPython 3.7.4 msgpack: 0.5.6 PID: 1650 CWD: /home/me/Desktop sys.argv: ['/usr/bin/borg', 'delete', '/<...>/borgrepo::third'] SSH_ORIGINAL_COMMAND: None $ borg info borgrepo::third Archive name: third Archive fingerprint: e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4 Comment: Hostname: <...> Username: me Time (start): Sun, 2019-10-20 08:54:22 Time (end): Sun, 2019-10-20 08:54:22 Duration: 0.07 seconds Number of files: 0 Command line: /usr/bin/borg create --exclude-if-present .nobackup --stats borgrepo::third /home/notme Utilization of maximum supported archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 0 B 0 B 429 B All archives: 480.81 MB 260.03 MB 129.74 MB Unique chunks Total chunks Chunk index: 2244 4823 Worst of all, further creates fail: $ borg create --exclude-if-present '.nobackup' --stats borgrepo::fourth /home/me Local Exception Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in main exit_code = archiver.run(args) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in run return set_ec(func(args)) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 573, in do_create create_inner(archive, cache) File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 538, in create_inner archive.save(comment=args.comment, timestamp=args.timestamp) File "/usr/lib/python3.7/site-packages/borg/archive.py", line 515, in save self.repository.commit() File "/usr/lib/python3.7/site-packages/borg/repository.py", line 438, in commit self.check_free_space() File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in check_free_space compact_working_space = sum(self.io.segment_size(segment) - free for segment, free in self.compact.items()) File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in compact_working_space = sum(self.io.segment_size(segment) - free for segment, free in self.compact.items()) File "/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in segment_size return os.path.getsize(self.segment_filename(segment)) File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or directory: '/.../borgrepo/data/0/7' Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux Borg: 1.1.10 Python: CPython 3.7.4 msgpack: 0.5.6 PID: 1669 CWD: /home/me/Desktop sys.argv: ['/usr/bin/borg', 'create', '--exclude-if-present', '.nobackup', '--stats', '/<...>/borgrepo::fourth', '/home/me'] SSH_ORIGINAL_COMMAND: None $ borg list borgrepo first Sat, 2019-10-19 17:43:39 [32e1e882ef4948ba8a59abcfbfb655df71704f88366506ba307b8b8d12bc52a1] second Sat, 2019-10-19 17:45:12 [b0459333ddece1ce236aecf55f666c5ab639c596f088d86c2fed09596b448a07] third Sun, 2019-10-20 08:54:22 [e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4] $ borg check --verify-data borgrepo $ echo $? 0 Sure, once I do adequate testing and have this all scripted, there won't be anymore such pilot errors. Still... From tw at waldmann-edv.de Sun Oct 20 11:04:24 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 20 Oct 2019 17:04:24 +0200 Subject: [Borgbackup] borg delete failure with stack trace In-Reply-To: <20191020100334.58aa9bd5@lapdog> References: <20191018195513.16eceed2@lapdog> <79c4c83c-adeb-85ae-68a8-cfbe2892288a@waldmann-edv.de> <20191020100334.58aa9bd5@lapdog> Message-ID: <0636907c-4507-1d6b-8db5-3e1d2650e412@waldmann-edv.de> >> Everything is fine. > > The repo looked too big, as if it had at least partially done that > mistaken create without excluding .nobackup. If you look at the repo before the next write operation, it will still have the uncommitted stuff in it from the interrupted borg create run. But these uncommitted segment files should be "rolled back" (deleted) to the last committed state at the next borg operation in this repo. > I decided to start from scratch for various unrelated reasons. > However, I made a different mistake this time, and the repo won't allow > creations of any more archives as a result. Here are the details: > > $ borg init --encryption none borgrepo > > $ borg create --exclude-if-present '.nobackup' --stats borgrepo::first /home/me > > $ borg create --exclude-if-present '.nobackup' --stats borgrepo::second /home/me > > all worked, then the mistake (the actual user name wasn't 'me', so the > actual typo wasn't 'notme' - it was a single letter omission): > > $ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/notme > /home/notme: [Errno 2] No such file or directory: '/home/notme' > > ---> ^C'ed it here - immediately after the error message, and > retried without the mistake: Well, no need to Ctrl-C here, it would have terminated soon after anyway (as it basically had nothing to do). Of course even if Ctrl-Cing it, it should not bring stuff into a problematic state... > $ borg create --exclude-if-present '.nobackup' --stats borgrepo::third /home/me > Archive third already exists > > it made the archive, or part of it, even though it had no source: > > $ borg list borgrepo > first Sat, 2019-10-19 17:43:39 > [32e1e882ef4948ba8a59abcfbfb655df71704f88366506ba307b8b8d12bc52a1] > second Sat, 2019-10-19 17:45:12 > [b0459333ddece1ce236aecf55f666c5ab639c596f088d86c2fed09596b448a07] > third Sun, 2019-10-20 08:54:22 > [e6277e63fe56ddbff1cd410dfe6826efd2e703a91a306d113e870a4df80c1bf4] > > but won't let go: > > $ borg delete borgrepo::third > Local Exception > Traceback (most recent call last): > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, > in main exit_code = archiver.run(args) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, > in run return set_ec(func(args)) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, > in wrapper return method(self, args, repository=repository, **kwargs) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1257, > in do_delete return self._delete_archives(args, repository) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 1310, > in _delete_archives repository.commit(save_space=args.save_space) > File "/usr/lib/python3.7/site-packages/borg/repository.py", line 438, > in commit self.check_free_space() > File "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, > in check_free_space compact_working_space = > sum(self.io.segment_size(segment) - free for segment, free in > self.compact.items()) File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in > compact_working_space = sum(self.io.segment_size(segment) - > free for segment, free in self.compact.items()) File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in > segment_size return os.path.getsize(self.segment_filename(segment)) > File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return > os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or > directory: '/<...>/borgrepo/data/0/7' > > Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 > 09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux > Borg: 1.1.10 Python: CPython 3.7.4 msgpack: 0.5.6 > PID: 1650 CWD: /home/me/Desktop > sys.argv: ['/usr/bin/borg', 'delete', > '/<...>/borgrepo::third'] SSH_ORIGINAL_COMMAND: None That looks like a bug. > $ borg info borgrepo::third > Number of files: 0 Yeah, as I said: nothing to do. > Worst of all, further creates fail: > > $ borg create --exclude-if-present '.nobackup' --stats borgrepo::fourth /home/me > Local Exception > Traceback (most recent call last): > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, > in main exit_code = archiver.run(args) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, > in run return set_ec(func(args)) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, > in wrapper return method(self, args, repository=repository, **kwargs) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 573, > in do_create create_inner(archive, cache) > File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 538, > in create_inner archive.save(comment=args.comment, > timestamp=args.timestamp) File > "/usr/lib/python3.7/site-packages/borg/archive.py", line 515, in save > self.repository.commit() File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 438, in > commit self.check_free_space() File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in > check_free_space compact_working_space = > sum(self.io.segment_size(segment) - free for segment, free in > self.compact.items()) File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 644, in > compact_working_space = sum(self.io.segment_size(segment) - > free for segment, free in self.compact.items()) File > "/usr/lib/python3.7/site-packages/borg/repository.py", line 1349, in > segment_size return os.path.getsize(self.segment_filename(segment)) > File "/usr/lib/python3.7/genericpath.py", line 50, in getsize return > os.stat(filename).st_size FileNotFoundError: [Errno 2] No such file or > directory: '/.../borgrepo/data/0/7' > > Platform: Linux <...> 5.3.6-1-MANJARO #1 SMP PREEMPT Sat Oct 12 > 09:30:05 UTC 2019 x86_64 Linux: arch Manjaro Linux > Borg: 1.1.10 Python: CPython 3.7.4 msgpack: 0.5.6 > PID: 1669 CWD: /home/me/Desktop > sys.argv: ['/usr/bin/borg', 'create', '--exclude-if-present', > '.nobackup', '--stats', '/<...>/borgrepo::fourth', > '/home/me'] SSH_ORIGINAL_COMMAND: None Same issue as above. I'll create an issue on github for this - thanks for finding it! -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From lazyvirus at gmx.com Mon Oct 21 16:42:50 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Mon, 21 Oct 2019 22:42:50 +0200 Subject: [Borgbackup] Prune and backup failure Message-ID: <20191021224250.6543fbbc@msi.defcon1.lan> Server: Debian buster Client: Debian stretch + backports BB libs installed as regular packages on both machines BB installed via pip3 on both machines ====================================== Hi listers, I had an issue with BB?; I don't know if it comes from the serveur or the client but it is not the first time, last time was a few months ago and the only solution I found to cure that was to erase this machine's repo and make another backup. This time, it made it's backup without any quirks, but when I tried to prune backups older than 3 months, it threw python errors finishing by: FileNotFoundError: [Errno 2] No such file or directory: '/BORG/data/0/510' (/BORG is where the repo is NFS mounted) Effectively, this file doesn't exist in the repo. Although there's nothing about memory or disks into log files, this machine being old may have disks unknown problem(s) (it boots on raid-1), I do not think there's a memory failure as it has been pushed up to OOM without any error thrown. So, I've got 2 questions: 1)- does it come from the client or from the server? 2)- what could be the reason of this? =============================== LAST LOG: NEW BACKUP # time BORG_BACKUP_SRV-4.sh PING borg.srv (192.168.1.200) 56(84) bytes of data. 64 bytes from borg.srv (192.168.1.200): icmp_seq=1 ttl=64 time=0.194 ms 64 bytes from borg.srv (192.168.1.200): icmp_seq=2 ttl=64 time=0.237 ms --- borg.srv ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.194/0.215/0.237/0.026 ms [ ok ] Stopping postgresql (via systemctl): postgresql.service. Local Exception Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/borg/repository.py", line 1317, in get_fd ts, fd = self.fds[segment] File "/usr/local/lib/python3.5/dist-packages/borg/lrucache.py", line 21, in __getitem__ value = self._cache[key] # raise KeyError if not found KeyError: 510 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 4501, in main exit_code = archiver.run(args) File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 4433, in run return set_ec(func(args)) File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 153, in wrapper kwargs['manifest'], kwargs['key'] = Manifest.load(repository, compatibility) File "/usr/local/lib/python3.5/dist-packages/borg/helpers.py", line 365, in load cdata = repository.get(cls.MANIFEST_ID) File "/usr/local/lib/python3.5/dist-packages/borg/repository.py", line 1073, in get return self.io.read(segment, offset, id) File "/usr/local/lib/python3.5/dist-packages/borg/repository.py", line 1437, in read fd = self.get_fd(segment) File "/usr/local/lib/python3.5/dist-packages/borg/repository.py", line 1319, in get_fd fd = open_fd() File "/usr/local/lib/python3.5/dist-packages/borg/repository.py", line 1300, in open_fd fd = open(self.segment_filename(segment), 'rb') FileNotFoundError: [Errno 2] No such file or directory: '/BORG/data/0/510' Platform: Linux srv-4 4.9.0-11-rt-686-pae #1 SMP PREEMPT RT Debian 4.9.189-3+deb9u1 (2019-09-20) i686 Linux: debian 9.11 Borg: 1.1.10 Python: CPython 3.5.3 msgpack: 0.5.6 PID: 19059 CWD: /root sys.argv: ['/usr/local/bin/borg', 'create', '--verbose', '--exclude-caches', '--show-rc', '--filter', 'AME', '--list', '--stats', '--checkpoint-interval', '600', '--compression', 'auto,zlib,6', '--exclude-from', '/usr/local/sbin/BORG_EXCLUSIONS.list', '::{hostname}-{utcnow}Z', '/'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 [ ok ] Starting postgresql (via systemctl): postgresql.service. real 0m39.998s user 0m2.184s sys 0m2.876s =============================== /LAST LOG: NEW BACKUP From tw at waldmann-edv.de Mon Oct 21 18:02:08 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 22 Oct 2019 00:02:08 +0200 Subject: [Borgbackup] Prune and backup failure In-Reply-To: <20191021224250.6543fbbc@msi.defcon1.lan> References: <20191021224250.6543fbbc@msi.defcon1.lan> Message-ID: <5afa79b1-557b-4ae1-2270-150ccb62e75b@waldmann-edv.de> > I had an issue with BB?; I don't know if it comes from the serveur or > the client but it is not the first time, last time was a few months ago > and the only solution I found to cure that was to erase this machine's > repo and make another backup. You could also try borg check [--repair]. > FileNotFoundError: [Errno 2] No such file or directory: > '/BORG/data/0/510' > (/BORG is where the repo is NFS mounted) > > Effectively, this file doesn't exist in the repo. OK, so there can be multiple causes: - assuming the repo index is valid and the file should be there: you have lost (at least) 1 file somehow. - the repo index is somehow invalid and pointing to a wrong segment file. - your fs (NFS) is not working correctly / reliably > Manifest.load(repository, compatibility) BTW, the missing segment seems to have the manifest (directory of archives). That's one of the last segment files produced in a backup run. It's rather bad if that is missing as it has all the pointers to the archives, but borg check --repair should be able to rebuild the manifest (takes a while though). That would also rebuild the repo index. > sys.argv: ['/usr/local/bin/borg', 'create', '--verbose', > '--exclude-caches', '--show-rc', '--filter', 'AME', '--list', '--stats', > '--checkpoint-interval', '600', '--compression', 'auto,zlib,6', > '--exclude-from', '/usr/local/sbin/BORG_EXCLUSIONS.list', > '::{hostname}-{utcnow}Z', '/'] SSH_ORIGINAL_COMMAND: None Unrelated to your problem, but just as a hint: checkpoint-interval 600 is rather frequent. this is good for unstable connections, but might impact performance if you have big indexes. zlib is not quite the fastest nor modern. maybe you rather want to consider the more modern zstd or lz4. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From lazyvirus at gmx.com Mon Oct 21 18:47:19 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Tue, 22 Oct 2019 00:47:19 +0200 Subject: [Borgbackup] Prune and backup failure In-Reply-To: <5afa79b1-557b-4ae1-2270-150ccb62e75b@waldmann-edv.de> References: <20191021224250.6543fbbc@msi.defcon1.lan> <5afa79b1-557b-4ae1-2270-150ccb62e75b@waldmann-edv.de> Message-ID: <20191022004719.01e131a9@msi.defcon1.lan> On Tue, 22 Oct 2019 00:02:08 +0200 Thomas Waldmann wrote: > > I had an issue with BB?; I don't know if it comes from the serveur or > > the client but it is not the first time, last time was a few months > > ago and the only solution I found to cure that was to erase this > > machine's repo and make another backup. > > You could also try borg check [--repair]. IIRC it failed the first time - I was in a hurry (no time to call help from the ML), so I zapped it and re-launched a backup that succeeded. But now I have time and I just launched a repair. > > FileNotFoundError: [Errno 2] No such file or directory: > > '/BORG/data/0/510' > > (/BORG is where the repo is NFS mounted) > > > > Effectively, this file doesn't exist in the repo. > > OK, so there can be multiple causes: That is what I feared. > - assuming the repo index is valid and the file should be there: you > have lost (at least) 1 file somehow. > > - the repo index is somehow invalid and pointing to a wrong segment > file. Noooo!? ;-) > - your fs (NFS) is not working correctly / reliably Nope, all other machines also use NFS and they never ever had a glitch. > > Manifest.load(repository, compatibility) > > BTW, the missing segment seems to have the manifest (directory of > archives). > > That's one of the last segment files produced in a backup run. > > It's rather bad if that is missing as it has all the pointers to the > archives, but borg check --repair should be able to rebuild the > manifest (takes a while though). That would also rebuild the repo > index. Ok, I just answered 'YES' to the repair and it did not throw anything at me, so now I'm waiting to see what will happen. I think once this will be settled, I'm gonna launch a long memory test, just in case (I've got another machine that blew a RAM without any good reason, except it's age (+15 ans)) As BB's very good with other machines (and very good, period.) > > sys.argv: ['/usr/local/bin/borg', 'create', '--verbose', > > '--exclude-caches', '--show-rc', '--filter', 'AME', '--list', > > '--stats', '--checkpoint-interval', '600', '--compression', > > 'auto,zlib,6', '--exclude-from', > > '/usr/local/sbin/BORG_EXCLUSIONS.list', '::{hostname}-{utcnow}Z', > > '/'] SSH_ORIGINAL_COMMAND: None > > Unrelated to your problem, but just as a hint: > > checkpoint-interval 600 is rather frequent. this is good for unstable > connections, but might impact performance if you have big indexes. At this time, the machines I backup have very low used disk space and I sometime forget to do the maintenance before launching BB - this delay was shortened especially about that, to make sure BB wouldn't restart from scratch if I have to kill it and relaunch it after maintenance. > zlib is not quite the fastest nor modern. At home, tests have shown that lz4 had an inferior score (less compression), so as I play here with my own money, I used the best compromise for my kind of data. zstd was good, but it took more time to achieve the compression (these machines are old, slow and are mostly mono-core.) > maybe you rather want to consider the more modern zstd or lz4. Thanks for your quick answer, I'll keep you aware about the repair results, but for now, I'm going to bed. Jean-Yves From lazyvirus at gmx.com Mon Oct 21 20:22:23 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Tue, 22 Oct 2019 02:22:23 +0200 Subject: [Borgbackup] Prune and backup failure In-Reply-To: <5afa79b1-557b-4ae1-2270-150ccb62e75b@waldmann-edv.de> References: <20191021224250.6543fbbc@msi.defcon1.lan> <5afa79b1-557b-4ae1-2270-150ccb62e75b@waldmann-edv.de> Message-ID: <20191022022223.7aafefa2@msi.defcon1.lan> On Tue, 22 Oct 2019 00:02:08 +0200 Thomas Waldmann wrote: I couldn't sleep, so I waited until the repair was finished. It worked, so I launch another backup, no problem and then launched my prune script, no problem too. I still duno from which machine it comes but as I told before, this client will be fully tested and pushed this WE as I do not have had any problem with other machines. Thanks Thomas. Jean-Yves > > I had an issue with BB?; I don't know if it comes from the serveur or > > the client but it is not the first time, last time was a few months > > ago and the only solution I found to cure that was to erase this > > machine's repo and make another backup. > > You could also try borg check [--repair]. > > > FileNotFoundError: [Errno 2] No such file or directory: > > '/BORG/data/0/510' > > (/BORG is where the repo is NFS mounted) > > > > Effectively, this file doesn't exist in the repo. > > OK, so there can be multiple causes: > > - assuming the repo index is valid and the file should be there: you > have lost (at least) 1 file somehow. > > - the repo index is somehow invalid and pointing to a wrong segment > file. > > - your fs (NFS) is not working correctly / reliably > > > Manifest.load(repository, compatibility) > > BTW, the missing segment seems to have the manifest (directory of > archives). > > That's one of the last segment files produced in a backup run. > > It's rather bad if that is missing as it has all the pointers to the > archives, but borg check --repair should be able to rebuild the > manifest (takes a while though). That would also rebuild the repo > index. > > > sys.argv: ['/usr/local/bin/borg', 'create', '--verbose', > > '--exclude-caches', '--show-rc', '--filter', 'AME', '--list', > > '--stats', '--checkpoint-interval', '600', '--compression', > > 'auto,zlib,6', '--exclude-from', > > '/usr/local/sbin/BORG_EXCLUSIONS.list', '::{hostname}-{utcnow}Z', > > '/'] SSH_ORIGINAL_COMMAND: None > > Unrelated to your problem, but just as a hint: > > checkpoint-interval 600 is rather frequent. this is good for unstable > connections, but might impact performance if you have big indexes. > > zlib is not quite the fastest nor modern. > > maybe you rather want to consider the more modern zstd or lz4. > > From jonikelee at gmail.com Wed Oct 23 11:49:05 2019 From: jonikelee at gmail.com (jonikelee at gmail.com) Date: Wed, 23 Oct 2019 11:49:05 -0400 Subject: [Borgbackup] borg delete failure with stack trace In-Reply-To: <0636907c-4507-1d6b-8db5-3e1d2650e412@waldmann-edv.de> References: <20191018195513.16eceed2@lapdog> <79c4c83c-adeb-85ae-68a8-cfbe2892288a@waldmann-edv.de> <20191020100334.58aa9bd5@lapdog> <0636907c-4507-1d6b-8db5-3e1d2650e412@waldmann-edv.de> Message-ID: <20191023114905.27941664@lapdog> I saw posted in github issue #4796 a request that I try 'borg delete --cache-only borgrepo'. I just tried that, then retried 'borg create --exclude-if-present '.nobackup' --stats borgrepo::fourth /home/me'. The result was the same stack trace as previously, with the FileNotFoundError about /<...>/borgrepo/data/0/7 From jronpaul at gmail.com Wed Oct 23 18:21:10 2019 From: jronpaul at gmail.com (mike abcdefg) Date: Wed, 23 Oct 2019 18:21:10 -0400 Subject: [Borgbackup] borg restore issue Message-ID: hello all. ive been using borg for a while. today my hardrive failed.. so i put in a new one. im trying to run borg list (repo) and getting perm denied running it as root. same error if i use the downloaded version, or the OS version. running linux mint. cat /etc/issue Linux Mint 19.2 Tina \n \l borg-linux64 1.1.10 borg-linux64 list /nas/backups/borg Exception ignored in: > Traceback (most recent call last): File "borg/repository.py", line 179, in __del__ AssertionError: cleanup happened in Repository.__del__ Local Exception Traceback (most recent call last): File "borg/repository.py", line 1317, in get_fd File "borg/lrucache.py", line 21, in __getitem__ KeyError: 50663 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "borg/archiver.py", line 4501, in main File "borg/archiver.py", line 4433, in run File "borg/archiver.py", line 151, in wrapper File "borg/repository.py", line 189, in __enter__ File "borg/repository.py", line 418, in open File "borg/repository.py", line 1352, in get_segment_magic File "borg/repository.py", line 1319, in get_fd File "borg/repository.py", line 1300, in open_fd PermissionError: [Errno 13] Permission denied: '/nas/backups/borg/data/5/50663' Platform: Linux Latitude-E6320 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 Linux: debian buster/sid Borg: 1.1.10 Python: CPython 3.5.7 msgpack: 0.5.6 PID: 18521 CWD: /root sys.argv: ['/home/user/Downloads/borg-linux64', 'list', '/nas/backups/borg'] SSH_ORIGINAL_COMMAND: None -------------- next part -------------- An HTML attachment was scrubbed... URL: From mateusz.kijowski at gmail.com Wed Oct 23 19:05:27 2019 From: mateusz.kijowski at gmail.com (Mateusz Kijowski) Date: Thu, 24 Oct 2019 01:05:27 +0200 Subject: [Borgbackup] borg restore issue In-Reply-To: References: Message-ID: > PermissionError: [Errno 13] Permission denied: '/nas/backups/borg/data/5/50663' Permission issues on a NFS mount ? czw., 24 pa? 2019 o 00:21 mike abcdefg napisa?(a): > > hello all. > ive been using borg for a while. > today my hardrive failed.. so i put in a new one. > im trying to run borg list (repo) > and getting perm denied running it as root. > same error if i use the downloaded version, or the OS version. > running linux mint. > > cat /etc/issue > Linux Mint 19.2 Tina \n \l > borg-linux64 1.1.10 > > borg-linux64 list /nas/backups/borg > Exception ignored in: > > Traceback (most recent call last): > File "borg/repository.py", line 179, in __del__ > AssertionError: cleanup happened in Repository.__del__ > Local Exception > Traceback (most recent call last): > File "borg/repository.py", line 1317, in get_fd > File "borg/lrucache.py", line 21, in __getitem__ > KeyError: 50663 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "borg/archiver.py", line 4501, in main > File "borg/archiver.py", line 4433, in run > File "borg/archiver.py", line 151, in wrapper > File "borg/repository.py", line 189, in __enter__ > File "borg/repository.py", line 418, in open > File "borg/repository.py", line 1352, in get_segment_magic > File "borg/repository.py", line 1319, in get_fd > File "borg/repository.py", line 1300, in open_fd > PermissionError: [Errno 13] Permission denied: '/nas/backups/borg/data/5/50663' > > Platform: Linux Latitude-E6320 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 > Linux: debian buster/sid > Borg: 1.1.10 Python: CPython 3.5.7 msgpack: 0.5.6 > PID: 18521 CWD: /root > sys.argv: ['/home/user/Downloads/borg-linux64', 'list', '/nas/backups/borg'] > SSH_ORIGINAL_COMMAND: None > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From jronpaul at gmail.com Thu Oct 24 13:35:41 2019 From: jronpaul at gmail.com (mike abcdefg) Date: Thu, 24 Oct 2019 13:35:41 -0400 Subject: [Borgbackup] borg restore issue In-Reply-To: References: Message-ID: It's actually cifs. It was working fine until my drive died on my computer On Wed, Oct 23, 2019, 7:05 PM Mateusz Kijowski wrote: > > PermissionError: [Errno 13] Permission denied: > '/nas/backups/borg/data/5/50663' > > Permission issues on a NFS mount ? > > czw., 24 pa? 2019 o 00:21 mike abcdefg napisa?(a): > > > > hello all. > > ive been using borg for a while. > > today my hardrive failed.. so i put in a new one. > > im trying to run borg list (repo) > > and getting perm denied running it as root. > > same error if i use the downloaded version, or the OS version. > > running linux mint. > > > > cat /etc/issue > > Linux Mint 19.2 Tina \n \l > > borg-linux64 1.1.10 > > > > borg-linux64 list /nas/backups/borg > > Exception ignored in: /nas/backups/borg>> > > Traceback (most recent call last): > > File "borg/repository.py", line 179, in __del__ > > AssertionError: cleanup happened in Repository.__del__ > > Local Exception > > Traceback (most recent call last): > > File "borg/repository.py", line 1317, in get_fd > > File "borg/lrucache.py", line 21, in __getitem__ > > KeyError: 50663 > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "borg/archiver.py", line 4501, in main > > File "borg/archiver.py", line 4433, in run > > File "borg/archiver.py", line 151, in wrapper > > File "borg/repository.py", line 189, in __enter__ > > File "borg/repository.py", line 418, in open > > File "borg/repository.py", line 1352, in get_segment_magic > > File "borg/repository.py", line 1319, in get_fd > > File "borg/repository.py", line 1300, in open_fd > > PermissionError: [Errno 13] Permission denied: > '/nas/backups/borg/data/5/50663' > > > > Platform: Linux Latitude-E6320 4.15.0-54-generic #58-Ubuntu SMP Mon Jun > 24 10:55:24 UTC 2019 x86_64 > > Linux: debian buster/sid > > Borg: 1.1.10 Python: CPython 3.5.7 msgpack: 0.5.6 > > PID: 18521 CWD: /root > > sys.argv: ['/home/user/Downloads/borg-linux64', 'list', > '/nas/backups/borg'] > > SSH_ORIGINAL_COMMAND: None > > > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hpj at urpla.net Mon Oct 28 12:20:50 2019 From: hpj at urpla.net (Hans-Peter Jansen) Date: Mon, 28 Oct 2019 17:20:50 +0100 Subject: [Borgbackup] Prune and backup failure In-Reply-To: <20191021224250.6543fbbc@msi.defcon1.lan> References: <20191021224250.6543fbbc@msi.defcon1.lan> Message-ID: <8457983.MvrNsNq8W5@xrated> Hi Jean-Yves, Am Montag, 21. Oktober 2019, 22:42:50 CET schrieb Bzzzz: > > This time, it made it's backup without any quirks, but when I tried to > prune backups older than 3 months, it threw python errors finishing by: > FileNotFoundError: [Errno 2] No such file or directory: > '/BORG/data/0/510' > (/BORG is where the repo is NFS mounted) > > Effectively, this file doesn't exist in the repo. > > Although there's nothing about memory or disks into log files, this > machine being old may have disks unknown problem(s) (it boots on raid-1), > I do not think there's a memory failure as it has been pushed up to OOM > without any error thrown. For the record, I saw this type of issues with flaky raid-1 setups: it effectively switches back and forth between the two disks with *all* kinds of adverse effects, such as vanishing files and directories... Check /proc/mdstat, setup mdadm mail option and check filesystems for other inconsistencies. Good luck, Pete From lazyvirus at gmx.com Mon Oct 28 13:04:38 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Mon, 28 Oct 2019 18:04:38 +0100 Subject: [Borgbackup] Prune and backup failure In-Reply-To: <8457983.MvrNsNq8W5@xrated> References: <20191021224250.6543fbbc@msi.defcon1.lan> <8457983.MvrNsNq8W5@xrated> Message-ID: <20191028180438.52ab7161@msi.defcon1.lan> On Mon, 28 Oct 2019 17:20:50 +0100 Hans-Peter Jansen wrote: Hi Hans > For the record, I saw this type of issues with flaky raid-1 setups: it > effectively switches back and forth between the two disks with *all* > kinds of adverse effects, such as vanishing files and directories... Nope, of course raid-1 and both disks were my first suspects but I found nothing in log files suggesting any raid failure or strange things involving one or both HDz, I also never lost a file or a directory. > Check /proc/mdstat, setup mdadm mail option and check filesystems for > other inconsistencies. There's a script doing this that sends an e-mail in case of problem but it was never activated. RAM is also ok, so my latter suspect is the power supply. I first dismissed it because it was no as old as the machine, but I already saw that in SOHO machines like this one (mostly because capacitors are never oversized compared to pro PSUs.) Thanks anyway, JY From joehesse at gmail.com Thu Nov 7 12:08:40 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Thu, 7 Nov 2019 11:08:40 -0600 Subject: [Borgbackup] Syncing directories with Borgbackup Message-ID: <764c0fbd-1b87-c015-9781-1d6aa3dd0cb8@gmail.com> I have done a successful borg backup of my home directory on PC1.? I want to use this backup to synchronize (like rsync) my home directory on PC1 to the home directory on PC2. I plan to do this by using "borg mount" on PC2 to mount my latest backup from PC1 to /tmp, on PC2, and then manually copy (cp -a -r -u) the home directory in /tmp to? /home. My question is: Is this the best way to do this? Thank you, Joe From imperator at jedimail.de Thu Nov 7 12:15:18 2019 From: imperator at jedimail.de (Sascha Ternes) Date: Thu, 7 Nov 2019 18:15:18 +0100 Subject: [Borgbackup] Syncing directories with Borgbackup In-Reply-To: <764c0fbd-1b87-c015-9781-1d6aa3dd0cb8@gmail.com> References: <764c0fbd-1b87-c015-9781-1d6aa3dd0cb8@gmail.com> Message-ID: <1e76ec98-47e8-1787-bd09-ae3148fffea6@jedimail.de> Hi Joseph, Am 07.11.19 um 18:08 schrieb Joseph Hesse: > I plan to do this by using "borg mount" on PC2 to mount my latest backup > from PC1 to /tmp, on PC2, and then manually copy (cp -a -r -u) the home > directory in /tmp to? /home. You should not mount to /tmp but to /mnt or a subdir of /tmp or /mnt. Besides this, "borg mount" or "borg extract" could be used. Sascha From lazyvirus at gmx.com Thu Nov 7 12:51:09 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Thu, 7 Nov 2019 18:51:09 +0100 Subject: [Borgbackup] Syncing directories with Borgbackup In-Reply-To: <764c0fbd-1b87-c015-9781-1d6aa3dd0cb8@gmail.com> References: <764c0fbd-1b87-c015-9781-1d6aa3dd0cb8@gmail.com> Message-ID: <20191107185109.38b1bff4@msi.defcon1.lan> On Thu, 7 Nov 2019 11:08:40 -0600 Joseph Hesse wrote: > I have done a successful borg backup of my home directory on PC1.? I > want to use this backup to synchronize (like rsync) my home directory > on PC1 to the home directory on PC2. > > I plan to do this by using "borg mount" on PC2 to mount my latest > backup from PC1 to /tmp, on PC2, and then manually copy (cp -a -r -u) > the home directory in /tmp to? /home. > > My question is: Is this the best way to do this? No, you should write scripts triggered by a crontab to automatize all that. On each of my machine, I've got /BORG on which is NFS mounted the machine's repo just before a new backup is performed and /SSHFS where I can mount a specific borg backup. You could go on with the same, then measure the time taken for each action (add 33-50% for security), and create crontabs as follow: PC1: * borg create /home /BORG * wait until avg time taken to backup+NN% is reached PC2: * borg mount backsrv::PC1/home_date (last backup) on /SSHFS * rsync from /SSHFS to /home OR cp -au /SSHFS/* /home Side effect of scripting: you do not care about databases (especially them, because all of them can't be faithfully binary-backup when running) because your scripts: check if a DB is running, if so, stop it, perform backup|restore, restart DB engine is it was running formerly. Jean-Yves From joehesse at gmail.com Fri Nov 8 08:27:59 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Fri, 8 Nov 2019 07:27:59 -0600 Subject: [Borgbackup] Newbie Questions Message-ID: <2b6ba9ed-fe9d-9f33-5c47-4d78147788c5@gmail.com> Thank you for answering my previous question. I am just learning how to use borgbackup.? I was not clear in my previous post as to how I am using borgbackup.? I own 2 identical laptops with the same OS, Fedora 31, and the same files in my home directory.? I need 2 laptops since I travel with them and there is the possibility that one of them dies due to a hardware malfunction. I do my work on PC1 and borgbackup PC1 to an external drive.? I run, from a script, borgbackup when I am finished working on PC1. There is no point in making it a cron job. In order to synchronize my home directory on PC1 with home on PC2, I plug the external drive into PC2, "borg mount" to /tmp/borg and then use rsync from the mounted directory to my real home directory on PC2. Since I travel to dangerous places there is also the possibility that both laptops and the external drive are stolen.? Before any travelling, I plan to use borgbackup with the cloud service rsync.net? If anything happens I will buy a new laptop and use the cloud service to restore the new laptop. I have 2 questions. 1.? When I set up the borg respository on my external drive I used "$ borg init --encryption=keyfile? localrepo"? The borg documentation says I should back up my key.? However, I just installed borgbackup on PC2 and was able to read my data without ever being asked for a key, just the passphrase.? When I use borg on the cloud, how should I do "borg init --encryption=??? user at hostname:backup"? 2.? I am comfortable with computer science data structures.? I do not understand the "chunking" that is used in "borg create". Please, if you can, point me to some documentation that explains this in some detail. I appreciate your help.? Thank you. Joe From tw at waldmann-edv.de Fri Nov 8 09:08:20 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Nov 2019 15:08:20 +0100 Subject: [Borgbackup] Newbie Questions In-Reply-To: <2b6ba9ed-fe9d-9f33-5c47-4d78147788c5@gmail.com> References: <2b6ba9ed-fe9d-9f33-5c47-4d78147788c5@gmail.com> Message-ID: <00d2366e-73d9-c6e5-8d8a-e6b4fe30abf1@waldmann-edv.de> > In order to synchronize my home directory on PC1 with home on PC2, I > plug the external drive into PC2, "borg mount" to /tmp/borg and then use > rsync from the mounted directory to my real home directory on PC2. That should be fine if you use rsync correctly for your use case (esp. using or not using the -a -H --delete options, read the man page). Be aware that borg mount does not support all metadata, but it might be good enough for what you need. If you need ACLs, xattrs, bsdflags or fs specific attributes, borg mount might be not good enough (just try it). > Since I travel to dangerous places there is also the possibility that > both laptops and the external drive are stolen.? Before any travelling, > I plan to use borgbackup with the cloud service rsync.net? If anything > happens I will buy a new laptop and use the cloud service to restore the > new laptop. At least for that repo, I'ld suggest using repokey. It might be also less problematic for using your usb drive repo from 2 machines. Of course use a good passphrase, so that when you lose your repo disk, the thief can not use your repokey. > 1.? When I set up the borg respository on my external drive I used "$ > borg init --encryption=keyfile? localrepo"? The borg documentation says > I should back up my key.? However, I just installed borgbackup on PC2 > and was able to read my data without ever being asked for a key, just > the passphrase. Then you either used no encryption (none) or repokey or you have the keyfile on PC2 also. If that is not the case, file a bug on github, please. > 2.? I am comfortable with computer science data structures.? I do not > understand the "chunking" that is used in "borg create". Please, if you > can, point me to some documentation that explains this in some detail. Chunking means to cut bigger files into smaller pieces (borg defaults to ~2MiB) before storing them into the repository. There should be some words about it in our docs. See also rolling hash / buzhash. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From kevhilton at gmail.com Sun Nov 10 23:33:34 2019 From: kevhilton at gmail.com (Kevin Hilton) Date: Sun, 10 Nov 2019 22:33:34 -0600 Subject: [Borgbackup] Return Code for borg break-lock Message-ID: Hi -- I've been using BB for about 3 years -- Simple setup which I use as a cron script to backup Laptop to NAS server. Everything working great so far. I recently discovered Vorta for BorgBackup and I really like the interface. I'm not sure how long this tool has been available however I'm in the process of testing this as an alternative to MacBook's Time Machine. Looking at my bash old script which I composed several years ago, I often ran the borg break-lock command prior to borg create command -- Here is snippet: REPOSITORY="ssh://@freenas/~/backup/borg/borgMacBookKMH-FreeNAS" OPTIONS="--progress --debug --stats --compression lzma,9 --exclude-caches" #Ive taken out the --progress option when run in a cron job export BORG_PASSPHRASE="****" borg break-lock $REPOSITORY borg create $OPTIONS $REPOSITORY::`hostname`-`date +%Y-%m-%d--%H:%M:%S` \ /Users/ \ --exclude '*/.cache/*' \ --exclude '*/.Cache/' \ --exclude '*/.Trash/' \ ... .... I remember having to add this break-lock command since when I was composing this script several years ago, the script would hang either due to a network issue or ssh issue or some other issue. After hanging, the repository would be locked and subsequent cron jobs running the script wouldn't actually backup anything. There also seems to not be any return code for this function -- or maybe I'm not seeing it when it runs with success -- like a zero. My questions are as follows: 1. Does borg break-lock return 0 on success? 2. Do I still require this statement with newer borg versions? I'm currently running version 1.1.10 3. Vorta has an option that I can run Pre Backup Command -- but requires a return code of 0 to proceed with the borg backup command. I was thinking of running the break-lock command here, however when trying this particular method, the program notifies me it receives a non-zero return code. 4. Maybe I'm just doing things incorrectly. I'm open to comments. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joehesse at gmail.com Sun Nov 17 12:05:46 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Sun, 17 Nov 2019 11:05:46 -0600 Subject: [Borgbackup] Determine if a chunk should be added to repository Message-ID: Hi, I've been reading all I can find about Deduplication.? The explanations say that hashes are used to determine if two chunks are identical or different.? Even though it is very rare, two chunks can have the same hash and be different.? So if chunk1's hash matches the hash of some chunk in the repository, it seems to me that a byte-by-byte comparison has to be done before deciding whether chunk1 should not be added to the repository.? Is this the case with Borg? Thank you, Joe From public at enkore.de Sun Nov 17 12:27:06 2019 From: public at enkore.de (Marian Beermann) Date: Sun, 17 Nov 2019 18:27:06 +0100 Subject: [Borgbackup] Determine if a chunk should be added to repository In-Reply-To: References: Message-ID: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> No. Borg uses content-addressed storage and does not handle hash collisions. That's why Borg uses a 256 bit cryptographic hash (HMAC-SHA2, BLAKE2). -Marian Am 17.11.19 um 18:05 schrieb Joseph Hesse: > Hi, > > I've been reading all I can find about Deduplication.? The explanations > say that hashes are used to determine if two chunks are identical or > different.? Even though it is very rare, two chunks can have the same > hash and be different.? So if chunk1's hash matches the hash of some > chunk in the repository, it seems to me that a byte-by-byte comparison > has to be done before deciding whether chunk1 should not be added to the > repository.? Is this the case with Borg? > > Thank you, > > Joe > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From joehesse at gmail.com Mon Nov 18 16:24:41 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Mon, 18 Nov 2019 15:24:41 -0600 Subject: [Borgbackup] Determine if a chunk should be added to repository In-Reply-To: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> References: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> Message-ID: <646c1994-2053-8f92-48cd-51255458104b@gmail.com> Thank you for your reply. Is the unique id for a content-addressed storage object a 256 bit hash?? Collisions are always possible, why doesn't Borg handle them, or, am I not understanding the idea behind content-addressed storage. Thank you, Joe On 11/17/19 11:27 AM, Marian Beermann wrote: > No. Borg uses content-addressed storage and does not handle hash > collisions. That's why Borg uses a 256 bit cryptographic hash > (HMAC-SHA2, BLAKE2). > > -Marian > > Am 17.11.19 um 18:05 schrieb Joseph Hesse: >> Hi, >> >> I've been reading all I can find about Deduplication.? The explanations >> say that hashes are used to determine if two chunks are identical or >> different.? Even though it is very rare, two chunks can have the same >> hash and be different.? So if chunk1's hash matches the hash of some >> chunk in the repository, it seems to me that a byte-by-byte comparison >> has to be done before deciding whether chunk1 should not be added to the >> repository.? Is this the case with Borg? >> >> Thank you, >> >> Joe >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Mon Nov 18 16:57:12 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 18 Nov 2019 22:57:12 +0100 Subject: [Borgbackup] Determine if a chunk should be added to repository In-Reply-To: <646c1994-2053-8f92-48cd-51255458104b@gmail.com> References: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> <646c1994-2053-8f92-48cd-51255458104b@gmail.com> Message-ID: <57ae7278-ce24-6347-0356-90f59aa97f48@waldmann-edv.de> Hash collisions have already been discussed / explained there: https://github.com/borgbackup/borg/issues/170 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From public at enkore.de Tue Nov 19 10:20:25 2019 From: public at enkore.de (Marian Beermann) Date: Tue, 19 Nov 2019 16:20:25 +0100 Subject: [Borgbackup] Determine if a chunk should be added to repository In-Reply-To: <646c1994-2053-8f92-48cd-51255458104b@gmail.com> References: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> <646c1994-2053-8f92-48cd-51255458104b@gmail.com> Message-ID: <6827f532-f0e0-3fa0-188f-6bb70fd74643@enkore.de> Hash collisions are always possible, but depending on the size of the hash and the number of items they can be highly improbable. For a hash with N output values and k inputs you can approximate the likelihood of at least one collision by k?/2N (for large k, N). For example, 1e12?/2**257 is ~4e-54. -Marian Am 18.11.19 um 22:24 schrieb Joseph Hesse: > Thank you for your reply. > Is the unique id for a content-addressed storage object a 256 bit hash?? > Collisions are always possible, why doesn't Borg handle them, or, am I > not understanding the idea behind content-addressed storage. > Thank you, Joe > > > On 11/17/19 11:27 AM, Marian Beermann wrote: >> No. Borg uses content-addressed storage and does not handle hash >> collisions. That's why Borg uses a 256 bit cryptographic hash >> (HMAC-SHA2, BLAKE2). >> >> -Marian >> >> Am 17.11.19 um 18:05 schrieb Joseph Hesse: >>> Hi, >>> >>> I've been reading all I can find about Deduplication.? The explanations >>> say that hashes are used to determine if two chunks are identical or >>> different.? Even though it is very rare, two chunks can have the same >>> hash and be different.? So if chunk1's hash matches the hash of some >>> chunk in the repository, it seems to me that a byte-by-byte comparison >>> has to be done before deciding whether chunk1 should not be added to the >>> repository.? Is this the case with Borg? >>> >>> Thank you, >>> >>> Joe >>> >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup > > From public at enkore.de Tue Nov 19 10:24:26 2019 From: public at enkore.de (Marian Beermann) Date: Tue, 19 Nov 2019 16:24:26 +0100 Subject: [Borgbackup] Determine if a chunk should be added to repository In-Reply-To: <646c1994-2053-8f92-48cd-51255458104b@gmail.com> References: <6b7951e3-63c7-871a-ace6-912370f4541c@enkore.de> <646c1994-2053-8f92-48cd-51255458104b@gmail.com> Message-ID: It is in my opinion much more likely (and more in line with history) that the hash function is broken than a random collision happening. I should point out that in the case of HMAC the requirements on the underlying hash function are quite weak ("has to look like a PRF"), to the point that you can still use the thoroughly broken MD5 in HMAC *today* and we have no known attacks on that construction (in fact, a proof was developed after MD5 was broken to show that HMAC-MD5 is still secure). I.e. being able to find collisions in the underlying hash function of a HMAC is not useful in breaking HMAC. It's a construction that has proven very resilient both in theory and practice. Borg uses HMAC-SHA2 or BLAKE2. -Marian From joehesse at gmail.com Mon Nov 25 10:24:26 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Mon, 25 Nov 2019 09:24:26 -0600 Subject: [Borgbackup] Two Users, 1 repository Message-ID: Hi, My wife and I use different linux laptops and we want to share a repository by moving an external hard drive from one laptop to the other.? Sharing a repository seems good to me since we have many identical files on our computers. Will this cause any problems, especially since .config/borg and .cache/borg will be different on each laptop? Thank you in advance for your response and for previous responses. Joe From lazyvirus at gmx.com Mon Nov 25 10:44:09 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Mon, 25 Nov 2019 16:44:09 +0100 Subject: [Borgbackup] Two Users, 1 repository In-Reply-To: References: Message-ID: <20191125164409.5a98740c@msi.defcon1.lan> On Mon, 25 Nov 2019 09:24:26 -0600 Joseph Hesse wrote: Hi Jo, > My wife and I use different linux laptops and we want to share a > repository by moving an external hard drive from one laptop to the > other.? Sharing a repository seems good to me since we have many > identical files on our computers. This may not be the idea of the year (though, it'll end in a few days;-p) see: https://borgbackup.readthedocs.io/en/stable/faq.html 1st paragraph. In fact, I don't think that deduplication would bring you anything interesting in terms of taken place on the HD as far as you just have two machines to backup (and note that because of the exclusive lock, you won't be able to backup them simultaneously.) IMHO, Compression is enough for what you're after. > Will this cause any problems, especially since .config/borg and > .cache/borg will be different on each laptop? As the FAQ says, it'll rebuild each local cache on each an every backup. > Thank you in advance for your response and for previous responses. You're welcome. Jean-Yves From archont at gmx.com Mon Nov 25 12:49:43 2019 From: archont at gmx.com (archont) Date: Mon, 25 Nov 2019 17:49:43 +0000 Subject: [Borgbackup] Two Users, 1 repository In-Reply-To: <20191125164409.5a98740c@msi.defcon1.lan> References: <20191125164409.5a98740c@msi.defcon1.lan> Message-ID: <77443aa135aca43399d5840ffa349c63@gmx.com> November 25, 2019 4:44 PM, "Bzzzz" wrote: > On Mon, 25 Nov 2019 09:24:26 -0600 > > In fact, I don't think that deduplication would bring you anything > interesting in terms of taken place on the HD as far as you just have > two machines to backup (and note that because of the exclusive lock, you > won't be able to backup them simultaneously.) > IMHO he can put the cache + config directly on the same drive as the backup. Also, this supposedly is an _external_ drive (USB?), so repository lock won't be an issue. What I would see as a bigger issue is inconvenience - having to attach the drive before backup. But if they are used to it, why not. BR ArchOnt From tw at waldmann-edv.de Mon Nov 25 12:54:49 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 25 Nov 2019 18:54:49 +0100 Subject: [Borgbackup] Two Users, 1 repository In-Reply-To: <77443aa135aca43399d5840ffa349c63@gmx.com> References: <20191125164409.5a98740c@msi.defcon1.lan> <77443aa135aca43399d5840ffa349c63@gmx.com> Message-ID: <205A0711-F114-48C1-AAFB-39336758F8AF@waldmann-edv.de> >IMHO he can put the cache + config directly on the same drive as the >backup. no, don't do that. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From lazyvirus at gmx.com Mon Nov 25 13:05:43 2019 From: lazyvirus at gmx.com (Bzzzz) Date: Mon, 25 Nov 2019 19:05:43 +0100 Subject: [Borgbackup] Two Users, 1 repository In-Reply-To: <77443aa135aca43399d5840ffa349c63@gmx.com> References: <20191125164409.5a98740c@msi.defcon1.lan> <77443aa135aca43399d5840ffa349c63@gmx.com> Message-ID: <20191125190543.557eb03e@msi.defcon1.lan> On Mon, 25 Nov 2019 17:49:43 +0000 "archont" wrote: > IMHO he can put the cache + config directly on the same drive as the > backup. Not a bad idea :) I would also suggest to backup root and users .ssh/ and cache+config in an encrypted zip file into another directory on the HD, just in case it moves to a server one of these days. > Also, this supposedly is an _external_ drive (USB?), so > repository lock won't be an issue. Whoops. BTW, is the impossibility to backup multiple machines at once a big no-no, programmatically speaking, or is there a way to circumvent this limitation ? Jean-Yves From archont at gmx.com Mon Nov 25 15:00:14 2019 From: archont at gmx.com (archont) Date: Mon, 25 Nov 2019 20:00:14 +0000 Subject: [Borgbackup] Two Users, 1 repository In-Reply-To: <205A0711-F114-48C1-AAFB-39336758F8AF@waldmann-edv.de> References: <205A0711-F114-48C1-AAFB-39336758F8AF@waldmann-edv.de> <20191125164409.5a98740c@msi.defcon1.lan> <77443aa135aca43399d5840ffa349c63@gmx.com> Message-ID: November 25, 2019 6:54 PM, "Thomas Waldmann" wrote: >> IMHO he can put the cache + config directly on the same drive as the >> backup. > > no, don't do that. Referring to these parts of BORG Doc? == (7.1) (Each client having its own config and cache) When the above attack model is extended to include multiple clients independently updating the same repository,then Borg fails to provide confidentiality. == and == (Clients sharing the same config & cache) Therefore in a multiple-client scenario a repository can trick a client into reusing counter values byignoring counter reservations and replaying the manifest (which will fail if the client has seen a more recent man-ifest or has a more recent nonce reservation). If the repository is untrusted, but a trusted synchronization channelexists between clients, the security database could be synchronized between them over said trusted channel. Thisis not part of Borg?s functionality. == From jan.klepek at gmail.com Thu Nov 28 14:09:05 2019 From: jan.klepek at gmail.com (Jan Klepek) Date: Thu, 28 Nov 2019 20:09:05 +0100 Subject: [Borgbackup] local and remote repository Message-ID: Hi, I have server where I backup lot of small files to remote storage box (100-150GB of source data). Backup process (either, first one or consecutive backup) takes long time (3-4hours), however deduplication is superb (60-70%). I have either tar.gz local files before backup (this reduces backup time to 1 hour) however deduplication is not so great (10%). So I'm looking for solution, create borg backup locally and move results to storage box. However I want to keep locally 1 full/monthly backup and few daily backups, where on storage server I want to keep 5-6 monthly backup, few weekly and few daily backups. Is this possible to achieve with borg? Thank you, Jan Klepek -------------- next part -------------- An HTML attachment was scrubbed... URL: From clickwir at gmail.com Thu Nov 28 16:01:48 2019 From: clickwir at gmail.com (Zack Coffey) Date: Thu, 28 Nov 2019 14:01:48 -0700 Subject: [Borgbackup] local and remote repository In-Reply-To: References: Message-ID: Borgmatic might be able to help accomplish this. I use it to define 2 destinations. 1 is local and the other is remote. Been running great for over a year now. On Thu, Nov 28, 2019 at 12:09 PM Jan Klepek wrote: > Hi, > > I have server where I backup lot of small files to remote storage box > (100-150GB of source data). Backup process (either, first one or > consecutive backup) takes long time (3-4hours), however deduplication is > superb (60-70%). > > I have either tar.gz local files before backup (this reduces backup time > to 1 hour) however deduplication is not so great (10%). > > So I'm looking for solution, create borg backup locally and move results > to storage box. However I want to keep locally 1 full/monthly backup and > few daily backups, where on storage server I want to keep 5-6 monthly > backup, few weekly and few daily backups. > > Is this possible to achieve with borg? > > Thank you, > Jan Klepek > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joehesse at gmail.com Sat Nov 30 04:19:50 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Sat, 30 Nov 2019 03:19:50 -0600 Subject: [Borgbackup] Cache Question Message-ID: <52b63186-59a8-4c38-281f-81171a582d65@gmail.com> Hi, I use borg to back up my entire home directory.? It seems to me that I should exclude .cache/borg and .config/borg.? My reasoning is that if I have to restore my entire home directory into a new computer, borg will recreate these directories and restoring the old ones will interfere with the restoration.? Is this correct? Thank you, Joe From felix.schwarz at oss.schwarz.eu Sat Nov 30 04:50:01 2019 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Sat, 30 Nov 2019 10:50:01 +0100 Subject: [Borgbackup] Cache Question In-Reply-To: <52b63186-59a8-4c38-281f-81171a582d65@gmail.com> References: <52b63186-59a8-4c38-281f-81171a582d65@gmail.com> Message-ID: <27ce6127-9d88-d247-6b17-d522325fc666@oss.schwarz.eu> Am 30.11.19 um 10:19 schrieb Joseph Hesse: > I use borg to back up my entire home directory.? It seems to me that I should > exclude .cache/borg and .config/borg.? My reasoning is that if I have to > restore my entire home directory into a new computer, borg will recreate these > directories and restoring the old ones will interfere with the restoration.? > Is this correct? You should be able to exclude ~/.cache entirely (assuming all applications use that directory properly). I still backup ~/.config/borg because I have multiple (encrypted) borg repos and that way I still have these keys covered. If you only use a single borg repo on that computer you don't need ~/.config/borg. Felix From joehesse at gmail.com Sun Dec 8 07:42:19 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Sun, 8 Dec 2019 06:42:19 -0600 Subject: [Borgbackup] Question about using BorgBackup on your site Message-ID: <9817d6c3-c6d4-2504-e1b6-50b9e6776530@gmail.com> Hi, I have the impression that if I subscribe to rsync.net using Borg, all I get is an empty space.? I use this empty space just like I use Borg on an external hard drive except the repository points to your space.? There is no Borg specific software running on rsync.net? Is this correct? Thank you, Joe From andrea.francesconi at gmail.com Sun Dec 8 09:15:12 2019 From: andrea.francesconi at gmail.com (fRANz) Date: Sun, 8 Dec 2019 15:15:12 +0100 Subject: [Borgbackup] Question about using BorgBackup on your site In-Reply-To: <9817d6c3-c6d4-2504-e1b6-50b9e6776530@gmail.com> References: <9817d6c3-c6d4-2504-e1b6-50b9e6776530@gmail.com> Message-ID: On Sun, Dec 8, 2019 at 1:42 PM Joseph Hesse wrote: > There is no Borg specific software running on rsync.net Is this correct? There is: "We support legacy borg versions for backward compatibility - currently 0.29 and 1.x branches" https://www.rsync.net/products/attic.html -f From joehesse at gmail.com Mon Dec 9 04:31:01 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Mon, 9 Dec 2019 03:31:01 -0600 Subject: [Borgbackup] Change repository id Message-ID: <992fa1d8-50a6-d4e8-ae75-2814e2bc7c06@gmail.com> Suppose I have two distinct repositories, say repo1 and repo2, as described in the Borg 1.1.10 FAQ.? Again suppose that whenever I do a "borg create" on one repo1 I do the same on repo2.? If one of the repos dies, say repo1 because of a hardware failure, I only have one repo with all my archives. In an attempt to get back two repos, I get new hardware and copy the contents of repo2 to the new hardware.? For good measure I delete the cache on my local computer.? According to the FAQ, there would be data loss since both repos now have the same ID. Can I change the repo ID on the new hardware?? How do I proceed from here to get back to two distinct repositories? Thank you, Joe From tw at waldmann-edv.de Mon Dec 9 05:15:35 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 9 Dec 2019 11:15:35 +0100 Subject: [Borgbackup] Change repository id In-Reply-To: <992fa1d8-50a6-d4e8-ae75-2814e2bc7c06@gmail.com> References: <992fa1d8-50a6-d4e8-ae75-2814e2bc7c06@gmail.com> Message-ID: <9f4143e0-7fcd-521c-dd90-e96d0af00af6@waldmann-edv.de> There are mainly 2 issues with cloning (making a 1:1 copy of) a repo: - getting it "somehow working" - this needs changing of the repo id (see repo_dir/config to a unique new value) before you ever use the copy. - getting it "cryptographically as secure as it can be" is maybe not easily possible as the data in repo1 and repo2 is encrypted with the same AES key and the same counter/IV values (AES CTR mode) and one can not assume that all future repo writes will be totally identical. And if they aren't, you are breaking AES security due to counter/IV reuse (same key, same counter, different plaintext). There are also some related infos about this in the borg docs/faq. There is no code in borg yet to transfer an archive from one repo to another, but you could do extract / create for all archives, which might take quite a while depending on your archive count/size. A hypothetical transfer command could be a bit better, but not much, as it would have to (at least) authenticate/decrypt/encrypt/authenticate all data. It could save writing all files to disk / reading them from disk again, though. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From charlesmccrea at gmail.com Mon Dec 23 00:04:47 2019 From: charlesmccrea at gmail.com (Charles Mccrea) Date: Mon, 23 Dec 2019 00:04:47 -0500 Subject: [Borgbackup] Raspberry pi 4 for borg server Message-ID: Hello Forum, Would the new raspberry pi 4 with 4gb ram and separate nic at gigabit speed be enough for an smb to use borg backup on prem on a Pi 4 to send data off-site to a receiving server with borg installed? Does anyone have experience or tested this yet? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joehesse at gmail.com Mon Dec 23 05:08:43 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Mon, 23 Dec 2019 04:08:43 -0600 Subject: [Borgbackup] Newbie questions Message-ID: <78cb6c75-1ded-3b1e-e483-cba12bd67278@gmail.com> Hello I have two questions that I don't seem able to find in the docs. What is the difference between running Borg backup with the repo on an external hard drive vs the repo on a server?? I thought in both cases Borg runs on the local machine, does its work and updates the repo, whether the repo resides on a drive or a server.? The reason I ask is that servers hosting Borg backup have Borg software running on them and I don't understand what the server specific Borg software does. Is there a problem with two or more local machines using the same repo? I thank you in advance. Joe From tw at waldmann-edv.de Mon Dec 23 05:22:59 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 23 Dec 2019 11:22:59 +0100 Subject: [Borgbackup] Raspberry pi 4 for borg server In-Reply-To: References: Message-ID: <81694ba8-b1eb-9b25-43fd-1ef052dc1abd@waldmann-edv.de> > Would the new raspberry pi 4 with 4gb ram and separate nic at gigabit > speed be enough for an smb to use borg backup on prem on a Pi 4 to send > data off-site to a receiving server with borg installed? I don't personally have a rpi4, but I used its predecessors. I usually advise against using rpi (or similar SBCs) for borg. Even for the rpi4, these potential issues remain: - RAM is not expandable, so no matter how much they have, it means game over for the case you'ld need more (now or some day in the future). - no ECC RAM - SD cards sometimes are crappy and the rpi OS is often stored on SD. - The rpi4 is faster than its predecessors, but still not on the level of PC or server hardware, neither when looking at speed nor when looking at reliability. So whether that is good enough for you depends a lot on how much data you have and how that will grow in future. Also on how important / valuable your data is. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Mon Dec 23 05:39:56 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 23 Dec 2019 11:39:56 +0100 Subject: [Borgbackup] Newbie questions In-Reply-To: <78cb6c75-1ded-3b1e-e483-cba12bd67278@gmail.com> References: <78cb6c75-1ded-3b1e-e483-cba12bd67278@gmail.com> Message-ID: <37b586a1-99dd-5ab9-e808-25b45e381f00@waldmann-edv.de> > I have two questions that I don't seem able to find in the docs. Maybe read them again, also the FAQ part. Some stuff you asked is answered there. > What is the difference between running Borg backup with the repo on an > external hard drive vs the repo on a server? Borg can write to a repo on a locally mounted filesystem or (via its client/server mode that usually uses ssh as a transport to talk to another remote borg process) via network to a repo on a (potentially remote) server. Of course you can also locally mount a network filesystem that stores on a remote server, e.g. sshfs. The difference to the client/server mode is that the latter has a borg process running on the repo server while with the network filesystem, the borg client only can do fs operations and has no agent (borg process) running on the repo server, so EVERY operation needs to go over the network fs, which is slower. > I thought in both cases > Borg runs on the local machine, does its work and updates the repo, > whether the repo resides on a drive or a server. Yes, but in the client/server case, there is another borg process running on the repo server. > The reason I ask is > that servers hosting Borg backup have Borg software running on them and > I don't understand what the server specific Borg software does. The server side process is doing storage-related low-level repo operations (get, put, commit, check, compact). The client side does the high-level stuff: deduplication, encryption, compression, dealing with archives, backups, restores, etc. > Is there a problem with two or more local machines using the same repo? Yes and no, see FAQ. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tve at voneicken.com Mon Dec 23 11:58:29 2019 From: tve at voneicken.com (Thorsten von Eicken) Date: Mon, 23 Dec 2019 16:58:29 +0000 Subject: [Borgbackup] Raspberry pi 4 for borg server In-Reply-To: <81694ba8-b1eb-9b25-43fd-1ef052dc1abd@waldmann-edv.de> References: <81694ba8-b1eb-9b25-43fd-1ef052dc1abd@waldmann-edv.de> Message-ID: <0100016f33b2207c-5397f6df-cc37-429e-998e-15432d99be4d-000000@email.amazonses.com> Rather than poo-pooing the use of SBCs outright, I think it's worth being more nuanced about it. I've been using an Odroid C1 and an Odroid HC2 as borg servers for many years now. The (important) context is that I'm backing up two home desktops, a laptop, and several small home servers. The HC2 runs off an EMMC, has GB ethernet and a fast SATA interface with a very nicely integrated mount for a 3.5" drive. It's also my media server (but I'm not a video hoarder). I've never perceived any memory/speed limitation. I do nightlies to the primary 3.5" drive and I have an additional identical drive on an external USB3-SATA bridge to which I rsync daily. I prefer this non-raid duplication 'cause it's dead simple. The OC1 has GB ethernet but the 2.5" disk is attached via USB2, hence slow. It's located out in the yard, buried, as a catastrophic backup as I live in a high-fire area. I do backups to it once a week, mostly copying stuff from the HC2. They're slow, but that's OK. As a bonus, I can take the USB drive truly off-site for "long-term" storage and replace it. I also ship backups once a clear to cloud storage but am weary about the costs and the time it takes to retrieve something (I don't have GB fiber to the home by a long shot). I do find value for decade-long archiving, though. What I like about this set-up is that (a) it uses little electricity, (b) the components are all cheap so you can afford a spare or else get one next-day, (c) there's nothing complicated in any of the set-up. Would I use this in a commercial setting: no. It all depends on the context... Thorsten On 12/23/19 2:22 AM, Thomas Waldmann wrote: >> Would the new raspberry pi 4 with 4gb ram and separate nic at gigabit >> speed be enough for an smb to use borg backup on prem on a Pi 4 to send >> data off-site to a receiving server with borg installed? > I don't personally have a rpi4, but I used its predecessors. > > I usually advise against using rpi (or similar SBCs) for borg. Even for > the rpi4, these potential issues remain: > > - RAM is not expandable, so no matter how much they have, it means game > over for the case you'ld need more (now or some day in the future). > > - no ECC RAM > > - SD cards sometimes are crappy and the rpi OS is often stored on SD. > > - The rpi4 is faster than its predecessors, but still not on the level > of PC or server hardware, neither when looking at speed nor when looking > at reliability. > > > So whether that is good enough for you depends a lot on how much data > you have and how that will grow in future. Also on how important / > valuable your data is. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joehesse at gmail.com Wed Dec 25 05:43:23 2019 From: joehesse at gmail.com (Joseph Hesse) Date: Wed, 25 Dec 2019 04:43:23 -0600 Subject: [Borgbackup] Problem with borg compact Message-ID: <8379bfb9-5d8d-bcf8-8c57-87582d6c7d52@gmail.com> Hello, I can't get "borg compact" to work for me.? The listing below comes from a repository on an external hard disk but I have had the same problem with a remote repository.? I am using borg version borg 1.1.10. ============================================================= [joe at lenny51 ~]$ borg list /borg/repo/JHBorgBackup Enter passphrase for key /borg/repo/JHBorgBackup: joe-2019-11-09T11:40:17????????????? Sat, 2019-11-09 11:40:17 [4fc0595c1ab8572111b5e46bffd3ed298c0b0fc19fdb2e621178fcb0a9dc7905] joe-2019-11-10T17:37:40????????????? Sun, 2019-11-10 17:37:40 [57b980d5bb468be3667a2604df9d4fdb4d140e89350a60ff35ad25872ad55c3c] joe-2019-11-11T10:30:37????????????? Mon, 2019-11-11 10:30:38 [5cd4cd34e9972b97dfdfc04267e7ab746b8442e5a23e84511ab7ee87f2234c19] joe-2019-11-12T10:36:31????????????? Tue, 2019-11-12 10:36:31 [11f0ddff9968ab9ac0d7808a6280e3cd7b27d532d0128c493ddd8d5773afb977] ... [joe at lenny51 ~]$ borg compact /borg/repo/JHBorgBackup usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] ??????????? [--debug-topic TOPIC] [-p] [--log-json] [--lock-wait SECONDS] ??????????? [--show-version] [--show-rc] [--umask M] [--remote-path PATH] ??????????? [--remote-ratelimit RATE] [--consider-part-files] ??????????? [--debug-profile FILE] [--rsh RSH] ??????????? ... borg: error: argument : invalid choice: 'compact' (choose from 'mount', 'serve', 'init', 'check', 'key', 'change-passphrase', 'create', 'extract', 'export-tar', 'diff', 'rename', 'delete', 'list', 'umount', 'info', 'break-lock', 'prune', 'upgrade', 'recreate', 'with-lock', 'config', 'help', 'debug', 'benchmark') ============================================================= Thank you for your help.? Happy holiday. Joe From imperator at jedimail.de Wed Dec 25 05:52:09 2019 From: imperator at jedimail.de (Imperator) Date: Wed, 25 Dec 2019 11:52:09 +0100 Subject: [Borgbackup] Problem with borg compact In-Reply-To: <8379bfb9-5d8d-bcf8-8c57-87582d6c7d52@gmail.com> References: <8379bfb9-5d8d-bcf8-8c57-87582d6c7d52@gmail.com> Message-ID: Hi Joe, borg compact does not seem to be available in 1.1.10, it's a new command in current development version (master): https://borgbackup.readthedocs.io/en/master/usage/compact.html Am 25.12.19 um 11:43 schrieb Joseph Hesse: > I can't get "borg compact" to work for me.? The listing below comes from > a repository on an external hard disk but I have had the same problem > with a remote repository.? I am using borg version borg 1.1.10. > > [joe at lenny51 ~]$ borg compact /borg/repo/JHBorgBackup > usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] > ??????????? [--debug-topic TOPIC] [-p] [--log-json] [--lock-wait SECONDS] > ??????????? [--show-version] [--show-rc] [--umask M] [--remote-path PATH] > ??????????? [--remote-ratelimit RATE] [--consider-part-files] > ??????????? [--debug-profile FILE] [--rsh RSH] > ??????????? ... > borg: error: argument : invalid choice: 'compact' (choose from > 'mount', 'serve', 'init', 'check', 'key', 'change-passphrase', 'create', > 'extract', 'export-tar', 'diff', 'rename', 'delete', 'list', 'umount', > 'info', 'break-lock', 'prune', 'upgrade', 'recreate', 'with-lock', > 'config', 'help', 'debug', 'benchmark') -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 554 bytes Desc: OpenPGP digital signature URL: From tw at waldmann-edv.de Wed Dec 25 08:40:50 2019 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 25 Dec 2019 14:40:50 +0100 Subject: [Borgbackup] Problem with borg compact In-Reply-To: <8379bfb9-5d8d-bcf8-8c57-87582d6c7d52@gmail.com> References: <8379bfb9-5d8d-bcf8-8c57-87582d6c7d52@gmail.com> Message-ID: <8b2efe42-af45-3672-118c-a3fad7a54deb@waldmann-edv.de> On the docs site, there is a version selector at the lower right side. It is a good idea to select the version you actually use. Except maybe generic stuff like FAQ or docs improvements not relating to new functionality. For that, it might be also a good idea to select "stable" (if you currently use a stable release, which is 1.1.x as of now). Then you'll get the latest docs for 1.1.x. "master" is the current development version (and will get released as 1.2.x some day) with bigger changes like e.g. the addition of "borg compact". -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt.