From tw at waldmann-edv.de Sun Oct 2 17:04:43 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 2 Oct 2022 23:04:43 +0200 Subject: [Borgbackup] borgbackup 2.0.0b3 released! Message-ID: <8b481693-b95e-0d59-a449-598251a4aaa9@waldmann-edv.de> Just released a new #borgbackup beta, please help testing! One of the new features is efficient repo-wide recompression. https://github.com/borgbackup/borg/releases/tag/2.0.0b3 From ayushpanchratan55 at gmail.com Fri Oct 14 08:53:22 2022 From: ayushpanchratan55 at gmail.com (Ayush Panchratan) Date: Fri, 14 Oct 2022 18:23:22 +0530 Subject: [Borgbackup] Backup with borgbase Message-ID: Hi, How to move the backup to borgbase cloud and move it to different server and restore it there.I'm stuck on how to do it. Also, how to do OS image backup using borg or borgmatic ? Thanks & Regards Ayush -------------- next part -------------- An HTML attachment was scrubbed... URL: From tschoening at am-soft.de Fri Oct 14 09:38:41 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Fri, 14 Oct 2022 15:38:41 +0200 Subject: [Borgbackup] Backup with borgbase In-Reply-To: References: Message-ID: <1644494037.20221014153841@am-soft.de> Guten Tag Ayush Panchratan, am Freitag, 14. Oktober 2022 um 14:53 schrieben Sie: > How to move the backup to borgbase cloud and move it to different server > and restore it there.I'm stuck on how to do it. If youd on't see any additonal upload for existing files and directories, it might simply not be supported at all. Otherwise you would only need to upload things there. https://borgbackup.readthedocs.io/en/stable/faq.html#how-do-i-rename-a-repository > Also, how to do OS image backup using borg or borgmatic ? You don't directly, but need some imaging tool and store its contents instead. You should really be careful if that is what you want and need. Remember that image container files are still large mostly, it might be impossible to create something like incremental/differential backups, it might be difficult to restore individual files etc. https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From ayushpanchratan55 at gmail.com Fri Oct 14 10:23:18 2022 From: ayushpanchratan55 at gmail.com (Ayush Panchratan) Date: Fri, 14 Oct 2022 19:53:18 +0530 Subject: [Borgbackup] Backup with borgbase In-Reply-To: <1644494037.20221014153841@am-soft.de> References: <1644494037.20221014153841@am-soft.de> Message-ID: Thank You for your valuable feedback,I'll look at it On Fri, Oct 14, 2022 at 7:27 PM Thorsten Sch?ning wrote: > Guten Tag Ayush Panchratan, > am Freitag, 14. Oktober 2022 um 14:53 schrieben Sie: > > > How to move the backup to borgbase cloud and move it to different server > > and restore it there.I'm stuck on how to do it. > > If youd on't see any additonal upload for existing files and > directories, it might simply not be supported at all. Otherwise you > would only need to upload things there. > > > https://borgbackup.readthedocs.io/en/stable/faq.html#how-do-i-rename-a-repository > > > Also, how to do OS image backup using borg or borgmatic ? > > You don't directly, but need some imaging tool and store its contents > instead. You should really be careful if that is what you want and > need. Remember that image container files are still large mostly, it > might be impossible to create something like incremental/differential > backups, it might be difficult to restore individual files etc. > > https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html > > Mit freundlichen Gr??en > > Thorsten Sch?ning > > -- > AM-SoFT IT-Service - Bitstore Hameln GmbH > Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK > > E-Mail: Thorsten.Schoening at AM-SoFT.de > Web: http://www.AM-SoFT.de/ > > Tel: +49 5151- 9468- 0 > Tel: +49 5151- 9468-55 > Mobil: +49 178-8 9468-04 > > AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 > Hameln > AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska > > > F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. > > Mit freundlichen Gr??en, > > Thorsten Sch?ning > > > Telefon: +49 5151 9468-55 > Fax: > E-Mail: TSchoening at am-soft.de > > AM-Soft IT-Service - Bitstore Hameln GmbH > Brandenburger Stra?e 7c > 31789 Hameln > > Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte > Informationen und ist ausschliesslich f?r den Adressaten bestimmt. > Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten > ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail > irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und > vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail > bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung > oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im > Vertrauen auf erlangte Information untersagt. > > This e-mail may contain confidential and/or privileged information and is > intended solely for the addressee. Access to this email by anyone else is > unauthorized. If you are not the intended recipient (or have received this > e-mail in error) please notify the sender immediately and destroy this > e-mail. If you are not the intended recipient, any disclosure, copying, > distribution or any action taken or omitted to be taken in reliance on it, > is prohibited and may be unlawful. > > Hinweise zum Datenschutz: bitstore.group/datenschutz > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From grumpy at manlymail.net Thu Oct 27 10:26:44 2022 From: grumpy at manlymail.net (grumpy at manlymail.net) Date: Thu, 27 Oct 2022 09:26:44 -0500 (CDT) Subject: [Borgbackup] extract Message-ID: Just a simple question from a simple mind. I wanted to retrieve a group of files, foo*, from a single directory. I finally got what I needed by using include and exclude patterns. This seems quite convoluted. Is there a simple way to do this? From tw at waldmann-edv.de Thu Oct 27 13:12:14 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 27 Oct 2022 19:12:14 +0200 Subject: [Borgbackup] extract In-Reply-To: References: Message-ID: <7ca4021b-7648-abbe-8517-eb75a53bdd27@waldmann-edv.de> > I wanted to retrieve a group of files, foo*, from a single directory. > I finally got what I needed by using include and exclude patterns. > This seems quite convoluted. > Is there a simple way to do this? borg mount repo::archive /mnt and then just copy whatever you need from the mountpoint. but be aware that the mounted FUSE filesystem does not support all metadata (e.g. does not support ACLs), but the basic stuff works. From vw at zen.net.au Tue Nov 1 23:08:50 2022 From: vw at zen.net.au (Vaughan Wickham) Date: Wed, 2 Nov 2022 03:08:50 +0000 Subject: [Borgbackup] Local backup media file system corrupt. Best way to recover? Message-ID: Hello, I have been performing borgbackup to a locally attached external drive. Unfortunately, something has happened to the file system of that drive, and I have to reformat the media to make it usable. So, I'm going to have to re-setup the local Borg repository. I also have an off-site borg repository and am wondering if I can use that repository to restore the local borg repository? Regards, Vaughan -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Wed Nov 2 12:27:59 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 2 Nov 2022 17:27:59 +0100 Subject: [Borgbackup] Local backup media file system corrupt. Best way to recover? In-Reply-To: References: Message-ID: <7f85498f-1f06-4d2e-22b2-6e1d3dbef128@waldmann-edv.de> > I also have an off-site borg repository and am wondering if I can use > that repository to restore the local borg repository? Sure, just completely copy back the repository onto local media. When first accessing the resulting repo, you might get some warnings because borg might notice it is not in the expected state. If you trust the repo, you can just ignore these / answer the questions accordingly. From vw at zen.net.au Sat Nov 5 16:57:13 2022 From: vw at zen.net.au (Vaughan Wickham) Date: Sat, 5 Nov 2022 20:57:13 +0000 Subject: [Borgbackup] FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <7f85498f-1f06-4d2e-22b2-6e1d3dbef128@waldmann-edv.de> Message-ID: IP address is NOT whitelisted: Yes SPF check stopped as domain server: yes content filtering stopped as whitelisted: Yes attachment filtering stopped as whitelisted: Yes hotmail filtering stopped as whitelisted: Yes Execute check null from address: Yes from address exists: Yes URL filtering stopped as whitelisted: Yes Hello, I have copied the off-site borg repository to the local disk and everything seemed to be fine. However, when the next backup of the off-site borg repository ran, it failed. Borg: Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID) 2022-11-03 12:39:20.826790650 ERROR: Failed to initialize Borg repository, borg rc 2! There are currently multiple repos with the same ID because I have copied the off-site repository to the local system. Regards, Vaughan -----Original Message----- > I also have an off-site borg repository and am wondering if I can use > that repository to restore the local borg repository? Sure, just completely copy back the repository onto local media. When first accessing the resulting repo, you might get some warnings because borg might notice it is not in the expected state. If you trust the repo, you can just ignore these / answer the questions accordingly. _______________________________________________ Borgbackup mailing list Borgbackup at python.org https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Mon Nov 7 06:57:15 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 7 Nov 2022 12:57:15 +0100 Subject: [Borgbackup] FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <7f85498f-1f06-4d2e-22b2-6e1d3dbef128@waldmann-edv.de> Message-ID: <79e2ee11-c97a-b3dc-fc42-efa7dc8b0d84@waldmann-edv.de> On 05.11.22 21:57, Vaughan Wickham wrote: > IP address is NOT whitelisted: Yes > SPF check stopped as domain server: yes > content filtering stopped as whitelisted: Yes > attachment filtering stopped as whitelisted: Yes > hotmail filtering stopped as whitelisted: Yes > Execute check null from address: Yes > from address exists: Yes > URL filtering stopped as whitelisted: Yes Can you stop including these in your mails? > Borg: Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID) ~/.config/borg/security/REPOID/... borg remembers some stuff there. if you are sure nothing weird is going on, you can delete some file from there. > 2022-11-03 12:39:20.826790650 ERROR: Failed to initialize Borg repository, borg rc 2! initialize? > There are currently multiple repos with the same ID because I have copied the off-site repository to the local system. You can of course make copies of borg repos and also restore a repo from such a copy. But never use multiple repos **with same ID** **with borg**, that breaks the AES-CTR mode crypto. From gdayvw at gmail.com Tue Nov 8 07:47:40 2022 From: gdayvw at gmail.com (gday vw) Date: Tue, 8 Nov 2022 23:47:40 +1100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? Message-ID: The problem here is that I'm not really clear on what I need to do. So, for clarity, let me state what I have done so far: 1. I had a local copy of a Borg repository that was lost when the filesystem was corrupted 2. I also have an off-site Borg repository which was still intact 3. Rather than creating a new local repository, I asked whether it was possible to copy the off-site repository 4. I was informed that it was possible to copy the off-site repository 5. I then proceeded to copy the entire off-site repository (i.e. I had a duplicate file structure locally). I didn't update any other borg files on the local system (i.e I made no changes to the contents of ~/.config) 6. After "restoring" the local Borg repository, I was able to add new archives to the "restored" local repository. So, everything looked 'sweet' 7. However, when I tried to update the off-site repository, that action failed because: "Borg: Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)" 8. There are undoubtedly multiple repos with the same ID, because clearly by me making a complete copy of the off-site repository, I have duplicated the off-site repository ID when I made a copy of the off-site Borg repository. What is not clear to me, is whether it is possible to change the repository ID of the local Borg repository which I have copied? And if it is possible to change the repository ID, what are the steps? I've had a look at the Borg documentation. I can't see any option with init to change the repository ID (although I don't claim to be a borg expert so I could be mistaken here). Ideally, I need advice on how to change the repository ID. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tschoening at am-soft.de Tue Nov 8 08:22:48 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Tue, 8 Nov 2022 14:22:48 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: Message-ID: <1578227333.20221108142248@am-soft.de> Guten Tag gday vw, am Dienstag, 8. November 2022 um 13:47 schrieben Sie: > What is not clear to me, is whether it is possible to change the repository > ID of the local Borg repository which I have copied? AFAIK: No, you need to use one of the available repos only and delete the other. In your case you most likely keep using the local copy, delete the off-site one and need to create a new BORG repo off-site again. You can't continue using both. https://borgbackup.readthedocs.io/en/stable/faq.html#can-i-copy-or-synchronize-my-repo-to-another-location https://borgbackup.readthedocs.io/en/stable/faq.html#faq-corrupt-repo Though, thinking about your use-case, it would obviously be great if one could keep using both repos as independent ones again. Otherwise off-site is losing all history and if the local repo fails again "tomorrow", off-site won't be of too much help anymore. Because of the current limitation, it might even make more sense to not use BORG off-site at all and only keep using RSYNC or alike local to off-site. This way off-site could use file syestem level snapshots to keep multiple versions of the repo around, just in case the local repo gets corrupted. The important difference is that RSYNC doesn't care too much and simply forwards local to off-site again, keeping older snapshots etc. around. OTOH, BORG will refuse to use off-site with existing data at all, like in your case. Reads to me like it might make sense to discuss this use-case and restriction further as an enhancement in GitHub. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From tw at waldmann-edv.de Tue Nov 8 11:43:38 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 8 Nov 2022 17:43:38 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: Message-ID: > 2. I also have an off-site Borg repository which was still intact > 3. Rather than creating a new local repository, I asked whether it was > possible to copy the off-site repository You can not use the same repo with same key material at 2 places, that is not secure because of nonce reuse. Practically you could hack the repo id of one repo (like +1), but the security problem does not go away by that. > 4. I was informed that it was possible to copy the off-site repository I assumed you have a repo-copy that was just a copy (like rclone / rsync) and NOT also used with borg on the remote site. > 7. However, when I tried to update the off-site repository, that action > failed because: > "Borg: Cache, or information obtained from the security directory is > newer than repository - this is either an attack or unsafe (multiple > repos with same ID)" That is because you used 2 repos with the same ID with borg. > What is not clear to me, is whether it is possible to change the > repository ID of the local Borg repository which I have copied? That would make it work, but it would not be cryptographically secure any more. > And if it is possible to change the repository ID, what are the steps? Make sure that repo_dir/config has always a unique ID (if you create a new repo, a random number is used. if you hack it manually, you can just do +1). After that, better delete the cache of both repos: borg delete --cache-only repo1 borg delete --cache-only repo2 First borg op after that will take a while, but it will clean up any mess that there might be now due to the duplicate IDs. > I can't see any option with init to change the repository ID. There is no such option because that is not secure. It just solves the issue that borg confuses the 2 repos because it would use the same config and cache path, but it does not solve the AES nonce reuse issue. BTW, borg2 will fix the AES-CTR mode issues by not using AES-CTR any more, so things will get a bit easier. E.g. you will be able to use "borg transfer" to efficiently transfer an archive from one repo to another related repo. From tschoening at am-soft.de Tue Nov 8 12:16:37 2022 From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=) Date: Tue, 8 Nov 2022 18:16:37 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: Message-ID: <1281937887.20221108181637@am-soft.de> Guten Tag Thomas Waldmann, am Dienstag, 8. November 2022 um 17:43 schrieben Sie: > BTW, borg2 will fix the AES-CTR mode issues by not using AES-CTR > any more, so things will get a bit easier. E.g. you will be able to > use "borg transfer" to efficiently transfer an archive from one repo to another related repo. But what about the concrete off-site backup use-case? BORG is used to init two repos at two different placves, backups are more or less always done to both different repos one after another. One of the repos gets damaged for any reason and is lost. the best in this case would really be to copy the leftover repo to the other place and reuse it as-is. Transferring individual archives might be far more difficult especially with repos with lots of archives and if one knows that all of the repo is wanted in the end. What's with the ID then? Will there be some way in BORG to change that ID to some random string? Or would one need to do it manually? Would it work and be safe because of not using AES-CTR anymore? Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From tw at waldmann-edv.de Tue Nov 8 13:31:57 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 8 Nov 2022 19:31:57 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: <1281937887.20221108181637@am-soft.de> References: <1281937887.20221108181637@am-soft.de> Message-ID: <3ddb5cd0-4e9b-dda5-54e1-181ef89b9ef0@waldmann-edv.de> >> BTW, borg2 will fix the AES-CTR mode issues by not using AES-CTR >> any more, so things will get a bit easier. E.g. you will be able to >> use "borg transfer" to efficiently transfer an archive from one repo to another related repo. > > But what about the concrete off-site backup use-case? BORG is used to > init two repos at two different placves, backups are more or less > always done to both different repos one after another. One of the > repos gets damaged for any reason and is lost. > > the best in this case would really be to copy the leftover repo to the > other place and reuse it as-is. There is no code yet to do that with borg2, but at least we do not have the crypto issue there (no aes-ctr, new session keys per backup, no counter to manage long term). So guess one would only need to change the repo-id to have separate cache/config storage. > Transferring individual archives might > be far more difficult especially with repos with lots of archives and > if one knows that all of the repo is wanted in the end. As it is now, "borg2 transfer" copies all archives to the target repo (by default). If you copy the whole repo, you also get all archives, but you can't "mix" anything. From gdayvw at gmail.com Tue Nov 8 16:02:40 2022 From: gdayvw at gmail.com (gday vw) Date: Wed, 9 Nov 2022 08:02:40 +1100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? Message-ID: > Make sure that repo_dir/config has always a unique ID (if you create a new > repo, a random number is used. if you hack it manually, you can just do +1). When I ls the local copy of the repo I see the following: config data hints.4404 index.4404 integrity.4404 nonce README And if I do ls: repo/config there are no files listed the config directory is empty (or if it's not I need to use some switch with ls) So, when you say +1, how / where do I enter a "1" fyi, I am using Borg in conjunction with rear, so perhaps the way that rear implements Borg is not the way that you assume. Ironically, I was willing to create a new local repository, but I felt that if I could use an off-site repository as a starting point it would be better because I would have all the backup history. And I still feel that is the case, which is why I am persisting with trying to work this out. FWIW the "cryptographically secure" considerations that have been mentioned don't concern me in the slightest. For me Borg is just a backup system for data. As long as the repository is intact, that is all that I'm concerned about. Whether the repository is "unique" or not, I couldn't care less. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Wed Nov 9 05:08:24 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 9 Nov 2022 11:08:24 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: Message-ID: <0538bb6f-7ea3-266e-18a7-0432802d3ee4@waldmann-edv.de> > config data ?hints.4404 ?index.4404 ?integrity.4404 ?nonce ?README > > > And if I do > > ls: repo/config > > there are no files listed > > the config directory is empty (or if it's not I need to use some switch > with ls) repo/config is a text FILE (if repo/ is the borg repository directory), not a directory. > fyi, I am using Borg in conjunction with rear, so perhaps the way that > rear implements Borg is not the?way that you assume. I don't have experience with rear, but I would assume that the borg repo directory looks as it usually does. > Ironically, I was willing to create a new local repository, but I felt > that if I could use an off-site repository as a starting point it would > be better because I would have all the backup history. And I still feel > that is the case, which is why I am persisting with trying to work this?out. > > > FWIW the "cryptographically secure" considerations?that have been > mentioned don't concern me in the slightest. OK, then you can use the id + 1 method as I described. From gdayvw at gmail.com Wed Nov 9 05:43:53 2022 From: gdayvw at gmail.com (gday vw) Date: Wed, 9 Nov 2022 21:43:53 +1100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <0538bb6f-7ea3-266e-18a7-0432802d3ee4@waldmann-edv.de> Message-ID: Unfortunately that wasn't the answer. I've now performed: borg delete --cache-only repo1 borg delete --cache-only repo2 And when I try to update the off-site repository, I'm still getting the error re multiple repos with the same ID On Wed, Nov 9, 2022 at 9:35 PM gday vw wrote: > I've missed the borg delete step > > Doh! > > Hopefully that is the answer > > On Wed, Nov 9, 2022 at 9:08 PM Thomas Waldmann wrote: > >> >> >> > config data hints.4404 index.4404 integrity.4404 nonce README >> > >> > >> > And if I do >> > >> > ls: repo/config >> > >> > there are no files listed >> > >> > the config directory is empty (or if it's not I need to use some switch >> > with ls) >> >> repo/config is a text FILE (if repo/ is the borg repository directory), >> not a directory. >> >> > fyi, I am using Borg in conjunction with rear, so perhaps the way that >> > rear implements Borg is not the way that you assume. >> >> I don't have experience with rear, but I would assume that the borg repo >> directory looks as it usually does. >> >> > Ironically, I was willing to create a new local repository, but I felt >> > that if I could use an off-site repository as a starting point it would >> > be better because I would have all the backup history. And I still feel >> > that is the case, which is why I am persisting with trying to work >> this out. >> > >> > >> > FWIW the "cryptographically secure" considerations that have been >> > mentioned don't concern me in the slightest. >> >> OK, then you can use the id + 1 method as I described. >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdayvw at gmail.com Thu Nov 10 17:27:27 2022 From: gdayvw at gmail.com (gday vw) Date: Fri, 11 Nov 2022 09:27:27 +1100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <0538bb6f-7ea3-266e-18a7-0432802d3ee4@waldmann-edv.de> Message-ID: When I try to update the off-site repository, I'm still getting the error re multiple repos with the same ID I've checked the config file of both repositories and the ID is different. Any thoughts as to why this is happening? On Wed, Nov 9, 2022 at 9:43 PM gday vw wrote: > Unfortunately that wasn't the answer. > > I've now performed: > > borg delete --cache-only repo1 > > borg delete --cache-only repo2 > > > And when I try to update the off-site repository, I'm still getting the > error re multiple repos with the same ID > > On Wed, Nov 9, 2022 at 9:35 PM gday vw wrote: > >> I've missed the borg delete step >> >> Doh! >> >> Hopefully that is the answer >> >> On Wed, Nov 9, 2022 at 9:08 PM Thomas Waldmann >> wrote: >> >>> >>> >>> > config data hints.4404 index.4404 integrity.4404 nonce README >>> > >>> > >>> > And if I do >>> > >>> > ls: repo/config >>> > >>> > there are no files listed >>> > >>> > the config directory is empty (or if it's not I need to use some >>> switch >>> > with ls) >>> >>> repo/config is a text FILE (if repo/ is the borg repository directory), >>> not a directory. >>> >>> > fyi, I am using Borg in conjunction with rear, so perhaps the way that >>> > rear implements Borg is not the way that you assume. >>> >>> I don't have experience with rear, but I would assume that the borg repo >>> directory looks as it usually does. >>> >>> > Ironically, I was willing to create a new local repository, but I felt >>> > that if I could use an off-site repository as a starting point it >>> would >>> > be better because I would have all the backup history. And I still >>> feel >>> > that is the case, which is why I am persisting with trying to work >>> this out. >>> > >>> > >>> > FWIW the "cryptographically secure" considerations that have been >>> > mentioned don't concern me in the slightest. >>> >>> OK, then you can use the id + 1 method as I described. >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdayvw at gmail.com Fri Nov 11 21:46:12 2022 From: gdayvw at gmail.com (gday vw) Date: Sat, 12 Nov 2022 13:46:12 +1100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <0538bb6f-7ea3-266e-18a7-0432802d3ee4@waldmann-edv.de> Message-ID: Hello, So I've tried some more troubleshooting borg list off-site_repo Response: Prompted for passphrase Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID) So I tried deleting the cache again borg delete --cache-only off-site_repo There was no response (or error) and surprisingly (to me) I wasn't prompted for the passphrase for this operation I then re-tried to list the repository Response: Prompted for passphrase Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID) The repository ID's are now definitely different. So there must be an issue with the cache or something else. On Fri, Nov 11, 2022 at 9:27 AM gday vw wrote: > When I try to update the off-site repository, I'm still getting the error > re multiple repos with the same ID > > I've checked the config file of both repositories and the ID is different. > > Any thoughts as to why this is happening? > > On Wed, Nov 9, 2022 at 9:43 PM gday vw wrote: > >> Unfortunately that wasn't the answer. >> >> I've now performed: >> >> borg delete --cache-only repo1 >> >> borg delete --cache-only repo2 >> >> >> And when I try to update the off-site repository, I'm still getting the >> error re multiple repos with the same ID >> >> On Wed, Nov 9, 2022 at 9:35 PM gday vw wrote: >> >>> I've missed the borg delete step >>> >>> Doh! >>> >>> Hopefully that is the answer >>> >>> On Wed, Nov 9, 2022 at 9:08 PM Thomas Waldmann >>> wrote: >>> >>>> >>>> >>>> > config data hints.4404 index.4404 integrity.4404 nonce README >>>> > >>>> > >>>> > And if I do >>>> > >>>> > ls: repo/config >>>> > >>>> > there are no files listed >>>> > >>>> > the config directory is empty (or if it's not I need to use some >>>> switch >>>> > with ls) >>>> >>>> repo/config is a text FILE (if repo/ is the borg repository directory), >>>> not a directory. >>>> >>>> > fyi, I am using Borg in conjunction with rear, so perhaps the way >>>> that >>>> > rear implements Borg is not the way that you assume. >>>> >>>> I don't have experience with rear, but I would assume that the borg >>>> repo >>>> directory looks as it usually does. >>>> >>>> > Ironically, I was willing to create a new local repository, but I >>>> felt >>>> > that if I could use an off-site repository as a starting point it >>>> would >>>> > be better because I would have all the backup history. And I still >>>> feel >>>> > that is the case, which is why I am persisting with trying to work >>>> this out. >>>> > >>>> > >>>> > FWIW the "cryptographically secure" considerations that have been >>>> > mentioned don't concern me in the slightest. >>>> >>>> OK, then you can use the id + 1 method as I described. >>>> _______________________________________________ >>>> Borgbackup mailing list >>>> Borgbackup at python.org >>>> https://mail.python.org/mailman/listinfo/borgbackup >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at silef.de Wed Nov 16 15:39:29 2022 From: sebastian at silef.de (Sebastian Felis) Date: Wed, 16 Nov 2022 21:39:29 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? Message-ID: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> Hi first of all: Thank you so much for this awesome backup software! It has such a great feature list and I am using it for my machines and backups for years now without a hassle. Now I struggle to restore a borg repository ona broken partition from another repo backup. Read checks seems to be fine with Remote: Starting repository index check Remote: Finished full repository check, no problems found. Starting archive consistency check... Remote: Verified integrity of /mnt/borg/index.11050 TAM-verified manifest but writing to the repo failes with following error: ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, in compact_segments ??? assert segments[segment] == 0, 'Corrupted segment reference count - corrupted index or hints' AssertionError: Corrupted segment reference count - corrupted index or hints I tried to remove the hints file on the repo side, deleted the client cache via "borg delete --cache-only" and run "borg check --debug --progress --repair": ... Remote: checking segment file /mnt/borg/data/5/11050... Remote: finished segment check at segment 11050 Remote: Starting repository index check Remote: Finished full repository check, no problems found. Starting archive consistency check... Remote: Verified integrity of /mnt/borg/index.11050 TAM-verified manifest Analyzing archive client-20110614-224401 (1/66) ... Analyzing archive client-20221103-223444 (66/66) RepositoryCache: current items 3847, size 289.49 MB / 999.87 MB, 22926 hits, 3847 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit Writing Manifest. Remote: Cleaned up 0 uncommitted segment files (== everything after segment 11050). Remote: Verified integrity of /mnt/borg/hints.11050 Committing repo (may take a while, due to compact_segments)... Remote: check_free_space: required bytes 361371486, free bytes 117106585600 Remote: Compaction started (threshold is 10%). Remote: compacting segment 2000 with usage count -84 (maybe freeable: 197.06% [276754300 bytes]) Remote: Traceback (most recent call last): ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve ??? res = f(**args) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, in commit ??? self.compact_segments(threshold) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, in compact_segments ??? assert segments[segment] == 0, 'Corrupted segment reference count - corrupted index or hints' AssertionError: Corrupted segment reference count - corrupted index or hints RemoteRepository: 213.05 kB bytes sent, 320.78 MB bytes received, 3935 messages sent Traceback (most recent call last): ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve ??? res = f(**args) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, in commit ??? self.compact_segments(threshold) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, in compact_segments ??? assert segments[segment] == 0, 'Corrupted segment reference count - corrupted index or hints' AssertionError: Corrupted segment reference count - corrupted index or hints Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64 Borg server: Linux: Unknown Linux Borg server: Borg: 1.2.0? Python: CPython 3.10.6 msgpack: 1.0.3 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse] Borg server: PID: 1086983? CWD: /home/sebastian Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', '/mnt/borg'] Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug' Platform: Linux client 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 Linux: Unknown Linux Borg: 1.1.16? Python: CPython 3.9.2 msgpack: 0.5.6.+borg1 PID: 9384? CWD: /home/sebastian sys.argv: ['/usr/bin/borg', 'check', '--debug', '--progress', '--repair'] SSH_ORIGINAL_COMMAND: None I am wondering why the full repository check reports "no problems found" but later the segments are corrupted? Is there a way to restore the segment references and fix that problem? Client runs borg version 1.1.16 on Debian/amd64 Server runs borg version 1.2.0 on Ubuntu/aarch64 (Pi4) I could restore other repos in the same way without problems, but the biggest failed and I would like to fix it. Best regards Sebastian From tw at waldmann-edv.de Thu Nov 17 07:27:03 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 Nov 2022 13:27:03 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> Message-ID: <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> > first of all: Thank you so much for this awesome backup software! You're welcome! > Now I struggle to restore a borg repository ona broken partition from > another repo backup. Read checks seems to be fine with > > > Remote: Starting repository index check > Remote: Finished full repository check, no problems found. > Starting archive consistency check... > Remote: Verified integrity of /mnt/borg/index.11050 > TAM-verified manifest Seems like the end of the log is missing here? Does it also say "Finished archive check" and does it terminate with rc 0? You can give --show-rc to output it or do it directly afterwards on the shell with echo $? . > but writing to the repo failes with following error: > > ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, > in compact_segments > ??? assert segments[segment] == 0, 'Corrupted segment reference count - > corrupted index or hints' > > AssertionError: Corrupted segment reference count - corrupted index or > hints Yeah, there seems to be some inconsistency between these. > I tried to remove the hints file on the repo side, deleted the client > cache via "borg delete --cache-only" and run "borg check --debug > --progress --repair": > > > ... > Remote: checking segment file /mnt/borg/data/5/11050... > Remote: finished segment check at segment 11050 > Remote: Starting repository index check > Remote: Finished full repository check, no problems found. > Starting archive consistency check... > Remote: Verified integrity of /mnt/borg/index.11050 > TAM-verified manifest > Analyzing archive client-20110614-224401 (1/66) > ... > Analyzing archive client-20221103-223444 (66/66) > RepositoryCache: current items 3847, size 289.49 MB / 999.87 MB, 22926 > hits, 3847 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit > Writing Manifest. > Remote: Cleaned up 0 uncommitted segment files (== everything after > segment 11050). > Remote: Verified integrity of /mnt/borg/hints.11050 > Committing repo (may take a while, due to compact_segments)... > Remote: check_free_space: required bytes 361371486, free bytes 117106585600 > Remote: Compaction started (threshold is 10%). > Remote: compacting segment 2000 with usage count -84 (maybe freeable: > 197.06% [276754300 bytes]) Uhoh, that looks weird. :-( ^^^ > Remote: Traceback (most recent call last): > > ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve > ??? res = f(**args) > > ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, > in commit > ??? self.compact_segments(threshold) > > ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, > in compact_segments > ??? assert segments[segment] == 0, 'Corrupted segment reference count - > corrupted index or hints' > > AssertionError: Corrupted segment reference count - corrupted index or > hints > > RemoteRepository: 213.05 kB bytes sent, 320.78 MB bytes received, 3935 > messages sent > Traceback (most recent call last): > > ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve > ??? res = f(**args) > > ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, > in commit > ??? self.compact_segments(threshold) > > ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, > in compact_segments > ??? assert segments[segment] == 0, 'Corrupted segment reference count - > corrupted index or hints' > > AssertionError: Corrupted segment reference count - corrupted index or > hints > > Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP > PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64 > Borg server: Linux: Unknown Linux > Borg server: Borg: 1.2.0? Python: CPython 3.10.6 msgpack: 1.0.3 fuse: > pyfuse3 3.2.0 [pyfuse3,llfuse] > Borg server: PID: 1086983? CWD: /home/sebastian > Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', > '/mnt/borg'] > Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug' > Platform: Linux client 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 > (2022-09-29) x86_64 > Linux: Unknown Linux > Borg: 1.1.16? Python: CPython 3.9.2 msgpack: 0.5.6.+borg1 > PID: 9384? CWD: /home/sebastian > sys.argv: ['/usr/bin/borg', 'check', '--debug', '--progress', '--repair'] > > SSH_ORIGINAL_COMMAND: None That's bad, I would have thought that your approach (deleting hints and cache, running borg check --repair) would fix the issue. I'll try to reproduce / find that issue, thanks for reporting it. Please open an issue about this on the issue tracker on github so we can better keep track of it. If possible, please keep a copy of the repository it the state where it is possible to reproduce this issue - just in case we need it for debugging. > I am wondering why the full repository check reports "no problems found" > but later the segments are corrupted? As of now, I do not think the segment files are corrupted. At least not in a way like an entry's crc32 not matching the entry contents any more, that would have been discovered by the borg check (repository part). From tw at waldmann-edv.de Thu Nov 17 14:23:25 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 Nov 2022 20:23:25 +0100 Subject: [Borgbackup] Subject: Re: FW: Local backup media file system corrupt. Best way to recover? In-Reply-To: References: <0538bb6f-7ea3-266e-18a7-0432802d3ee4@waldmann-edv.de> Message-ID: > Cache, or information obtained from the security directory is newer So, did you also check the .config/borg/security/REPOID directory? Guess you need to delete the manifest-timestamp from there. From sebastian at silef.de Thu Nov 17 15:37:16 2022 From: sebastian at silef.de (Sebastian Felis) Date: Thu, 17 Nov 2022 21:37:16 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> Message-ID: <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> Hi Thomas Thank you for your fast response. On 11/17/22 13:27, Thomas Waldmann wrote: >> Now I struggle to restore a borg repository ona broken partition from >> another repo backup. Read checks seems to be fine with >> >> >> Remote: Starting repository index check >> Remote: Finished full repository check, no problems found. >> Starting archive consistency check... >> Remote: Verified integrity of /mnt/borg/index.11050 >> TAM-verified manifest > > Seems like the end of the log is missing here? I tried to summarize the log in the previous mail. Does the log below is missing important information? > Does it also say "Finished archive check" and does it terminate with > rc 0? You can give --show-rc to output it or do it directly afterwards > on the shell with echo $? . See below. It reports "Remote: Finished full repository check, no problems found." The return code is 0. I will rerun the check and will provide the full log. This will take a while... >> but writing to the repo failes with following error: >> >> ?? File "/usr/lib/python3/dist-packages/borg/repository.py", line >> 864, in compact_segments >> ???? assert segments[segment] == 0, 'Corrupted segment reference >> count - corrupted index or hints' >> >> AssertionError: Corrupted segment reference count - corrupted index >> or hints > > Yeah, there seems to be some inconsistency between these. Could that be produced by a merge of an backup repo to the original one? At the end I did an rsync -a from the backup to the original repo so the state should be the state of the backup + newer files of the original repo. E.g. the backup might have the segments up to 10480 but the original (and broken filesystem) has already segment 11050. >> I tried to remove the hints file on the repo side, deleted the client >> cache via "borg delete --cache-only" and run "borg check --debug >> --progress --repair": >> >> >> ... >> Remote: checking segment file /mnt/borg/data/5/11050... >> Remote: finished segment check at segment 11050 >> Remote: Starting repository index check >> Remote: Finished full repository check, no problems found. >> Starting archive consistency check... >> Remote: Verified integrity of /mnt/borg/index.11050 >> TAM-verified manifest >> Analyzing archive client-20110614-224401 (1/66) >> ... >> Analyzing archive client-20221103-223444 (66/66) >> RepositoryCache: current items 3847, size 289.49 MB / 999.87 MB, >> 22926 hits, 3847 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC >> hit >> Writing Manifest. >> Remote: Cleaned up 0 uncommitted segment files (== everything after >> segment 11050). >> Remote: Verified integrity of /mnt/borg/hints.11050 >> Committing repo (may take a while, due to compact_segments)... >> Remote: check_free_space: required bytes 361371486, free bytes >> 117106585600 >> Remote: Compaction started (threshold is 10%). >> Remote: compacting segment 2000 with usage count -84 (maybe freeable: >> 197.06% [276754300 bytes]) > > Uhoh, that looks weird. :-( ^^^ What should be the expected output if that is weird? >> Remote: Traceback (most recent call last): >> >> ?? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in >> serve >> ???? res = f(**args) >> >> ?? File "/usr/lib/python3/dist-packages/borg/repository.py", line >> 494, in commit >> ???? self.compact_segments(threshold) >> >> ?? File "/usr/lib/python3/dist-packages/borg/repository.py", line >> 864, in compact_segments >> ???? assert segments[segment] == 0, 'Corrupted segment reference >> count - corrupted index or hints' >> >> AssertionError: Corrupted segment reference count - corrupted index >> or hints >> >> RemoteRepository: 213.05 kB bytes sent, 320.78 MB bytes received, >> 3935 messages sent >> Traceback (most recent call last): >> >> ?? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in >> serve >> ???? res = f(**args) >> >> ?? File "/usr/lib/python3/dist-packages/borg/repository.py", line >> 494, in commit >> ???? self.compact_segments(threshold) >> >> ?? File "/usr/lib/python3/dist-packages/borg/repository.py", line >> 864, in compact_segments >> ???? assert segments[segment] == 0, 'Corrupted segment reference >> count - corrupted index or hints' >> >> AssertionError: Corrupted segment reference count - corrupted index >> or hints >> >> Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP >> PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64 >> Borg server: Linux: Unknown Linux >> Borg server: Borg: 1.2.0? Python: CPython 3.10.6 msgpack: 1.0.3 fuse: >> pyfuse3 3.2.0 [pyfuse3,llfuse] >> Borg server: PID: 1086983? CWD: /home/sebastian >> Borg server: sys.argv: ['/usr/bin/borg', 'serve', >> '--restrict-to-path', '/mnt/borg'] >> Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug' >> Platform: Linux client 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 >> (2022-09-29) x86_64 >> Linux: Unknown Linux >> Borg: 1.1.16? Python: CPython 3.9.2 msgpack: 0.5.6.+borg1 >> PID: 9384? CWD: /home/sebastian >> sys.argv: ['/usr/bin/borg', 'check', '--debug', '--progress', >> '--repair'] >> >> SSH_ORIGINAL_COMMAND: None > > That's bad, I would have thought that your approach (deleting hints > and cache, running borg check --repair) would fix the issue. > > I'll try to reproduce / find that issue, thanks for reporting it. Thank you very much. Do you think it can be caused by the different borg versions of client and server? > Please open an issue about this on the issue tracker on github so we > can better keep track of it. I will open an issue with the full log when it is ready. > If possible, please keep a copy of the repository it the state where > it is possible to reproduce this issue - just in case we need it for > debugging. OK. If I can provide any other valuable info or data, please do not hesitate to ask for that. >> I am wondering why the full repository check reports "no problems >> found" but later the segments are corrupted? > > As of now, I do not think the segment files are corrupted. At least > not in a way like an entry's crc32 not matching the entry contents any > more, that would have been discovered by the borg check (repository > part). Good to hear. I hope I can get the repo fixed - besides transfer the old repo to a new one. Best regards and thank you for your help and time Sebastian From tw at waldmann-edv.de Thu Nov 17 15:51:56 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 Nov 2022 21:51:56 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> Message-ID: <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> > I will rerun the check and will provide the full log. "borg check -v REPO" please, so it runs repository AND archives checks. But first check your rsync command, see below. > Could that be produced by a merge of an backup repo to the original one? > At the end I did an rsync -a from the backup to the original repo so the > state should be the state of the backup + newer files of the original > repo. Uhoh, you can not do it like that and mix ("merge") stuff together. If you want to rsync a valid repo from location SRC to location DST, you need something like: rsync -a --delete SRC/ DST/ The --delete is important to delete files in DST that are not any more in SRC, otherwise you might get a weird mix up. Did you forget the --delete? >>> I tried to remove the hints file on the repo side, deleted the client >>> cache via "borg delete --cache-only" and run "borg check --debug >>> --progress --repair": I just checked in the code: borg check --repair rebuilds the hints and index from information found in the segment files only. >>> Remote: compacting segment 2000 with usage count -84 (maybe freeable: >>> 197.06% [276754300 bytes]) >> >> Uhoh, that looks weird. :-( ^^^ > > > What should be the expected output if that is weird? A usage count is how many valid (not deleted) PUTs are still in a segment file. And a count obviously can not be negative. Also freeable space can be 100% max. >> That's bad, I would have thought that your approach (deleting hints >> and cache, running borg check --repair) would fix the issue. Still think so. ^^^ > Do you think it can be caused by the different borg versions of client > and server? No. From sebastian at silef.de Thu Nov 17 17:04:36 2022 From: sebastian at silef.de (Sebastian Felis) Date: Thu, 17 Nov 2022 23:04:36 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> Message-ID: <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> On 11/17/22 21:51, Thomas Waldmann wrote: >> I will rerun the check and will provide the full log. > > "borg check -v REPO" please, so it runs repository AND archives checks. > > But first check your rsync command, see below. I am running now: borg check --verify-data --repair --debug --progress --show-version --show-rc >> Could that be produced by a merge of an backup repo to the original >> one? At the end I did an rsync -a from the backup to the original >> repo so the state should be the state of the backup + newer files of >> the original repo. > > Uhoh, you can not do it like that and mix ("merge") stuff together. > > If you want to rsync a valid repo from location SRC to location DST, > you need something like: > > rsync -a --delete SRC/ DST/ > > The --delete is important to delete files in DST that are not any more > in SRC, otherwise you might get a weird mix up. > > Did you forget the --delete? No. It was on purpose not to run --delete. I have a broken btrfs partition and tried to recover it with the help of the backup. My thought was more a merge of btrfs recovery and rsync from the backup repo. And if borg reports that all segments are valid what corrupts the segment reference count? My assumption is that borg will find and repair the history with a last succeeding commit. Eg. borg backup has segment 10480, the original repo 11050 than borg could find the last commit in segment 11047. Is my assumption valid? The next borg recovery check will be with preceding rsync with --delete. Thank you for the hint. >>>> I tried to remove the hints file on the repo side, deleted the >>>> client cache via "borg delete --cache-only" and run "borg check >>>> --debug --progress --repair": > > I just checked in the code: borg check --repair rebuilds the hints and > index from information found in the segment files only. So it should be fine to delete hints and index files on the repo and call borg check --repair. Than the "usage count" and "freeable space" should be fine? Have a good evening Sebastian From tw at waldmann-edv.de Thu Nov 17 17:27:09 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 Nov 2022 23:27:09 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> Message-ID: > I am running now: borg check --verify-data --repair --debug --progress > --show-version --show-rc That will be rather slow, but very thorough. >>> Could that be produced by a merge of an backup repo to the original >>> one? At the end I did an rsync -a from the backup to the original >>> repo so the state should be the state of the backup + newer files of >>> the original repo. >> >> Uhoh, you can not do it like that and mix ("merge") stuff together. >> >> If you want to rsync a valid repo from location SRC to location DST, >> you need something like: >> >> rsync -a --delete SRC/ DST/ >> >> The --delete is important to delete files in DST that are not any more >> in SRC, otherwise you might get a weird mix up. >> >> Did you forget the --delete? > > > No. It was on purpose not to run --delete. Welcome to unknown territory. As this may create a rather unusual mixup in the segment files that usually can not exist, I am not sure how good/bad borg can handle this. Maybe the traceback you got is caused by this. > And if borg reports that all segments are valid what corrupts the > segment reference count? Could be that although each segment file looks valid (== all the crc32 checks of its entries are ok), the overall state when looking at all segments in their on-disk order is not valid and is not produced by normal borg operations? > My assumption is that borg will find and repair > the history with a last succeeding commit. Eg. borg backup has segment > 10480, the original repo 11050 than borg could find the last commit in > segment 11047. Is my assumption valid? If your repo is in normal mode (NOT in append-only mode), compact_segments will shuffle data from older non-compact segments to newer (then compact) segments. Thus, how data of misc. backup runs maps to segment files is maybe not as simple as you think it is. >> I just checked in the code: borg check --repair rebuilds the hints and >> index from information found in the segment files only. > > > So it should be fine to delete hints and index files on the repo and > call borg check --repair. Than the "usage count" and "freeable space" > should be fine? Usually yes. From jolson at kth.se Mon Nov 21 14:04:45 2022 From: jolson at kth.se (Jonas Olson) Date: Mon, 21 Nov 2022 20:04:45 +0100 Subject: [Borgbackup] The Windows version that used to be Message-ID: In Chocolatey (a package manager and package repository for Windows), someone used to provide a package for BorgBackup [0] a few years ago, but it ceased being updated, and was removed. What's the story behind this package? Did BorgBackup for Windows use to be in a state fit for release, or was this packaging premature, and should better not have been done? Regards, Jonas Olson [0] From sebastian at silef.de Mon Nov 28 14:02:36 2022 From: sebastian at silef.de (Sebastian Felis) Date: Mon, 28 Nov 2022 20:02:36 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> Message-ID: Hi Thomas and borg users On 11/17/22 23:27, Thomas Waldmann wrote: >> I am running now: borg check --verify-data --repair --debug >> --progress --show-version --show-rc > > That will be rather slow, but very thorough. Sorry for the late reply. I have results of the command above with following log. I did not delete the hints and index files before. I had hope that rsync --delete would fix the job, but... ----- borg check --repair --debug --progress --show-version --show-rc using builtin fallback logging configuration borgbackup version 1.1.16 35 self tests completed in 0.18 seconds SSH command line: ['ssh', '-i', '/home/user/.ssh/user at client', 'user at server', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 38 self tests completed in 0.32 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/mnt/borg' Remote: Resolved repository path to '/mnt/borg' This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! Type 'YES' if you understand this and want to continue: Remote: Starting repository check Remote: Verified integrity of /mnt/borg/index.11057 Remote: Verified integrity of /mnt/borg/hints.11057 Remote: Replaying segments?? 0% ... Remote: Replaying segments? 95% Remote: Verified integrity of /mnt/borg/index.11061 Remote: Read committed index of transaction 11061 Remote: Cleaned up 1 uncommitted segment files (== everything after segment 11061). Remote: Segment transaction is??? 11061 Remote: Determined transaction is 11061 Remote: Found 11004 segments Remote: Checking segments 0.0% Remote: checking segment file /mnt/borg/data/0/1... ... Remote: checking segment file /mnt/borg/data/5/11061... Remote: finished segment check at segment 11061 Remote: Starting repository index check Remote: Finished full repository check, no problems found. Starting archive consistency check... Remote: Verified integrity of /mnt/borg/index.11061 TAM-verified manifest Analyzing archive client-20110614-224401 (1/68) ... Analyzing archive client-20221103-223444 (66/68) Analyzing archive client-20221112-144451.checkpoint (67/68) Analyzing archive client-20221112-220336.checkpoint (68/68) RepositoryCache: current items 3849, size 289.73 MB / 998.12 MB, 23086 hits, 3849 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit Writing Manifest. Remote: Cleaned up 0 uncommitted segment files (== everything after segment 11061). Remote: Verified integrity of /mnt/borg/hints.11061 Committing repo (may take a while, due to compact_segments)... Remote: check_free_space: required bytes 361371686, free bytes 117087985664 Remote: Compaction started (threshold is 10%). Remote: compacting segment 2000 with usage count -84 (maybe freeable: 197.06% [276754300 bytes]) Remote: Traceback (most recent call last): ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve ??? res = f(**args) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, in commit ??? self.compact_segments(threshold) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, in compact_segments ??? assert segments[segment] == 0, 'Corrupted segment reference count - corrupted index or hints' AssertionError: Corrupted segment reference count - corrupted index or hints RemoteRepository: 213.41 kB bytes sent, 321.02 MB bytes received, 3939 messages sent Traceback (most recent call last): ? File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve ??? res = f(**args) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 494, in commit ??? self.compact_segments(threshold) ? File "/usr/lib/python3/dist-packages/borg/repository.py", line 864, in compact_segments ??? assert segments[segment] == 0, 'Corrupted segment reference count - corrupted index or hints' AssertionError: Corrupted segment reference count - corrupted index or hints Borg server: Platform: Linux server 5.15.0-1017-raspi #19-Ubuntu SMP PREEMPT Fri Oct 14 08:22:47 UTC 2022 aarch64 Borg server: Linux: Unknown Linux Borg server: Borg: 1.2.0? Python: CPython 3.10.6 msgpack: 1.0.3 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse] Borg server: PID: 1717358? CWD: /home/user Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', '/mnt/borg'] Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --debug' Platform: Linux client 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 Linux: Unknown Linux Borg: 1.1.16? Python: CPython 3.9.2 msgpack: 0.5.6.+borg1 PID: 806530? CWD: /home/user sys.argv: ['/usr/bin/borg', 'check', '--repair', '--debug', '--progress', '--show-version', '--show-rc'] SSH_ORIGINAL_COMMAND: None terminating with error status, rc 2 ----- The log does not tell me any news and reported the same strange errors "compacting segment 2000 with usage count -84" and "Corrupted segment reference count - corrupted index or hints" Further the "rsync -a --delete backup/ repo/" did not help because the cronjob for the rsync backup job with ~"rsync -a repo/ backup/" was not disabled and created another remote copy :-/ Any hints to proceed to get a writable repo again? >> And if borg reports that all segments are valid what corrupts the >> segment reference count? > > Could be that although each segment file looks valid (== all the crc32 > checks of its entries are ok), the overall state when looking at all > segments in their on-disk order is not valid and is not produced by > normal borg operations? But I assume that borgs internal structure is comparable to git tree which I do know. So if the archive information with the root entry, file tree and the file leafs are OK, all data should be fine. There might be additional objects which are not referenced but overall the data should be there and safe. Isn't it? So can I reconstruct/rebuilt the repo, eg. by the new transfer command of borg 2? Is there an alternative for borg 1.2? >>> I just checked in the code: borg check --repair rebuilds the hints >>> and index from information found in the segment files only. >> >> >> So it should be fine to delete hints and index files on the repo and >> call borg check --repair. Than the "usage count" and "freeable space" >> should be fine? > > Usually yes. I never tried to delete integrity file, too. But would it be worth it delete index, hints and integrity files and run "borg check --repair"? Thank you for your help and time Sebastian From tw at waldmann-edv.de Tue Nov 29 05:12:07 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 29 Nov 2022 11:12:07 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> Message-ID: <25291cff-4955-82a5-63a9-5bb43c32599b@waldmann-edv.de> Next time do not use --verify-data. I guess it is not useful in this case and makes the check run take much longer. > Remote: compacting segment 2000 with usage count -84 (maybe freeable: > 197.06% [276754300 bytes]) Your repo is still in a very weird state. > Further the "rsync -a --delete backup/ repo/" did not help because the > cronjob for the rsync backup job with ~"rsync -a repo/ backup/" was not > disabled and created another remote copy :-/ I hope you still have a consistent, valid state of that repo somewhere. Sounds a bit like your rsync activities are creating weird mixups? > Any hints to proceed to get a writable repo again? Get a valid repo files state at that place (either start from empty directory or do not forget the --delete option of rsync and do not have rsync jobs running in parallel), delete the repo index and the hints, run borg check --repair. This should either rebuild index and hints and result in a valid, proven state or it proves that the repo state you have is very invalid and beyond repair. > But I assume that borgs internal structure is comparable to git tree > which I do know. We have docs, if you read them, you do not to jump to conclusions / assumptions. > So can I reconstruct/rebuilt the repo, eg. by the new transfer command > of borg 2? Is there an alternative for borg 1.2? borg2 transfer needs a borg 1.2 repo in good/valid state, thus it is not useful for your case. From sebastian at silef.de Tue Nov 29 17:25:52 2022 From: sebastian at silef.de (Sebastian Felis) Date: Tue, 29 Nov 2022 23:25:52 +0100 Subject: [Borgbackup] How to fix corrupted index or hints? In-Reply-To: <25291cff-4955-82a5-63a9-5bb43c32599b@waldmann-edv.de> References: <11641f91-3433-4b2b-8fd5-a3f219feef24@silef.de> <702492ef-a72c-a256-e28f-d89e93074690@waldmann-edv.de> <846ccff8-ebf4-a047-174e-8d30ddca3d4f@silef.de> <228a4fa5-3295-1a8f-4658-06340d51de7a@waldmann-edv.de> <9ba4310e-a07a-2736-0c10-3c8bf6feedf0@silef.de> <25291cff-4955-82a5-63a9-5bb43c32599b@waldmann-edv.de> Message-ID: On 11/29/22 11:12, Thomas Waldmann wrote: > Your repo is still in a very weird state. > >> Further the "rsync -a --delete backup/ repo/" did not help because >> the cronjob for the rsync backup job with ~"rsync -a repo/ backup/" >> was not disabled and created another remote copy :-/ > > I hope you still have a consistent, valid state of that repo somewhere. So I read: My repo has a weird state which can not be repaired. Full stop. Mm. I will do my homework and search for an offline backup which is not in a weird state... >> But I assume that borgs internal structure is comparable to git tree >> which I do know. > We have docs, if you read them, you do not to jump to conclusions / > assumptions. If I have a look to https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#the-object-graph and https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#keys I do not get the match in my head between my repo objects are all clean and the error of corrupted segment reference count. My understanding is: - The data structure of borg is a cryptographic DAG. If a single bit of an object content would toggle, it would lead to verification error and the upper ids (file, dir, and archive) would change to be correct (again) - Borg lists archives. So there is a manifest and a object graph - The object graph is valid and all objects (archives, directories, files, chunks) can be referenced and do exist - All objects are valid and their data integrity is verified And my big question mark is: Why does the repair job fails while the verification of DAG and objects are fine? And why is the repo in a "read only" mode since creating archives fails? Don't get me wrong or bad. I still love borg, appreciate the endless work and have trust in it. I am also very happy about your responses. Still there are some open questions on my side. Best regards Sebastian From kevin.mccall at yahoo.com Fri Dec 9 16:05:55 2022 From: kevin.mccall at yahoo.com (kevin.mccall at yahoo.com) Date: Fri, 9 Dec 2022 21:05:55 +0000 (UTC) Subject: [Borgbackup] Archive File Integrity Check Against Local File System References: <1064918266.2029357.1670619955789.ref@mail.yahoo.com> Message-ID: <1064918266.2029357.1670619955789@mail.yahoo.com> ?I am new to borg, would anyone be willing to explain to me the best way to verify an archive is identical to the files in the local filesytem that were backed up?? After reading through the documentation my understanding of command borg check is integrity checks of the repository structure and metadata at the block level. I read the command?borg export-tar /path/to/repo::archive-name - | tar --compare -f - -C /path/to/compare/to ?would complete an integrity check between an archive and the local filesystem. Is this correct? How is tar comparing these files with hashes or just ctime or mtime? Is there a better way to ensure files are identical? I look forward to learning how you verify your backups.? Thanks Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.mccall at yahoo.com Fri Dec 9 16:20:10 2022 From: kevin.mccall at yahoo.com (kevin.mccall at yahoo.com) Date: Fri, 9 Dec 2022 21:20:10 +0000 (UTC) Subject: [Borgbackup] Archive File Integrity Check Against Local File System In-Reply-To: <1064918266.2029357.1670619955789@mail.yahoo.com> References: <1064918266.2029357.1670619955789.ref@mail.yahoo.com> <1064918266.2029357.1670619955789@mail.yahoo.com> Message-ID: <1124211088.2033237.1670620810552@mail.yahoo.com> Does this borg check switch compare all the files in the archive to the local file system? Or does this performed at the block level?? The?--verify-data?option will perform a full integrity verification (as opposed to checking the CRC32 of the segment) of data, which means reading the data from the repository, decrypting and decompressing it. This is a cryptographic verification, which will detect (accidental) corruption. For encrypted repositories it is tamper-resistant as well, unless the attacker has access to the keys. It is also very slow. On Friday, December 9, 2022 at 04:05:55 PM EST, kevin.mccall at yahoo.com wrote: ?I am new to borg, would anyone be willing to explain to me the best way to verify an archive is identical to the files in the local filesytem that were backed up?? After reading through the documentation my understanding of command borg check is integrity checks of the repository structure and metadata at the block level. I read the command?borg export-tar /path/to/repo::archive-name - | tar --compare -f - -C /path/to/compare/to ?would complete an integrity check between an archive and the local filesystem. Is this correct? How is tar comparing these files with hashes or just ctime or mtime? Is there a better way to ensure files are identical? I look forward to learning how you verify your backups.? Thanks Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From tschoening at am-soft.de Sat Dec 10 04:29:13 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Sat, 10 Dec 2022 10:29:13 +0100 Subject: [Borgbackup] Archive File Integrity Check Against Local File System In-Reply-To: <1124211088.2033237.1670620810552@mail.yahoo.com> References: <1064918266.2029357.1670619955789.ref@mail.yahoo.com> <1064918266.2029357.1670619955789@mail.yahoo.com> <1124211088.2033237.1670620810552@mail.yahoo.com> Message-ID: <416251890.20221210102913@am-soft.de> Guten Tag kevin.mccall--- via Borgbackup, am Freitag, 9. Dezember 2022 um 22:20 schrieben Sie: > Does this borg check switch compare all the files in the archive > to the local file system? Or does this performed at the block level? Neither of both, BORG only checks that what it sees makes sense to BORG itself. It assumes that data read during the backup is OK and not corrupted, because that is taken care at the transport level from the source disk over the source file system etc. to target file system and disk. >How is tar comparing these files with hashes or just ctime or mtime? According to some docs, it doesn't compare file contents itself and might not tell you about possibly missing files in the backup: > The `--compare' (`-d') operation, as its name implies, causes tar to > compare files and directories in the archive with their counterparts > (files of the same name) in the file system, and report back > differences in file size, mode, owner and modification date. When > performing the `--compare' (`-d') operation, tar acts only on files > actually in the archive--it will ignore files in the active file > system that do not exist in the archive. If tar with `--compare' > (`-d') specified is given, as a file name argument, the name of a > file that does not exist in the archive, it will return an error > message. https://www.math.utah.edu/docs/info/tar_2.html#SEC15 > I look forward to learning how you verify your backups. Most people simply don't and only have an occasional look if things make somewhat sense every few months or years or ... :-) Remember that your backed up data is a frozen point in time and to really compare you would need to keep the source data at exactly that point in time. Freezing data is mostly done using temporary snapshots, so you either need to keep that as long as you want to verify, possibly occupying storage space longer than necessary. Or you need to backup and verify instantly afterwards, which includes reading and transferring all data multiple times, hence larger backup times, but allows you to remove the snapshots sooner. So the default approach of all backup software I know of is to not verify against source data, but rely on correctly working reads and transfers and simply error out if some software reports errors during such operations. Everything else is either not available at all or optional and at least I didn't came accross that anyone uses such features really often. Fast backup times, sooner freed temporary snapshot storage etc. always became more important. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska From tschoening at am-soft.de Sun Dec 11 04:58:24 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Sun, 11 Dec 2022 10:58:24 +0100 Subject: [Borgbackup] Archive File Integrity Check Against Local File System In-Reply-To: <76108444.637688.1670678474951@mail.yahoo.com> References: <1064918266.2029357.1670619955789.ref@mail.yahoo.com> <1064918266.2029357.1670619955789@mail.yahoo.com> <1124211088.2033237.1670620810552@mail.yahoo.com> <416251890.20221210102913@am-soft.de> <76108444.637688.1670678474951@mail.yahoo.com> Message-ID: <1071508842.20221211105824@am-soft.de> Guten Tag kevin.mccall at yahoo.com, am Samstag, 10. Dezember 2022 um 14:21 schrieben Sie: > If I wanted to complete a file verification against the local file > system how do you recommend doing so with an Borg archive?? Take the backup based on some snapshot and after the backup itself has successfully finished, mount the backup again to compare the file tree of the snapshot and that of the mounted backup archive using some specialized tools producing some output about some differences. That output needs to be additionally processed to get the result you actually want, e.g. error out on any difference at all, error out on missing files only, error out on files with different checksums or stuff like that. Things heavily depend on the tools you use to compare and which output they produce. Best approach most likely is to error out either on any changes at all or implement some black- or whitelist to check for special conditions of interest. As often, we are not the only ones thinking of these kinds of problems and I found some interesting suggestions for the necessary compare tools: > rsync --dry-run --recursive --delete --links --checksum --verbose /dir1/ /dir2/ > dirdiff_2.txt > diff --brief --recursive --no-dereference --new-file --no-ignore-file-name-case /dir1 /dir2 > dirdiff_1.txt https://stackoverflow.com/a/40986505/2055163 "borg mount" makes the archive easily available without copying things unnecessarily, e.g. your used compare tool might recognize chanbges in file sizes already, in which case it wouldn't need to read the entire files at all. That most likely wouldn't be possible with your former TAR-approach. https://borgbackup.readthedocs.io/en/stable/usage/mount.html https://borgbackup.readthedocs.io/en/stable/usage/mount.html#borg-umount Last but not least, I'll strongly suggest to use BORGMATIC to implement your approaches. It supports a mostly configuration driven approach allowing to put your configs and stuff into GIT/SVN/... AND supports hooks on various events. I use that mechanism to create snapshots BEFORE backups and delete them and send summary mails AFTER each backup. The AFTER hook would be the place to compare things before cleaning the snapshots up. The following is how I call things, while you need to implement the actual scripts on your own of course: > hooks: > before_backup: > - > > '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/before_backup/local_zfs_datasets.sh' > 'bts.ams' > 'root' > 'rpool/ROOT/pve-1' 'rpool/data/encr/home' > after_backup: > - > > '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/after_backup/local_zfs_datasets.sh' > 'bts.ams' > 'root' > 'rpool/ROOT/pve-1' 'rpool/data/encr/home' > on_error: > - | > export BGM_ERR_VAR_CONFIG_PATH="{configuration_filename}" > export BGM_ERR_VAR_REPO="{repository}" > export BGM_ERR_VAR_ERROR_MSG="{error}" > export BGM_ERR_VAR_ERROR_OUT="$(cat << 'EOT' > {output} > EOT > )" > > '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/on_error/local_zfs_datasets.sh' 'bts.ams' 'root' 'rpool/ROOT/pve-1' 'rpool/data/encr/home' https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ https://torsion.org/borgmatic/ The author of that tool was VERY helpful and open for suggestions and discussions and stuff in the past to me. And I asked for somewhat weird stuff already. ;-) Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska From kevin.mccall at yahoo.com Sun Dec 11 19:16:34 2022 From: kevin.mccall at yahoo.com (kevin.mccall at yahoo.com) Date: Mon, 12 Dec 2022 00:16:34 +0000 (UTC) Subject: [Borgbackup] Archive File Integrity Check Against Local File System In-Reply-To: <1071508842.20221211105824@am-soft.de> References: <1064918266.2029357.1670619955789.ref@mail.yahoo.com> <1064918266.2029357.1670619955789@mail.yahoo.com> <1124211088.2033237.1670620810552@mail.yahoo.com> <416251890.20221210102913@am-soft.de> <76108444.637688.1670678474951@mail.yahoo.com> <1071508842.20221211105824@am-soft.de> Message-ID: <684649362.2334524.1670804194223@mail.yahoo.com> Really great, thank you sincerely for sharing the knowledge. This is helpful. On Sunday, December 11, 2022 at 04:58:51 AM EST, Thorsten Sch?ning wrote: Guten Tag kevin.mccall at yahoo.com, am Samstag, 10. Dezember 2022 um 14:21 schrieben Sie: > If I wanted to complete a file verification against the local file > system how do you recommend doing so with an Borg archive?? Take the backup based on some snapshot and after the backup itself has successfully finished, mount the backup again to compare the file tree of the snapshot and that of the mounted backup archive using some specialized tools producing some output about some differences. That output needs to be additionally processed to get the result you actually want, e.g. error out on any difference at all, error out on missing files only, error out on files with different checksums or stuff like that. Things heavily depend on the tools you use to compare and which output they produce. Best approach most likely is to error out either on any changes at all or implement some black- or whitelist to check for special conditions of interest. As often, we are not the only ones thinking of these kinds of problems and I found some interesting suggestions for the necessary compare tools: > rsync --dry-run --recursive --delete --links --checksum --verbose /dir1/ /dir2/ > dirdiff_2.txt > diff --brief --recursive --no-dereference --new-file --no-ignore-file-name-case /dir1 /dir2 > dirdiff_1.txt https://stackoverflow.com/a/40986505/2055163 "borg mount" makes the archive easily available without copying things unnecessarily, e.g. your used compare tool might recognize chanbges in file sizes already, in which case it wouldn't need to read the entire files at all. That most likely wouldn't be possible with your former TAR-approach. https://borgbackup.readthedocs.io/en/stable/usage/mount.html https://borgbackup.readthedocs.io/en/stable/usage/mount.html#borg-umount Last but not least, I'll strongly suggest to use BORGMATIC to implement your approaches. It supports a mostly configuration driven approach allowing to put your configs and stuff into GIT/SVN/... AND supports hooks on various events. I use that mechanism to create snapshots BEFORE backups and delete them and send summary mails AFTER each backup. The AFTER hook would be the place to compare things before cleaning the snapshots up. The following is how I call things, while you need to implement the actual scripts on your own of course: > hooks: >? ? before_backup: >? ? ? ? - > >? ? ? ? ? '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/before_backup/local_zfs_datasets.sh' >? ? ? ? ? 'bts.ams' >? ? ? ? ? 'root' >? ? ? ? ? 'rpool/ROOT/pve-1' 'rpool/data/encr/home' >? ? after_backup: >? ? ? ? - > >? ? ? ? ? '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/after_backup/local_zfs_datasets.sh' >? ? ? ? ? 'bts.ams' >? ? ? ? ? 'root' >? ? ? ? ? 'rpool/ROOT/pve-1' 'rpool/data/encr/home' >? ? on_error: >? ? ? ? - | >? ? ? ? ? export BGM_ERR_VAR_CONFIG_PATH="{configuration_filename}" >? ? ? ? ? export BGM_ERR_VAR_REPO="{repository}" >? ? ? ? ? export BGM_ERR_VAR_ERROR_MSG="{error}" >? ? ? ? ? export BGM_ERR_VAR_ERROR_OUT="$(cat << 'EOT' >? ? ? ? ? {output} >? ? ? ? ? EOT >? ? ? ? ? )" > >? ? ? ? ? '/usr/local/am_soft_it/libs_bin/backup/bgm_hooks/on_error/local_zfs_datasets.sh' 'bts.ams' 'root' 'rpool/ROOT/pve-1' 'rpool/data/encr/home' https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ https://torsion.org/borgmatic/ The author of that tool was VERY helpful and open for suggestions and discussions and stuff in the past to me. And I asked for somewhat weird stuff already. ;-) Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web:? ? http://www.AM-SoFT.de/ Tel:? +49 5151-? 9468- 0 Tel:? +49 5151-? 9468-55 Mobil: +49? 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska _______________________________________________ Borgbackup mailing list Borgbackup at python.org https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- An HTML attachment was scrubbed... URL: From joehesse at gmail.com Mon Dec 19 04:38:08 2022 From: joehesse at gmail.com (Joseph Hesse) Date: Mon, 19 Dec 2022 03:38:08 -0600 Subject: [Borgbackup] Delete Obsolete Chunks Message-ID: <3f3e06d5-54f5-1780-4d54-23701e8ec16e@gmail.com> Hello, Suppose I create a new file on my local computer, then do a Borg backup and then delete both the new file on my local computer and the Borg backup. At this point there is no way that the file could be recovered using Borg backup. Are the chunks from this file removed from the collection of chunks that constitute Borg backup? Thank you, Joe From tschoening at am-soft.de Mon Dec 19 05:33:57 2022 From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=) Date: Mon, 19 Dec 2022 11:33:57 +0100 Subject: [Borgbackup] Delete Obsolete Chunks In-Reply-To: <3f3e06d5-54f5-1780-4d54-23701e8ec16e@gmail.com> References: <3f3e06d5-54f5-1780-4d54-23701e8ec16e@gmail.com> Message-ID: <236995859.20221219113357@am-soft.de> Guten Tag Joseph Hesse, am Montag, 19. Dezember 2022 um 10:38 schrieben Sie: > Are the chunks from this file removed from the collection of chunks > that constitute Borg backup? Yes, after running "borg compact": https://borgbackup.readthedocs.io/en/stable/faq.html#how-do-i-remove-files-from-an-existing-backup Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From tw at waldmann-edv.de Sat Dec 24 18:57:25 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 25 Dec 2022 00:57:25 +0100 Subject: [Borgbackup] borgbackup 1.2.3 released! Message-ID: borgbackup 1.2.3 was just released (mostly bug fixes)! https://github.com/borgbackup/borg/releases/tag/1.2.3 --- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393