From brianofish at gmail.com Mon Jul 3 12:59:14 2023 From: brianofish at gmail.com (Patrick Schaaf) Date: Mon, 3 Jul 2023 18:59:14 +0200 Subject: [Borgbackup] Feasible to have some readonly activity on separate lock / unlocked? Message-ID: Hi, I think I understand the design constraint of the repo data accesses all having an exclusive lock, because the server side can't look inside most of the data due to encryption, thus can't be very much specific about what can or cannot be allowed concurrently. I totally get how any kind of modification to the repository - backing up stuff, deleting archives, compaction - would need an exclusive lock. However, wouldn't it be feasible to have operations like rlist, list, diff, and maybe even extract and transfer, on existing archives and where no compaction is going on, to run in parallel, and maybe hopefully even in parallel to new backups being made? That would be a huge usability / scriptability improvement, in my eyes. Sorry if this has been asked too often / discussed to death before. Happy to read up on that in issue tracker or mailing list archives. best regards Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunguponcontent at gmail.com Wed Jul 19 14:59:21 2023 From: hunguponcontent at gmail.com (Default User) Date: Wed, 19 Jul 2023 14:59:21 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID Message-ID: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> Hello! How many times have you heard this one before . . . I am using Borg Version 1.2.4 on Debian 12 (Bookworm) Stable, on a local x86 64-bit computer. ? I normally use Vorta 0.8.0-10 as a front end to Borg. Both Borg and Vorta are .debs, not flatpaks, snaps, or appimages, and are the current versions from the Debian Stable repositories. I have two separate but identical repositories backed up separately, as suggested in the Borg documentation. ?I will call them Repository_1 and Repository_2. Both repositories are on a local usb hard drive (sda). Both repositories are then backed up to another local usb hard drive (sdb), using rsync. I don't know what happened, but last night, when I did: borg -v info [Repository_1 path on (sda)] I got the notorious message: "Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)" [Repository_2 path on (sda)] tests okay. [Repository_ path on (sdb)] and [Repository_2 path on (sdb)] both seem to be okay. Per the Borg 1.2.4 documentation, I did: borg delete --keep-security-info [Repository_1 path on (sda)], then: rsync -aH [Repository_1 path on (sdb)]/ [Repository_1 path on (sda)]. (Yes I did be sure to put a backslash after the source directory path). I still got the same error message. I ran: borg -v check [Repository_1 path on (sda)]. No problems were detected. so, (I hope) the data is okay. In /home/[user]/.config/borg there are a number of folders, all with which seem to be hexadecimal folder names. Two of these hexadecimal- named folders each contain a file named "location", which holds the identical [Repository_1 path(sda)] entry. But the two hexadecimal-named folders each have a different 16-digit hexadecimal number entry in their "nonce" files.? They also have different entries in the "manifest-timestamp" files. The first "manifest-timestamp" file says: 2023-02-05T03:30:05.967949 The second "manifest-timestamp" file says: 2023-07-19T01:21:01.098864 Is there a way to "repair" [Repository_1 path(sda)]? Or, if not, is there a way to make [Repository_1 path(sda)] a "duplicate" of [Repository_2 path(sda)], or [Repository_2 path on (sdb)], both of which are believed to be okay? Any assistance would be greatly appreciated! From hunguponcontent at gmail.com Wed Jul 19 15:05:56 2023 From: hunguponcontent at gmail.com (Default User) Date: Wed, 19 Jul 2023 15:05:56 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> Message-ID: On Wed, 2023-07-19 at 14:59 -0400, Default User wrote: > Hello! > > How many times have you heard this one before . . . > > I am using Borg Version 1.2.4 on Debian 12 (Bookworm) Stable, on a > local x86 64-bit computer. ? > > I normally use Vorta 0.8.0-10 as a front end to Borg.? Both Borg and > Vorta are .debs, not flatpaks, snaps, or appimages, and are the > current > versions from the Debian Stable repositories. > > I have two separate but identical repositories backed up separately, > as > suggested in the Borg documentation. ?I will call them Repository_1 > and > Repository_2. Both repositories are on a local usb hard drive (sda). > > Both repositories are then backed up to another local usb hard drive > (sdb), using rsync. > > I don't know what happened, but last night, when I did: > > borg -v info [Repository_1 path on (sda)] > > I got the notorious message: > > "Cache, or information obtained from the security directory is newer > than repository - this is either an attack or unsafe (multiple repos > with same ID)" > > [Repository_2 path on (sda)] tests okay. > [Repository_ path on (sdb)] and [Repository_2 path on (sdb)] both > seem > to be okay. > > Per the Borg 1.2.4 documentation, I did: > > borg delete --keep-security-info [Repository_1 path on (sda)], then: > rsync -aH [Repository_1 path on (sdb)]/ [Repository_1 path on (sda)]. > (Yes I did be sure to put a backslash after the source directory > path). > > I still got the same error message. > > I ran: > borg -v check [Repository_1 path on (sda)]. > No problems were detected.? so, (I hope) the data is okay. > > In /home/[user]/.config/borg there are a number of folders, all with > which? seem to be hexadecimal folder names. Two of these hexadecimal- > named folders each contain a file named "location", which holds the > identical [Repository_1 path(sda)] entry. > > But the two hexadecimal-named folders each have a different 16-digit > hexadecimal number entry in their "nonce" files.? > > They also have different entries in the "manifest-timestamp" files.? > The first "manifest-timestamp" file says: 2023-02-05T03:30:05.967949 > The second "manifest-timestamp" file says: 2023-07-19T01:21:01.098864 > > Is there a way to "repair" [Repository_1 path(sda)]? > Or, if not, is there a way to make [Repository_1 path(sda)] a > "duplicate" of [Repository_2 path(sda)], or [Repository_2 path on > (sdb)], both of which are believed to be okay? > > Any assistance would be greatly appreciated! > > Sorry to reply to my own post, but I forgot to mention that for all of the repositories mentioned, Encryption is: repokey-blake2. Compression is: LZ4 (default). If there is any other information which would be helpful, please let me know. Thanks! From hunguponcontent at gmail.com Wed Jul 19 15:43:58 2023 From: hunguponcontent at gmail.com (Default User) Date: Wed, 19 Jul 2023 15:43:58 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> Message-ID: On Wed, 2023-07-19 at 15:05 -0400, Default User wrote: > On Wed, 2023-07-19 at 14:59 -0400, Default User wrote: > > Hello! > > > > How many times have you heard this one before . . . > > > > I am using Borg Version 1.2.4 on Debian 12 (Bookworm) Stable, on a > > local x86 64-bit computer. ? > > > > I normally use Vorta 0.8.0-10 as a front end to Borg.? Both Borg > > and > > Vorta are .debs, not flatpaks, snaps, or appimages, and are the > > current > > versions from the Debian Stable repositories. > > > > I have two separate but identical repositories backed up > > separately, > > as > > suggested in the Borg documentation. ?I will call them Repository_1 > > and > > Repository_2. Both repositories are on a local usb hard drive > > (sda). > > > > Both repositories are then backed up to another local usb hard > > drive > > (sdb), using rsync. > > > > I don't know what happened, but last night, when I did: > > > > borg -v info [Repository_1 path on (sda)] > > > > I got the notorious message: > > > > "Cache, or information obtained from the security directory is > > newer > > than repository - this is either an attack or unsafe (multiple > > repos > > with same ID)" > > > > [Repository_2 path on (sda)] tests okay. > > [Repository_ path on (sdb)] and [Repository_2 path on (sdb)] both > > seem > > to be okay. > > > > Per the Borg 1.2.4 documentation, I did: > > > > borg delete --keep-security-info [Repository_1 path on (sda)], > > then: > > rsync -aH [Repository_1 path on (sdb)]/ [Repository_1 path on > > (sda)]. > > (Yes I did be sure to put a backslash after the source directory > > path). > > > > I still got the same error message. > > > > I ran: > > borg -v check [Repository_1 path on (sda)]. > > No problems were detected.? so, (I hope) the data is okay. > > > > In /home/[user]/.config/borg there are a number of folders, all > > with > > which? seem to be hexadecimal folder names. Two of these > > hexadecimal- > > named folders each contain a file named "location", which holds the > > identical [Repository_1 path(sda)] entry. > > > > But the two hexadecimal-named folders each have a different 16- > > digit > > hexadecimal number entry in their "nonce" files.? > > > > They also have different entries in the "manifest-timestamp" > > files.? > > The first "manifest-timestamp" file says: 2023-02- > > 05T03:30:05.967949 > > The second "manifest-timestamp" file says: 2023-07- > > 19T01:21:01.098864 > > > > Is there a way to "repair" [Repository_1 path(sda)]? > > Or, if not, is there a way to make [Repository_1 path(sda)] a > > "duplicate" of [Repository_2 path(sda)], or [Repository_2 path on > > (sdb)], both of which are believed to be okay? > > > > Any assistance would be greatly appreciated! > > > > > > > Sorry to reply to my own post, but I forgot to mention that for all > of > the repositories mentioned, > > Encryption is: repokey-blake2. > Compression is: LZ4 (default). > > If there is any other information which would be helpful, please let > me > know. > > Thanks! > Further information which might be of use: borg -v config [Repository_1 path(sda)] id returns a long hexadecimal number, which is dos NOT exist in: /home/[user]/.cache/borg !! And a mistake . . . The previously mentioned "In /home/[user]/.config/borg" should have been "In /home/[user]/.config/borg/security". I apologize. :) From tw at waldmann-edv.de Thu Jul 20 07:43:07 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 20 Jul 2023 13:43:07 +0200 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> Message-ID: <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> > I have two separate but identical repositories backed up separately, as > suggested in the Borg documentation. If you ran "borg init" for each of them, then they are not identical, but each has a different repository id, which borg uses to differentiate between repos (so each repo has a different cache and security directory), even if they are located at the same path (mountpoint). > Both repositories are then backed up to another local usb hard drive > (sdb), using rsync. By doing that (which is NOT recommended by the docs), you create borg repos with identical IDs and borg can not differentiate between them any more (and might get confused and give such messages as seen in the subject). Additionally, IF you would back up to multiple of them using borg (and not just always rsyncing), you'ld additionally create crypto security issues. BTW, sda and sdb are not necessarily stable names, linux could decide to just swap them (or use sdc, ...). If you want stable names, always use some (U)UID. > They also have different entries in the "manifest-timestamp" files. You could try removing that one. From hunguponcontent at gmail.com Thu Jul 20 20:23:52 2023 From: hunguponcontent at gmail.com (Default User) Date: Thu, 20 Jul 2023 20:23:52 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> Message-ID: <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> On Thu, 2023-07-20 at 13:43 +0200, Thomas Waldmann wrote: > > > > I have two separate but identical repositories backed up > > > > separately, as > > > > suggested in the Borg documentation. > > > > If you ran "borg init" for each of them, then they are not > > identical, > > but each has a different repository id, which borg uses to > > differentiate > > between repos (so each repo has a different cache and security > > directory), even if they are located at the same path (mountpoint). > > > > > > Both repositories are then backed up to another local usb hard > > > > drive > > > > (sdb), using rsync. > > > > By doing that (which is NOT recommended by the docs), you create > > borg > > repos with identical IDs and borg can not differentiate between > > them > > any > > more (and might get confused and give such messages as seen in the > > subject). > > > > Additionally, IF you would back up to multiple of them using borg > > (and > > not just always rsyncing), you'ld additionally create crypto > > security > > issues. > > > > BTW, sda and sdb are not necessarily stable names, linux could > > decide > > to > > just swap them (or use sdc, ...). If you want stable names, always > > use > > some (U)UID. > > > > > > > > They also have different entries in the "manifest-timestamp" > > > > files. > > > > You could try removing that one. > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup Hi, Thomas! Thanks for the reply. Perhaps this would help to clarify: In 2023-02, I created two repositories, one immediately after the other, using borg init. So both had different "ID" nombers assigned by borg. Let me just call them Repository 1 and Repository 2. When created, I immediately did a full backup of user home [/home/[user]/] to both Repository 1 and Repository 2, using borg. So far, all of this was to one external usb hard drive, which I will call Drive A. Both Repository 1 and Repository 2 were then copied by rsync from Drive A to another external hard drive which I will call Drive B.? At that point, Repository 1 and Repository 2 (should) have had separate, but different ID numbers, .cache entries, .config/security entries, and mount points. I have never copied or backed up any borg repositories directly to Drive B using borg. Anything copied to Drive B has been done by rsync. My compter identifies the hard drives (to itself) and manages them, using UUID numbers. But I, as a human, have difficulty thinking in terms of long, non-intuitive UUID numbers, I tend to think in terms of old-fashioned device and mount point names (i.e., /dev/sda, /mnt/hda), or labels assigned by me to the drives (i.e., Drive01, Drive02, etc.) I am sorry for any confusion on that point. Regarding: > "They also have different entries in the "manifest-timestamp" files." At this point, there are two different entries in /home/[user]/.config/borg/security for Repository 1. Each has a different long hexadecimal number identifier. Both entries have different manifest-timestamps: First one: 2023-02-05T03:30:05.967949 Second one:? 2023-07-19T01:21:01.098864 So, should I "try" removing the first one? The second one? Both? How? Should I try removing one or both complete entries in /home/[user]/.config/borg/security? Also note: there is currently NO entry in /home/[user]/.cache/borg that corresponds to Repository 1. Further, although of course at this point borg -v list Repository 1 does not work (just gives the error message), Vorta DOES show Repository 1, but it seems to be missing any archives after 2023-07-10. Strange and inexplicable. --------------- So, at this point, things are so messed up that the real solution would seem to be to somehow: 1) remove all data, .cache, and .config/security entries referring to Repository 1, from Drive A. 2) copy the "good" copy of Repository 1 from Drive B, to Drive A. 3) have borg recreate any necessary metadata, including entries in /home/[user]/.cache/borg and /home/[user]/.config/borg/security.config/security referring to Repository 1 on Drive A, if necessary. 4) have borg use the "good" data now on the re-created Repository 1 on Drive A, which has been copied from Repository 1 on Drive B. 5) Now be able to use the new, re-created Repository 1 on Drive A, as though nothing had happened! Note: repokey-Bake2 was being used before. Ideally, that would still be the case. But using any encryption is not really necessary at all. If the encryption can not be restored, then can the encryption be completely removed? If so, would it be possible to re-encrypt later? [Note: I did see in the borg mailing list archives a thread from last year, regarding this subject. Unfortunately, it left me more confused than enlightened.] But, if none of this is possible, then why did I bother to scrupulously keep two sets of borg backups, and then keep two (theoretically) exact copies of THOSE with rsync? It seems like all of that time and effort was for nothing! Please let me know, so I can figure out what my best course of action is. As a worst case scenario, I may have to: A: 1) Abandon the data from the borg backups made up to this point. 2) Uninstall and purge borg and Vorta completely. 3) Do fresh installs of borg and Vorta. 4) Start using with new data. Hope that the problem I have now does not re-occur so that I lose that dat, too. B:) Wait patiently for borg 2.0 to reach release status. And then for it to become available as a Flatpak, or as a .deb in Debian Stable backports (hopefully sometime this century). After that, repeat with Vorta. C: Forget borg. Just use rsync for data backups. Slower. Uses a LOT more space. And time. (De-duplication is great (when it works). I would hate to lose it. But what is the point if you can't reliably save/access/restore/use your backed up data?) Advice/comments welcome.? But please, "no bully". Thanks! From lazyvirus at gmx.com Thu Jul 20 20:51:18 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Fri, 21 Jul 2023 02:51:18 +0200 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> Message-ID: <20230721025118.71a006bb@msi.defcon1.lan> On Thu, 20 Jul 2023 20:23:52 -0400 Default User wrote: Hi, wild guess : your USB disks are mounted on /DISK_A and /DISK_B (or /whatever/DISK_?) for clarity and discrimination, perhaps even using PARTLABEL or LABEL instead of UUID or PARTUUID. > Advice/comments welcome.? > But please, "no bully". If you don't have a gazillion yottabyte to backup, the fastest way to solve this is to recreate a fresh backup from scratch. This is often the best solution when something went wrong to avoid losing hours or days of researches - there's of course a risk, but as far as SMART and dmesg do not show anything bad, it stays small. Jean-Yves From hunguponcontent at gmail.com Thu Jul 20 22:13:50 2023 From: hunguponcontent at gmail.com (Default User) Date: Thu, 20 Jul 2023 22:13:50 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <20230721025118.71a006bb@msi.defcon1.lan> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> <20230721025118.71a006bb@msi.defcon1.lan> Message-ID: On Fri, 2023-07-21 at 02:51 +0200, Bzzzz wrote: > On Thu, 20 Jul 2023 20:23:52 -0400 > Default User wrote: > > Hi, > > wild guess : your USB disks are mounted on /DISK_A and /DISK_B (or > /whatever/DISK_?) for clarity and discrimination, perhaps even using > PARTLABEL or LABEL instead of UUID or PARTUUID. > > > Advice/comments welcome.? > > But please, "no bully". > > If you don't have a gazillion yottabyte to backup, the fastest way to > solve this is to recreate a fresh backup from scratch. > > This is often the best solution when something went wrong to avoid > losing hours or days of researches - there's of course a risk, but as > far as SMART and dmesg do not show anything bad, it stays small. > > Jean-Yves > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup Hi, Jean-Yves! The two external usb drives are shown by the "mount" command as: "/dev/sda1 on/ media-[user]/HDD00001" and "/dev/sdb1 on /media/[user]/HDD00002"? Borg info is of course not available for Repository 1, but for Repository 2, it reports: Original size Compressed size Deduplicated size All archives: 2.51 TB 2.37 TB 37.73 GB Unique chunks Total chunks Chunk index: 54280 2306762 df -h shows /home as 30Gb used, 159Gb available. (All data in /home would be in /home/[user]/). So, a fresh backup (or two) would not be trivial, but should be doable. But of course this is current data only. It does not include any "historical" data. It would of course only move from this time forward. Note: I have just been using borg (Vorta) to do daily backups of /home/[user]/, and nothing else. From lazyvirus at gmx.com Thu Jul 20 22:53:05 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Fri, 21 Jul 2023 04:53:05 +0200 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> <20230721025118.71a006bb@msi.defcon1.lan> Message-ID: <20230721045305.439dea97@msi.defcon1.lan> On Thu, 20 Jul 2023 22:13:50 -0400 Default User wrote: > On Fri, 2023-07-21 at 02:51 +0200, Bzzzz wrote: > > On Thu, 20 Jul 2023 20:23:52 -0400 > > Default User wrote: > > > > Hi, > > > > wild guess : your USB disks are mounted on /DISK_A and /DISK_B (or > > /whatever/DISK_?) for clarity and discrimination, perhaps even using > > PARTLABEL or LABEL instead of UUID or PARTUUID. > > > > > Advice/comments welcome.? > > > But please, "no bully". > > > > If you don't have a gazillion yottabyte to backup, the fastest way > > to solve this is to recreate a fresh backup from scratch. > > > > This is often the best solution when something went wrong to avoid > > losing hours or days of researches - there's of course a risk, but > > as far as SMART and dmesg do not show anything bad, it stays small. > > > > Jean-Yves > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup > > > Hi, Jean-Yves! > > The two external usb drives are shown by the "mount" command as: > "/dev/sda1 on/ media-[user]/HDD00001" > and > "/dev/sdb1 on /media/[user]/HDD00002"? Wahoo, up to 0xFFFFF disks, I don't know if the USB will supply enough current for all of them ;-p) > Borg info is of course not available for Repository 1, but for > Repository 2, it reports: > > Original size Compressed size Deduplicated size > All archives: 2.51 TB 2.37 TB 37.73 GB > > Unique chunks Total chunks > Chunk index: 54280 2306762 > > df -h shows /home as 30Gb used, 159Gb available. > (All data in /home would be in /home/[user]/). > > So, a fresh backup (or two) would not be trivial, but should be > doable. ? 30 GB on the first backup should be something like 300 minutes. > But of course this is current data only. It does not include > any "historical" data. It would of course only move from this time > forward. Whether it is convenient or not is of course up to your needs. > Note: I have just been using borg (Vorta) to do daily backups of > /home/[user]/, and nothing else. May be you have hit a bug in this software. I use bash scripts and never have had this kind of trouble, but I do not make backup copy. IF you decide for a brand new backup, you might want to : * put any copy order into a script file (always doing a repetitive task manually being a baaad idea, especially if you work when tired) - as a matter of fact, any important thing where you could goof (bad source, bad target, bad path, etc) is better written once for all in a nice (and triple checked) script, * copy all important data to one of those USB disk (or another) with a simple cp or rsync, just in case, * wipe the other one and use it as your borgbackup repo, * smoke a rollmops and conquer the (known) world. Jean-Yves From hunguponcontent at gmail.com Fri Jul 21 11:43:57 2023 From: hunguponcontent at gmail.com (Default User) Date: Fri, 21 Jul 2023 11:43:57 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <20230721045305.439dea97@msi.defcon1.lan> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> <20230721025118.71a006bb@msi.defcon1.lan> <20230721045305.439dea97@msi.defcon1.lan> Message-ID: <8e8e8141fba7cc4d0977e7e9b21230f9ecd8f695.camel@gmail.com> On Fri, 2023-07-21 at 04:53 +0200, Bzzzz wrote: > On Thu, 20 Jul 2023 22:13:50 -0400 > Default User wrote: > > > On Fri, 2023-07-21 at 02:51 +0200, Bzzzz wrote: > > > On Thu, 20 Jul 2023 20:23:52 -0400 > > > Default User wrote: > > > > > > Hi, > > > > > > wild guess : your USB disks are mounted on /DISK_A and /DISK_B > > > (or > > > /whatever/DISK_?) for clarity and discrimination, perhaps even > > > using > > > PARTLABEL or LABEL instead of UUID or PARTUUID. > > > > > > > Advice/comments welcome.? > > > > But please, "no bully". > > > > > > If you don't have a gazillion yottabyte to backup, the fastest > > > way > > > to solve this is to recreate a fresh backup from scratch. > > > > > > This is often the best solution when something went wrong to > > > avoid > > > losing hours or days of researches - there's of course a risk, > > > but > > > as far as SMART and dmesg do not show anything bad, it stays > > > small. > > > > > > Jean-Yves > > > _______________________________________________ > > > Borgbackup mailing list > > > Borgbackup at python.org > > > https://mail.python.org/mailman/listinfo/borgbackup > > > > > > Hi, Jean-Yves! > > > > The two external usb drives are shown by the "mount" command as: > > "/dev/sda1 on/ media-[user]/HDD00001" > > and > > "/dev/sdb1 on /media/[user]/HDD00002"? > > Wahoo, up to 0xFFFFF disks, I don't know if the USB will supply > enough > current for all of them ;-p) > > > Borg info is of course not available for Repository 1, but for > > Repository 2, it reports: > > > > ?????????????? Original size????? Compressed size??? Deduplicated > > size > > All archives:??????? 2.51 TB????????????? 2.37 TB???????????? 37.73 > > GB > > > > ?????????????? Unique chunks???????? Total chunks > > Chunk index:?????????? 54280????????????? 2306762 > > > > df -h shows /home as 30Gb used, 159Gb available. > > (All data in /home would be in /home/[user]/). > > > > So, a fresh backup (or two) would not be trivial, but should be > > doable. > > ? 30 GB on the first backup should be something like 300 minutes. > > > But of course this is current data only. It does not include > > any "historical" data. It would of course only move from this time > > forward. > > Whether it is convenient or not is of course up to your needs. > > > Note: I have just been using borg (Vorta) to do daily backups of > > /home/[user]/, and nothing else. > > May be you have hit a bug in this software. > I use bash scripts and never have had this kind of trouble, but I do > not make backup copy. > > IF you decide for a brand new backup, you might want to : > > * put any copy order into a script file (always doing a repetitive > task > ? manually being a baaad idea, especially if you work when tired) - > as > ? a matter of fact, any important thing where you could goof (bad > ? source, bad target, bad path, etc) is better written once for all > in > ? a nice (and triple checked) script, > > * copy all important data to one of those USB disk (or another) with > a > ? simple cp or rsync, just in case, > > * wipe the other one and use it as your borgbackup repo, > > * smoke a rollmops and conquer the (known) world. > > Jean-Yves > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup Hi, Jean-Yves! Thanks for the your suggestions. I am still pondering what to do. I am still holding out hope that there is a way to restore Repository 1 from Drive B, to Repository 1 on Drive A. Or even to restore either Repository 2 from either Drive A or Repository 2 from Drive B, to Repository 1 on Drive A. I'm waiting. Patiently. :) BTW, I am curious as to your thinking regarding using one external usb drive for rsync backups, and the other one for borg backups. May is ask what is our rationale for that? And, yes you are correct about the use of bash scripts for doing backups.? May I add that the script should always be looked at before using it, to ensure that the script takes into account any changes in what is being backed up that may make the script inappropriate, until the script is updated. Thanks! From lazyvirus at gmx.com Fri Jul 21 11:58:24 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Fri, 21 Jul 2023 17:58:24 +0200 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <8e8e8141fba7cc4d0977e7e9b21230f9ecd8f695.camel@gmail.com> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> <20230721025118.71a006bb@msi.defcon1.lan> <20230721045305.439dea97@msi.defcon1.lan> <8e8e8141fba7cc4d0977e7e9b21230f9ecd8f695.camel@gmail.com> Message-ID: <20230721175824.32c3ea5e@msi.defcon1.lan> On Fri, 21 Jul 2023 11:43:57 -0400 Default User wrote: > Hi, Jean-Yves! Ho, > Thanks for the your suggestions. > I am still pondering what to do. Well, this is what I was talking about, the time taken to ponder is already largely more than the backup time? > > BTW, I am curious as to your thinking regarding using one external usb > drive for rsync backups, and the other one for borg backups. > May is ask what is our rationale for that? Security, you first make a literal backup (copy) of your data on this disk, just in case, then you borgbackup the machine ; if the machine's disk break during the backup, you still have your data. The law of Murphy having a special affinity with IT, don't think bad, always think worse. Jean-Yves From hunguponcontent at gmail.com Fri Jul 21 14:08:21 2023 From: hunguponcontent at gmail.com (Default User) Date: Fri, 21 Jul 2023 14:08:21 -0400 Subject: [Borgbackup] Problem: multiple repos with same ID In-Reply-To: <20230721175824.32c3ea5e@msi.defcon1.lan> References: <635e39de5f554eb7a6bfc859166698653c771a13.camel@gmail.com> <49efdf67-c62f-388c-231e-daf8a8dbc0bc@waldmann-edv.de> <1e073dc883a57174d5d086ae5e3ce5b26353f52f.camel@gmail.com> <20230721025118.71a006bb@msi.defcon1.lan> <20230721045305.439dea97@msi.defcon1.lan> <8e8e8141fba7cc4d0977e7e9b21230f9ecd8f695.camel@gmail.com> <20230721175824.32c3ea5e@msi.defcon1.lan> Message-ID: <78520e9a8ff6d33c9947cfa16ca3b7a648e7fcd3.camel@gmail.com> On Fri, 2023-07-21 at 17:58 +0200, Bzzzz wrote: > On Fri, 21 Jul 2023 11:43:57 -0400 > Default User wrote: > > > Hi, Jean-Yves! > > Ho, > > > Thanks for the your suggestions. > > I am still pondering what to do. > > Well, this is what I was talking about, the time taken to ponder is > already largely more than the backup time? > > > > > BTW, I am curious as to your thinking regarding using one external > > usb > > drive for rsync backups, and the other one for borg backups.? > > May is ask what is our rationale for that? > > Security, you first make a literal backup (copy) of your data on this > disk, just in case, then you borgbackup the machine ; if the > machine's > disk break during the backup, you still have your data. > > The law of Murphy having a special affinity with IT, don't think bad, > always think worse. > > Jean-Yves > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup Hi, Jean-Yves! > "Well, this is what I was talking about, the time taken to ponder is already largely more than the backup time?" True. But as any experienced carpenter will say, "Measure twice, cut once". Thanks for explaining your rationale. As for Murphy's Law, unfortunately, I am very familiar with it. It was written based upon my life experience! :O With regards, D. User From tw at waldmann-edv.de Wed Aug 30 09:00:48 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 30 Aug 2023 15:00:48 +0200 Subject: [Borgbackup] borgbackup 1.2.5 released, including a security fix! Message-ID: borgbackup 1.2.5 was just released (including a security fix)! https://github.com/borgbackup/borg/releases/tag/1.2.5 --- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Thu Aug 31 17:51:15 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 31 Aug 2023 23:51:15 +0200 Subject: [Borgbackup] borgbackup 1.2.6 released, including a security fix! Message-ID: borgbackup 1.2.6 was just released (including a security fix)! https://github.com/borgbackup/borg/releases/tag/1.2.6 Basically it is the same as the 1.2.5 release of yesterday, but fixes issue #7791. https://github.com/borgbackup/borg/issues/7791 Also, the cython version restrictions were removed, so package maintainers can now use any working Cython version they want. The bundled .c files in the pypi package were built with latest Cython 0.29.x (which we use since years), but there are good chances that the rather fresh 3.0.x releases would work also. --- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From bjo at schafweide.org Sun Sep 3 02:49:49 2023 From: bjo at schafweide.org (Bjoern Franke) Date: Sun, 3 Sep 2023 08:49:49 +0200 Subject: [Borgbackup] Data integrity error: Archive authentication did not verify Message-ID: <4ca9f632-ad53-4414-ba66-ea96d7283dcc@schafweide.org> Hi, with a repo which ran a recreate some weeks ago I get "security: repository checks ok, allowing access Synchronizing chunks cache... Archives: 18, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 18. Fetching and building archive index for home-2023-02-28T11:35:01.226774 ... Archive authentication DISABLED. RepositoryCache: current items 0, size 0 B / 982.22 MB, 0 hits, 1 misses, 0 slow misses (+0.0s), 0 evictions, 0 ENOSPC hit RemoteRepository: 302 B bytes sent, 56.27 kB bytes received, 8 messages sent Data integrity error: Archive authentication did not verify" I stumpled upon https://github.com/borgbackup/borg/discussions/7787, the installed version is 1.2.6, but I think some archives also have been created with 1.2.5. Any ideas how to fix this? Regards Bjoern From bjo at schafweide.org Sun Sep 3 05:40:39 2023 From: bjo at schafweide.org (Bjoern Franke) Date: Sun, 3 Sep 2023 11:40:39 +0200 Subject: [Borgbackup] Data integrity error: Archive authentication did not verify In-Reply-To: <4ca9f632-ad53-4414-ba66-ea96d7283dcc@schafweide.org> References: <4ca9f632-ad53-4414-ba66-ea96d7283dcc@schafweide.org> Message-ID: <95c0c53f-147a-43ff-9e3a-936d01289301@schafweide.org> Hi again, > Hi, > > with a repo which ran a recreate some weeks ago I get > > "security: repository checks ok, allowing access > Well, reading the upgrade instructions helped. Sorry for the noise. BTW, is the following issue caused by borgmatic? BORG_WORKAROUNDS=ignore_invalid_archive_tam borgmatic borg list --format='{name} {time} tam:{tam}{NL}' Borg's list subcommand is supported natively by borgmatic. Try this instead: borgmatic list Invalid placeholder "time" in string: {time} Regards Bjoern From bkborg at kirk.de Sun Sep 3 10:04:35 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sun, 3 Sep 2023 16:04:35 +0200 Subject: [Borgbackup] borgbackup 1.2.6 released, including a security fix! In-Reply-To: References: Message-ID: <61b2002e-c9d3-d391-e43d-4f64225554db@kirk.de> Hi, are the steps described in > necessary if borg's encryption was not used when creating the repos? Am 31.08.23 um 23:51 schrieb Thomas Waldmann: > borgbackup 1.2.6 was just released (including a security fix)! > > https://github.com/borgbackup/borg/releases/tag/1.2.6 > > Basically it is the same as the 1.2.5 release of yesterday, but fixes > issue #7791. > > https://github.com/borgbackup/borg/issues/7791 > > Also, the cython version restrictions were removed, so package > maintainers can now use any working Cython version they want. > > The bundled .c files in the pypi package were built with latest Cython > 0.29.x (which we use since years), but there are good chances that the > rather fresh 3.0.x releases would work also. > > --- > > GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Mit freundlichem Gru? Best regards ? Kirkorowicz From felix.schwarz at oss.schwarz.eu Fri Sep 8 03:03:03 2023 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Fri, 8 Sep 2023 09:03:03 +0200 Subject: [Borgbackup] borgbackup 1.2.6 released, including a security fix! In-Reply-To: References: Message-ID: <56dc664b-e238-c2ed-57cd-19dda619cd48@oss.schwarz.eu> Hi Thomas, thank you for all your work on borgbackup, including handling this bug. I updated Fedora and EPEL 9 to provide borgbackup 1.2.6. My question is related to borg 1.1.x which we ship for EPEL 7 and EPEL 8. From reading the CVE I think these versions are vulnerable as well, right? Do you plan on backporting the patch or are you aware if someone else does that? In general I am not too concerned about this security issue (as it requires repo-level write access) but if there is an easy way I would for sure add a patch. Best, Felix From tw at waldmann-edv.de Fri Sep 8 08:41:13 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Sep 2023 14:41:13 +0200 Subject: [Borgbackup] borgbackup 1.2.6 released, including a security fix! In-Reply-To: <61b2002e-c9d3-d391-e43d-4f64225554db@kirk.de> References: <61b2002e-c9d3-d391-e43d-4f64225554db@kirk.de> Message-ID: <7a8765d8-a4de-4801-a1ce-9d3bfde571ce@waldmann-edv.de> > are the steps described in >> > necessary if borg's encryption was not used when creating the repos? Yes, the steps should be done for all repos, including not encrypted repos. From tw at waldmann-edv.de Fri Sep 8 08:49:08 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 8 Sep 2023 14:49:08 +0200 Subject: [Borgbackup] borgbackup 1.2.6 released, including a security fix! In-Reply-To: <56dc664b-e238-c2ed-57cd-19dda619cd48@oss.schwarz.eu> References: <56dc664b-e238-c2ed-57cd-19dda619cd48@oss.schwarz.eu> Message-ID: <97d2efe1-9aa0-4fa7-8336-d2a29f633062@waldmann-edv.de> > I updated Fedora and EPEL 9 to provide borgbackup 1.2.6. Great, thanks! > My question is related to borg 1.1.x which we ship for EPEL 7 and EPEL > 8. From reading the CVE I? think these versions are vulnerable as well, > right? > Do you plan on backporting the patch or are you aware if someone else > does that? Currently nobody is working on that. Maybe I'll give it a try. Another 1.1.x release is not planned though (1.1 is EOL as far as I am concerned), but of course some changesets in 1.1-maint branch would be good for whoever wants to pick them up. > In general I am not too concerned about this security issue (as it > requires repo-level write access) but if there is an easy way I would > for sure add a patch. Yeah, an attacker would need quite some access: repo write access and also write access to backup source files. Besides fixing the vulnerability, it would be also good if most people apply the repo upgrade steps, so we have that off the table ASAP. From richjunk at pacbell.net Fri Sep 8 18:29:44 2023 From: richjunk at pacbell.net (Richard) Date: Fri, 8 Sep 2023 15:29:44 -0700 Subject: [Borgbackup] Current Off-Line Documentation In-Reply-To: References: Message-ID: The latest 1.x version of borgbackup off-line documentation appears to be v1.2.3. Where is the newer (v1.2.6) version of the offline documentation? I am looking here: https://readthedocs.org/projects/borgbackup/downloads/ Thanks. From w.schuermann at posteo.de Sat Sep 9 00:09:15 2023 From: w.schuermann at posteo.de (=?UTF-8?Q?Winfried_Sch=c3=bcrmann?=) Date: Sat, 9 Sep 2023 04:09:15 +0000 Subject: [Borgbackup] Current Off-Line Documentation In-Reply-To: References: Message-ID: <959227a6-76c3-1b17-d369-88814c628201@posteo.de> Off-line? Download the documentation of version 1.2.6 to your courrent working directory using "wget": wget -rkpE -np https://borgbackup.readthedocs.io/en/stable/ To read, find index.html. Winfried Am 09.09.23 um 00:29 schrieb Richard: > > > The latest 1.x version of borgbackup off-line documentation appears to > be v1.2.3.? Where is the newer (v1.2.6) version of the offline > documentation??? I am looking here: > > https://readthedocs.org/projects/borgbackup/downloads/ > > Thanks. > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Thu Sep 14 17:58:56 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 14 Sep 2023 23:58:56 +0200 Subject: [Borgbackup] borg 2.0.0b7 released! Message-ID: <747ade90-e6a9-44a3-adf5-d82b1989f5e8@waldmann-edv.de> Just released a new #borgbackup 2 beta: https://github.com/borgbackup/borg/releases/tag/2.0.0b7 #linux #bsd #macos From felix.schwarz at oss.schwarz.eu Wed Sep 27 02:45:29 2023 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Wed, 27 Sep 2023 08:45:29 +0200 Subject: [Borgbackup] borg outputs "Failed to securely erase old repository config file" twice in borg check Message-ID: <24b80066-a4b3-755e-9377-101fc8f8ab2c@oss.schwarz.eu> Hi *, recently there was a change on our storage platform which hosts our borg repos. Now the remote filesystem does not support hardlinks anymore. So every "borg check" outputs some lines like this: # borg check --repository-only Remote: Failed to securely erase old repository config file (hardlinks not supported). Old repokey data, if any, might persist on physical storage. Remote: Failed to securely erase old repository config file (hardlinks not supported). Old repokey data, if any, might persist on physical storage. and that is really annoying because - as you can guess - this triggers an email if the "check" job is run by cron. After reading the discussion in various github issues (especially https://github.com/borgbackup/borg/issues/3591#issuecomment-363362442) I think I have a rough idea why borg outputs the warning but I think it is not relevant in our case (not using repokeys). # rpm -q borgbackup borgbackup-1.1.18-2.el7.x86_64 Questions: - Is there a way silence the warning? - Why does "borg check" want to rewrite the config? Thank you for your time, Felix From tw at waldmann-edv.de Wed Sep 27 11:21:59 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Sep 2023 17:21:59 +0200 Subject: [Borgbackup] borg outputs "Failed to securely erase old repository config file" twice in borg check In-Reply-To: <24b80066-a4b3-755e-9377-101fc8f8ab2c@oss.schwarz.eu> References: <24b80066-a4b3-755e-9377-101fc8f8ab2c@oss.schwarz.eu> Message-ID: <0d6a8c0d-a26c-4c92-a478-168abb0fcfc7@waldmann-edv.de> > Remote: Failed to securely erase old repository config file (hardlinks > not supported). Old repokey data, if any, might persist on physical > storage. That's usually not a security problem, except if you changed your passphrase to re-protect the key after a passphrase disclosure. > and that is really annoying because - as you can guess - this triggers > an email if the "check" job is run by cron. Ah, I see. > After reading the discussion in various github issues (especially > https://github.com/borgbackup/borg/issues/3591#issuecomment-363362442) I > think I have a rough idea why borg outputs the warning but I think it is > not relevant in our case (not using repokeys). Yeah, in that case you are not affected. > - Is there a way silence the warning? IIRC not in borg, but you could use this to filter it out: borg ... | grep -v Failed.to.securely.erase.old.repository.config > - Why does "borg check" want to rewrite the config? Not sure why precisely it is in that case, but some of the reasons are: - keeping the state of progress of some repo-level operations (e.g. borg check - but not sure if that was already in 1.1.x) - change of passphrase with repokey - borg config commands that change repo config values There is a ticket about splitting the config into usually-static and dynamic parts, but it is not done yet. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From kaycarlson at disroot.org Thu Sep 28 13:17:31 2023 From: kaycarlson at disroot.org (Kay) Date: Thu, 28 Sep 2023 19:17:31 +0200 Subject: [Borgbackup] I accidentally pushed /home/$USER/.config/borg into public git repo. Big security risk? Message-ID: Hi I accidentally pushed /home/$USER/.config/borg into public git repo. Is it a big security risk? But nobody has the passphrase. Kay From lazyvirus at gmx.com Fri Sep 29 12:37:18 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Fri, 29 Sep 2023 18:37:18 +0200 Subject: [Borgbackup] I accidentally pushed /home/$USER/.config/borg into public git repo. Big security risk? In-Reply-To: References: Message-ID: <20230929183718.147cbddd@msi.defcon1.lan> On Thu, 28 Sep 2023 19:17:31 +0200 Kay via Borgbackup wrote: > Hi Ho > I accidentally pushed /home/$USER/.config/borg into public git repo. > Is it a big security risk? Normally no, as there's no risks even when the key's in the repo. > But nobody has the passphrase. It depends on the password strength, if it is : "admin1234", you're already doomed, if it is : "the-0bes3s_neVeR^|anDeD at zE?M0On", you're less doomed ;-p) It also depends on where the repo is located and who has access. If you really fret about this, you can sshfs mount this repo's backups to backup them one after another into another BB repo. > Kay Rule of thumb, when you have to push data to the public area, never work from $HOME, use a reserved and secluded place for work dedicated to exportation. Jean-Yves From tw at waldmann-edv.de Sat Sep 30 08:42:47 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 30 Sep 2023 14:42:47 +0200 Subject: [Borgbackup] I accidentally pushed /home/$USER/.config/borg into public git repo. Big security risk? In-Reply-To: References: Message-ID: > I accidentally pushed /home/$USER/.config/borg into public git repo. Is > it a big security risk? > > But nobody has the passphrase. If you have a good passphrase, i don't think it's a big risk. The keys in borg/keys/... are AES enrypted using a key derived from your passphrase. So, your security is now "having the passphrase" kind of security, similar as if you used repokey mode (which stores the key in the repo config). -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393